Client-calculated content metadata-based voice inquiry service quality (QoS)

Abstract

To deal with a traffic abrupt increase in a processing stack of a server base.SOLUTION: A method includes receiving an automated speech recognition (ASR) request from a user device that includes a speech input captured by the user device and content metadata associated with the speech input. The content metadata is generated by the user device. The method also includes determining a priority score for the ASR request based on the content metadata. The method includes caching the ASR request in a pre-processing backlog of pending ASR requests each having a corresponding priority score. The pending ASR requests in the pre-processing backlog are ranked in order of the priority scores. The method also includes providing, from the pre-processing backlog, one or more of the pending ASR requests to a backend-side ASR module. Pending ASR requests associated with higher priority scores are processed before pending ASR requests associated with lower priority scores.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to the quality of service (QoS) of voice queries based on content metadata calculated by a client.

Background Art

[0002] In voice-enabled environments (such as homes, workplaces, schools, automobiles, etc.), a user can speak loudly to a computer-based system with queries or commands, and the system can field the queries and provide answers or execute functions based on the commands. A voice-enabled environment can be realized using a network of connected microphone devices arranged in various rooms or areas of the environment. These devices can use hotwords to discern whether a given utterance is directed at the system or at another individual within the environment. Thus, the devices may operate in a sleep or hibernation state and wake up only when a detected utterance contains a hotword. Query processing performed by a backend server is costly and may overload the server with more queries than it can process at one time. For example, if a hotword is included in a TV program or commercial during a large-scale event, the server may become overloaded and a failure may occur.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is room for improvement in the quality of service (QoS) of voice queries based on content metadata calculated by a client. [Means for solving the problem]

[0005] Voice-enabled devices (e.g., user devices that run voice assistants) are designed to process inquiries and commands spoken aloud by the user, allowing the device to field and answer those inquiries or perform functions based on those commands. Through the use of "hotwords" (also called "keywords," "attention words," "wake-up phrases / words," "trigger phrases," or "voice action start commands"), which are predetermined terms / phrases uttered to draw the attention of the voice-enabled device and are reserved by agreement, the voice-enabled device can distinguish between utterances directed at the system (i.e., to initiate wake-up processing to process one or more terms following the hotword in the utterance) and utterances directed at individuals in the environment. Typically, voice recognition devices operate in a sleep or low-power state to conserve battery power, processing incoming audio data (voice data) to detect spoken hotwords. For example, in low-power mode, the voice-enabled device takes in the input audio via a microphone and uses a hotword detector trained to detect the presence of hotwords in the input audio. When a hotword is detected in the input audio, the voice-enabled device initiates a wake-up process to process the hotword and / or any other terms in the input audio that follow the hotword.

[0006] Typically, after a speech-enabled device detects the presence of a hotword in a speech utterance (e.g., input speech) and wakes up, the speech-enabled device transmits the hotword and one or more other terms following the hotword over the network to a server-based processing stack (also called a query processing backend) which includes at least an automatic speech recognition (ASR) configured to process the hotword and / or any other terms following the hotword. Here, the ASR may process the received speech as an ASR request and transcribe the hotword and / or other terms following the hotword into corresponding text. This text may be provided to an interpreter layer to determine the speech query specified by the input speech and provide the query to the appropriate components to perform the actions associated with the query. Therefore, when a user of a voice-enabled device speaks something like, "Hey Google, what restaurants are still open right now?", the voice-enabled device may wake up in response to the detection of the hotword "Hey Google" and provide the term following the hotword, "Whatnear restaurants are still open right now," to the server-based processing stack for processing in response to the voice query. In this example, the automatic speech recognition (ASR) in the server-based processing stack transcribes the voice query into corresponding text, the interpretation layer determines that a search for nearby restaurants' opening hours is needed, and the search engine retrieves a list of search results including nearby restaurants that are currently open. The search results are provided to the voice-enabled device for display or audio output. In some scenarios, the server-based processing stack also includes a text-to-speech (TTS) converter configured to convert the list of search results into synthesized speech, which is then provided back to the voice-enabled device for audio output.

[0007] A server-based processing stack is configured to process voice queries received from multiple voice-enabled devices associated with an entire user population. This could involve millions of voice-enabled devices sending voice queries for processing by the server-based processing stack. Processing voice queries is an expensive task, and in some situations, the server-based processing stack can become overloaded with more voice queries than it can process in a given time. For example, if a hotword or other term that sounds similar to a hotword exists in a large television event (e.g., a commercial during the Super Bowl), a nearby voice-enabled device (e.g., one close to a television in a home) might detect the hotword and issue an unintended voice query to the server-based processing stack, causing a very large spike in traffic to the server-based processing stack. While it is conceivable that the server-based processing stack could simply drop voice queries that were not initiated by actual users and / or are not time-critical, identifying such queries without initiating expensive processing is difficult.

[0008] The implementation of this specification is directed to a query processing backend (e.g., a server-based processing stack) that receives Automated Speech Recognition (ASR) requests from a voice-enabled device (e.g., a user device). Each ASR request includes a corresponding voice input captured by the user device with a voice query for processing, as well as content metadata generated by the voice-enabled device and associated with the voice input. Based on the content metadata associated with the voice input, the query processing backend can determine the priority score of each ASR request and cache the ASR requests in a pre-processing backlog of pending ASR requests, each with a corresponding priority score and ranked in order of priority score. The query processing backend may then provide one or more pending ASR requests from the pre-processing backlog to the backend ASR module (or other components of the query processing backend) based on the processing availability of the backend ASR module. Here, instead of the backend Auto Speech Recognition (ASR) module being overloaded during traffic spikes by attempting to process each pending ASR request on a first-come, first-served basis, ASR requests are prioritized so that the backend ASR module processes pending ASR requests associated with higher priority scores before processing those associated with lower priority scores. When a new ASR request comes in, pending ASR requests in the pre-processing backlog are sorted based on their priority scores. Ideally, ASR requests associated with unintended voice queries that are unlikely to be initiated by actual users and / or are not time-critical are assigned lower priority scores.In this way, Auto Speech Recognition ASR requests associated with lower priority scores remain in the pre-processing backlog during traffic spikes, allowing the backend Auto Speech Recognition ASR module to process requests associated with higher priority scores first.

[0009] In some cases, a priority score below a certain threshold may simply result in the corresponding Auto Speech Recognition (ASR) request being dropped. Furthermore, a low priority score determined from content metadata can be a strong indicator that the processing of the corresponding Auto Speech Recognition (ASR) request will be unsuccessful. For example, content metadata may indicate very poor audio quality associated with the audio input, and therefore, poor audio quality can indicate that the backend Auto Speech Recognition (ASR) module has difficulty successfully transcribing the audio data into corresponding text. Simultaneously, poor audio quality associated with audio captured by the speech-enabled device may indicate that the user who spoke the hotword (or a similar-sounding word) was not in close proximity to the speech-enabled device and therefore likely did not intend to provide audio input to it. Content metadata can also indicate whether the audio input was likely spoken by the user associated with the speech-enabled device. For example, the hotword detector in the speech-enabled device can compare the audio input to the user's voice profile to determine whether the audio input was likely spoken by that user. If content metadata indicates that a user likely spoke the voice input, the corresponding Auto Speech Recognition (ASR) request may be assigned a higher priority score than if another user spoke the voice input. Conversely, if content metadata indicates that broadcast audio from another user or a voice broadcasting device (e.g., a television, music speaker, or other non-human source capable of outputting acoustic sound) initiated the voice input, the corresponding Auto Speech Recognition (ASR) request may be assigned a lower priority score. Content metadata can consist of any type of data computed / generated by the voice-enabled device and included in the Auto Speech Recognition (ASR) request provided to the query processing backend, allowing the query processing backend to prioritize the importance of the Auto Speech Recognition (ASR) requests without performing any processing on them (or with at least a very limited amount of processing).From the aforementioned perspective, the content metadata associated with the voice input represents the likelihood that the corresponding Auto Speech Recognition (ASR) request will be successfully processed by the backend Auto Speech Recognition (ASR) module, and / or that the processing of the corresponding Auto Speech Recognition (ASR) request will affect the user associated with the voice-enabled device.

[0010] In home audio devices, voice processing is often performed on a server and during peak times, which can create a large backlog of voice processing requests. Some of these requests may be genuine, while others may be the result of broadcast audio (e.g., voice output from non-human sources such as television, radio, or synthesized speech). The object of this invention is to provide a method for improving the processing of large volumes of voice recognition requests. By prioritizing requests, the voice recognition module can process more important or urgent requests while lowering the priority of others. Thus, the use of the voice recognition module can be optimized when it is overloaded.

[0011] One aspect of this disclosure provides a method for providing quality of service for voice queries. The method comprises receiving an Auto Speech Recognition (ASR) request from a user device in data processing hardware at the query processing backend. The Auto Speech Recognition ASR request comprises a voice input captured by the user device and content metadata associated with the voice input. The voice input comprises a voice query, and the content metadata is generated by the user device. The method also comprises the data processing hardware determining a priority score for the Auto Speech Recognition ASR request based on the content metadata associated with the voice input. The method further comprises the data processing hardware caching the Auto Speech Recognition ASR requests in a pre-processing backlog of pending Auto Speech Recognition ASR requests, each having a corresponding priority score. The pending Auto Speech Recognition ASR requests in the pre-processing backlog are ranked in order of priority score. The method further comprises the data processing hardware providing one or more of the pending Auto Speech Recognition ASR requests from the pre-processing backlog to the backend Auto Speech Recognition ASR module based on the processing availability of the backend Auto Speech Recognition ASR module. Pending Auto Speech Recognition (ASR) requests associated with a higher priority score are processed by the backend Auto Speech Recognition (ASR) module before pending Auto Speech Recognition (ASR) requests associated with a lower priority score.

[0012] An implementation of this disclosure may include one or more of the following optional features. In some implementations, the backend Auto Speech Recognition (ASR) module is configured to process each pending Auto Speech Recognition (ASR) request in response to receiving each pending Auto Speech Recognition (ASR) request from the preprocessing backlog of pending Auto Speech Recognition (ASR) requests and to produce an Auto Speech Recognition (ASR) result for the corresponding speech input associated with the pending Auto Speech Recognition (ASR) request. In some examples, the method further includes, in response to caching one or more new Auto Speech Recognition (ASR) requests in the preprocessing backlog of pending Auto Speech Recognition (ASR) requests, having the data processing hardware re-rank (re-prioritize) the pending Auto Speech Recognition (ASR) requests in the preprocessing backlog in order of priority score. Furthermore, or alternatively, the method may include, by the data processing hardware, refusing to allow any pending Auto Speech Recognition (ASR) requests present in the preprocessing backlog to be processed by the backend Auto Speech Recognition (ASR) module for a period of time that satisfies a timeout threshold. In some implementations, the method further includes, in response to receiving a new Automated Speech Recognition ASR request with a priority score lower than the priority score threshold, having the data processing hardware refuse to allow the new Automated Speech Recognition ASR request to be processed by the backend Automated Speech Recognition ASR module.

[0013] Content metadata associated with a speech input may indicate the likelihood that the corresponding Auto Speech Recognition (ASR) request will be successfully processed by the backend Auto Speech Recognition (ASR) module. In some implementations, content metadata associated with a speech input may indicate the likelihood that processing the corresponding Auto Speech Recognition (ASR) request will affect the user associated with the user device. Content metadata associated with a speech input and generated by the user device may include at least one of the following: A login metric indicating whether the user associated with the user device is logged into the user device. A speaker identification score for the speech input indicating the likelihood that the speech input matches a speaker profile associated with the user device. A broadcast voice score for the speech input indicating the likelihood that the speech input corresponds to broadcast or synthesized voice output from a non-human source. A hotword confidence score indicating the likelihood that one or more terms preceding the speech query in the speech input correspond to a predefined hotword. An activity metric indicating whether a multi-turn interaction is in progress between the user device and the query processing backend. A speech signal score for the speech input. A spatial localization score (spatial-location score) indicating the user's distance and location to the user device. A transcription of the voice input generated by an on-device automatic speech recognition (ASR) module residing on the user device. User device behavior signals indicating the current behavior of the user device, or environmental condition signals indicating the current environmental conditions related to the user device.

[0014] In some implementations, the user device is configured to capture a voice input containing a voice query in response to detecting a hotword preceding the voice query in the voice utterance, generate content metadata associated with the voice input, and send a corresponding Auto Speech Recognition ASR request to the data processing hardware. The voice input may further contain a hotword. In some examples, the method further comprises sending an on-device processing instruction from the data processing hardware to the user device. The on-device processing instruction provides one or more criteria for locally processing at least a portion of any new voice inputs captured by the user device on-device if the user device determines that the query processing backend is overloaded. In these examples, the user device may be configured to determine that the query processing backend is overloaded by at least one of the following: obtaining historical data associated with previous Auto Speech Recognition ASR requests communicated by the user device to the data processing hardware; receiving a schedule of past and / or predicted overload conditions in the query processing backend from the data processing hardware; or receiving an on-the-fly overload condition status notification from the data processing hardware indicating a current overload condition in the processing backend. Furthermore, one or more criteria for locally processing at least a portion of any new voice input may include instructing the user device to transcribe the new voice input using a local automatic speech recognition (ASR) module residing on the device; interpret the transcription of the new voice input to determine the voice query corresponding to the new voice input; determine whether the user device can perform the actions associated with the voice query corresponding to the new voice input; or, if the user device cannot perform the actions associated with the voice query, to send the transcription of the voice input to the query processing system.In some implementations, an on-device processing instruction provides one or more criteria, which include one or more thresholds that the corresponding portion of the content metadata must satisfy before the user device sends an Auto Speech Recognition ASR request to the query processing backend. In some examples, the on-device processing instruction further instructs the user device to drop the Auto Speech Recognition ASR request if at least one of the thresholds is not satisfied.

[0015] Another aspect of this disclosure provides a system for providing quality of service for voice queries. The system comprises data processing hardware for query processing backends and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform actions. Actions include receiving an Auto Speech Recognition (ASR) request from a user device. An ASR request comprises a voice input captured by the user device and content metadata associated with the voice input. The voice input comprises a voice query, and the content metadata is generated by the user device. The action also includes determining a priority score for the ASR request based on the content metadata associated with the voice input, and caching the ASR request in a pre-processing backlog of pending ASR requests, each having a corresponding priority score. Pending ASR requests in the pre-processing backlog are ranked in order of their priority scores. The operation further includes providing the backend-side automatic speech recognition (ASR) module with one or more pending ASR requests from the preprocessing backlog, based on the processing availability of the ASR module. Pending ASR requests associated with higher priority scores are processed by the backend-side automatic speech recognition (ASR) module before pending ASR requests associated with lower priority scores.

[0016] This embodiment may include one or more of the following optional features. In some implementations, the backend Auto Speech Recognition (ASR) module is configured to process each pending Auto Speech Recognition (ASR) request in response to receiving each pending Auto Speech Recognition (ASR) request from the preprocessing backlog of pending Auto Speech Recognition (ASR) requests and to produce an Auto Speech Recognition (ASR) result for the corresponding speech input associated with the pending Auto Speech Recognition (ASR) request. In some examples, the operation further includes reranking (reordering) the pending Auto Speech Recognition (ASR) requests in the preprocessing backlog in order of priority score in response to caching one or more new Auto Speech Recognition (ASR) requests in the preprocessing backlog of pending Auto Speech Recognition (ASR) requests. Furthermore, or alternatively, the operation may further include refusing to process any pending Auto Speech Recognition (ASR) requests present in the preprocessing backlog for a period of time that satisfies a timeout threshold, by the backend Auto Speech Recognition (ASR) module. In some implementations, the behavior further includes, in response to receiving new Auto Speech Recognition ASR requests with respective priority scores lower than the priority score threshold, refusing to allow the new Auto Speech Recognition ASR requests to be processed by the backend Auto Speech Recognition ASR module.

[0017] Content metadata associated with voice input may indicate the likelihood that the corresponding Auto Speech Recognition (ASR) request will be successfully processed by the backend Auto Speech Recognition (ASR) module. In some examples, the content metadata associated with voice input may indicate that the processing of the corresponding Auto Speech Recognition (ASR) request will affect the user associated with the user device. Content metadata associated with a voice input and generated by the user device may include at least one of the following: a login metric indicating whether the user associated with the user device is logged into the user device; a speaker identification score for the voice input indicating the likelihood that the voice input matches a speaker profile associated with the user device; a broadcast voice score for the voice input indicating the likelihood that the voice input corresponds to broadcast or synthesized voice output from a non-human source; a hotword confidence score indicating the likelihood that one or more terms preceding the voice query in the voice input correspond to a predefined hotword; an activity metric indicating whether a multi-turn interaction is in progress between the user device and the query processing backend; a voice signal score for the voice input; a spatial localization (spatial-positioning) score indicating the user's relative distance and relative position to the user device; a transcript of the voice input generated by an on-device automatic speech recognition (ASR) module residing on the user device; and a user device behavior signal indicating the current behavior of the user device, or an environmental condition signal indicating the current environmental conditions associated with the user device.

[0018] In some implementations, the user device is configured to capture a voice input containing a voice query in response to detecting a hotword preceding the voice query in the voice utterance, generate content metadata associated with the voice input, and send a corresponding Auto Speech Recognition ASR request to the data processing hardware. The voice input may further contain a hotword. In some examples, the operation further comprises sending an on-device processing instruction to the user device. The on-device processing instruction provides one or more criteria for locally processing at least a portion of any new voice inputs captured by the user device on-device if the user device determines that the query processing backend is overloaded. In these examples, the user device may be configured to determine that the query processing backend is overloaded by at least one of the following: retrieving historical data associated with previous Auto Speech Recognition ASR requests communicated by the user device to the data processing hardware; receiving a schedule of past and / or predicted overload conditions in the query processing backend from the data processing hardware; or receiving an on-the-fly overload condition status notification from the data processing hardware indicating a current overload condition in the processing backend. In further examples, one or more criteria for locally processing at least a portion of any new speech input include instructing the user device to transcribe the new speech input using a local automatic speech recognition (ASR) module residing on the device, interpret the transcription of the new speech input to determine the speech query corresponding to the new speech input, determine whether the user device can perform the action associated with the speech query corresponding to the new speech input, or, if the user device cannot perform the action associated with the speech query, send the transcription of the speech input to the query processing system. In some implementations, the on-device processing instruction providing one or more criteria includes one or more thresholds that the corresponding portion of the content metadata must satisfy in order for the user device to send the automatic speech recognition (ASR) request to the query processing backend.In some examples, the on-device processing instructions further instruct the user device to drop the automatic speech recognition (ASR) request if at least one of the thresholds is not satisfied.

[0019] Details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

Brief Description of the Drawings

[0020] [Figure 1] FIG. schematically shows an example of a system for prioritizing pending automatic speech recognition (ASR) requests received from a user device. [Figure 2] FIG. schematically shows an example of a user device that generates content metadata associated with voice input captured by the user device. [Figure 3A] FIG. schematically shows an exemplary voice query service quality (QoS) manager configured to continuously re-rank pending automatic speech recognition (ASR) requests. [Figure 3B] FIG. schematically shows an exemplary voice query service quality (QoS) manager configured to continuously re-rank pending automatic speech recognition (ASR) requests. [Figure 3C] FIG. schematically shows an exemplary voice query service quality (QoS) manager configured to continuously re-rank pending automatic speech recognition (ASR) requests. [Figure 4] FIG. schematically shows the QoS manager of FIG. 1 that provides on-device processing instructions to the user device. [Figure 5] FIG. is a flowchart showing an example arrangement of operations of a method for processing a pending automatic speech recognition (ASR) request in a query processing stack based on the availability of processing in the query processing stack. [Figure 6] FIG. is a flowchart of an example arrangement of operations for a method of executing on-device processing instructions when a server-based query processing stack is overloaded. [Figure 7] Schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein.

Best Mode for Carrying Out the Invention

[0021] Like reference symbols in the various drawings indicate like elements. Referring to Figure 1, in some implementations, system 100 includes user devices 200, 200a to 200n, each associated with users 10, 10a to 10n, who can communicate with a remote system 140 via a network 130. Some users 10 may be associated with multiple user devices 200, and / or some user devices 200 may be associated with multiple users 10 (e.g., family members in a home). The remote system 140 may be a distributed system (e.g., a cloud computing environment) with scalable / elastic resources 142. Resources 142 include computing resources 144 (e.g., data processing hardware) and / or storage resources 146 (e.g., memory hardware). In some implementations, the remote system 140 includes a voice query quality of service (QoS) manager 300 and a query processing stack 700, 700a. The query processing stack 700a may be called a query processing backend, server-based, or backend-side query processing stack 700a. The Quality of Service (QoS) manager 300 is configured to receive an Automatic Speech Recognition (ASR) request 102 from the user device 200, which includes audio data 103 and content metadata 110 associated with the voice input 104. The QoS manager 300 may then assign a priority score 311 to each ASR request 102, at least based on the content metadata 110. Using the priority score 311 assigned to each received ASR request 102, and based on the processing availability of the query processing stack 700, the QoS manager 300 assigns a ranking 312 to each ASR request 102, and provides the ASR requests 102 to the query processing stack 700 in order of the highest to lowest ranking 312. That is, an ASR request with a ranking 312 associated with a higher priority score 311 is processed by the query processing stack 700 before an ASR request 102 with a ranking 312 associated with a lower priority score 311.

[0022] The user device 200 includes data processing hardware 204 and memory hardware 206. The user device 200 may also include an audio capture device (e.g., a microphone) for capturing audio input 104 from the user 10 and converting it into audio data 103 (e.g., an electrical signal). In some implementations, the data processing hardware 204 is configured to run query processing stacks 700, 700b instead of, or in place of, the query processing stack 700a that runs on the remote system 140. For example, query processing stack 700b may include some of the same or different components as query processing stack 700a that runs on the remote system 140, including, but not limited to, at least one of the on-device automatic speech recognition (ASR) module 220n (Figure 2), interpreter module 720, or text-to-speech (TTS) module 730. In some examples, the user device 200 runs an on-device automatic speech recognition ASR module 220n (Figure 2) for rapidly generating low-fidelity transcriptions without requiring a network connection. Meanwhile, the remote system 140 runs a server-based automatic speech recognition ASR module 710 that can generate high-fidelity transcriptions at the expense of higher latency compared to on-device low-fidelity transcriptions. The user device 200 can be any computing device capable of communicating with the remote system 140 via the network 130. The user device 200 includes, but is not limited to, desktop computing devices and mobile computing devices such as laptops, smart speakers, smart displays, tablets, smartphones, and wearable computing devices (e.g., headsets and / or watches).

[0023] In the illustrated example, user 10 may provide voice input 104 to user device 200 by speaking, and user device 200 may capture voice input 104 and convert it into audio data 103. User device 200 may then send an Auto Speech Recognition ASR request 102, comprising the audio data 103 associated with voice input 104 and content metadata 110, to a Quality of Service QoS manager 300a running on remote system 140. Here, the Quality of Service QoS manager 300 assigns a ranking 312 to the Auto Speech Recognition ASR request 102 based at least on the content metadata 110, and sends the Auto Speech Recognition ASR request 102 to the query processing stack 700 in order from the highest ranking 312 to the lowest ranking 312. User device 200 may optionally run a query processing stack 700b on-device to process the Auto Speech Recognition ASR request 102, or a portion of the Auto Speech Recognition ASR request 102. For example, if network communication is down or unavailable, the user device 200 may locally run the query processing stack 700b to process the Automated Speech Recognition ASR request 102. In some examples, the user device 200 may locally run the query processing stack 700b to process the Automated Speech Recognition ASR request 102 if it is determined that the Automated Speech Recognition ASR request 102 is time-sensitive, for example, if it is an Automated Speech Recognition ASR request 102 that sets a one-minute timer. Implementations of this specification further include the user device 200 locally running the query processing stack 700b in a scenario where a Quality of Service (QoS) manager 300a running on the remote system 140 determines / detects that the server-based query processing stack 700 is overloaded and / or is currently experiencing a large spike in Automated Speech Recognition ASR requests 102 to be processed.

[0024] The content metadata 110 acquired by the Quality of Service (QoS) manager 300 may include at least one of the following: a speech recognition category associated with the voice input 104, an application running on the user device 200 associated with the user 10, the user 10's location at the time the voice input 104 was received, a user identifier that identifies the user 10, demographic information associated with the user 10, whether the user 10 is signed in to the user device 200, whether the user 10 is currently part of a multi-turn interaction with the system 100, localization information that determines the distance and location of the user 10 to the user device 200, or ad-likeliness information (Figure 2) obtained by performing additional classification (addclassifier) ​​on the user device 200 that leverages data / information from multiple sources 220, 220a to 200n. Specifically, as will be explained in more detail below with reference to Figure 2, the user device 200 is configured to generate content metadata 110 associated with each audio input 104 and to provide an associated Auto Speech Recognition ASR request 102, which includes the content metadata 110 and the audio input 104 (e.g., audio data 103), to a Quality of Service (QoS) manager 300a running on the remote system 140.

[0025] The Quality of Service (QoS) manager 300 comprises a ranker 310 and a pre-processing backlog 320. The ranker 310 is configured to analyze the content metadata 110 contained in the Automated Speech Recognition (ASR) request 102 to determine the likelihood that the ASR request 102 is genuine, and to assign a corresponding ranking 312 to the ASR request 102 based on its likelihood of being genuine. As used herein, a “genuine” ASR request 102 refers to an ASR request 102 having a voice input 104 spoken by an actual user 10 and instructed to the user device 200 as a voice query for processing by the query processing stack 700. In some examples, the Quality of Service (QoS) manager 300 determines a priority score 311 corresponding to each ASR request 102 received, based on the content metadata 110 contained in the ASR request 102. Here, the Quality of Service (QoS) manager 300 may assign a ranking 312 to each Auto Speech Recognition (ASR) request 102 based on the corresponding priority score 311. Thus, the priority score 311 based on the content metadata 110 of each received Auto Speech Recognition (ASR) request 102 may indicate the possibility that the Auto Speech Recognition (ASR) request is genuine. For example, the content metadata 110 may indicate that the Auto Speech Recognition (ASR) request 102 came from a source other than any of the users 10, 10a to 10n, such as from a non-human source (e.g., television or radio) 12 during a television / radio commercial, announcement, or other program. Therefore, ranker 310 determines a low priority score 311 for automatic speech recognition ASR request 102 because the content metadata 110 indicates that the automatic speech recognition ASR request 102 is unlikely to be genuine, and thereby ranker 310 assigns a low ranking 312 to the automatic speech recognition ASR request 102 based on the low likelihood that the request 102 is genuine.In another example, content metadata 110 indicates that the Auto Speech Recognition ASR request 102 came from one of users 10, 10a to 10n, and therefore ranker 310 may determine a high priority score 311 for the Auto Speech Recognition ASR request 102 and assign a high ranking 312 to the Auto Speech Recognition ASR request 102 based on the likelihood that the Auto Speech Recognition ASR request 102 is genuine. In addition or alternatively, ranker 310 may analyze audio data 103 and / or any other appropriate information in addition to or instead of content metadata 110 to determine the likelihood that the Auto Speech Recognition ASR request 102 is genuine.

[0026] After determining the priority score 311 and assigning a ranking 312 to each Auto Speech Recognition ASR request 102, ranker 310 caches the pending Auto Speech Recognition ASR requests 102 in a pre-processing backlog 320, each having a corresponding priority score 311 (previously determined by ranker 310). Here, the pending Auto Speech Recognition ASR requests 102 in the pre-processing backlog 320 are ranked in order of priority score 311 so that the query processing stack 700 processes the pending Auto Speech Recognition ASR requests 102 associated with a higher ranking 312 before processing the pending Auto Speech Recognition ASR requests 102 associated with a lower ranking 312.

[0027] The ranker 310 continuously or semi-continuously receives a list 322 of pending Auto Speech Recognition ASR requests 102 in the preprocessing backlog 320, and reranks the pending Auto Speech Recognition ASR requests 102 when they are received by the Quality of Service (QoS) manager 300 over the network 130. For example, the ranker 310 may determine that the new Auto Speech Recognition ASR request 102 has a corresponding priority score 311 that is higher than any priority score 311 in the list 322 of pending Auto Speech Recognition ASR requests 102 currently cached in the preprocessing backlog 320 while it is waiting to be processed by the query processing stack 700. The backlog 320 may provide the ranker 310 with a list 322 of pending Auto Speech Recognition ASR requests 102 as feedback, and the ranker 310 may assign the new Auto Speech Recognition ASR request 102 a higher ranking 312 than the ranking 312 in the list 322 of pending Auto Speech Recognition ASR requests 102, so that the new Auto Speech Recognition ASR request 102 takes precedence over other pending Auto Speech Recognition ASR requests 102 in the backlog 320 for processing by the query processing stack 700. In some implementations, the ranker 310 drops at least one of the pending Auto Speech Recognition ASR requests 102 in the list 322. For example, the ranker 310 may determine that the number of pending Auto Speech Recognition ASR requests 102 in the backlog 320 exceeds a predetermined threshold. In order to free up space in the backlog 320 and / or increase bandwidth for new Automated Speech Recognition ASR requests 102 with a higher ranking 312, ranker 310 may drop at least one of the pending Automated Speech Recognition ASR requests 102 associated with a lower ranking 312. Furthermore or alternatively, ranker 310 may drop at least one of the pending Automated Speech Recognition ASR requests 102 in the list 322 of pending Automated Speech Recognition ASR requests 102 that have timed out, i.e., have been held in the backlog 320 for a period exceeding a predetermined threshold.

[0028] Each pending Auto Speech Recognition ASR request 102 in the preprocessing backlog 320 awaits processing by the query processing stack 700 based on the ranking 312 assigned to the pending Auto Speech Recognition ASR request 102. For example, the highest-ranked Auto Speech Recognition ASR request 102 (e.g., an Auto Speech Recognition ASR request 102 associated with the highest priority score 311) is processed before the second-highest-ranked Auto Speech Recognition ASR request 102 (e.g., an Auto Speech Recognition ASR request 102 associated with the next highest priority score 311), the second-highest-ranked Auto Speech Recognition ASR request 102 is processed before the third-highest-ranked Auto Speech Recognition ASR request 102, and so on. The backlog 320 continuously or semi-continuously communicates the list 322 to the ranker 310 to facilitate the re-ranking of pending Auto Speech Recognition ASR requests 102.

[0029] The query processing stacks 700 and 700a on the remote system 140 receive each Auto Speech Recognition (ASR) request 102 that has not been dropped or timed out and includes audio data 103 and content metadata 110 associated with the voice input 104, in descending order of ranking 312 from the Quality of Service (QoS) manager 300. The query processing stack 700 comprises at least an Auto Speech Recognition (ASR) module 710, an interpreter module 720, or a Text-to-Speech (TTS) module 730. The Auto Speech Recognition (ASR) module 710 may perform various operations on the Auto Speech Recognition (ASR) request 102, such as processing, noise modeling, acoustic modeling, language modeling, and annotation, to generate a speech recognition result (e.g., a transcription) for the voice input 104. The Auto Speech Recognition (ASR) module 710 sends this speech recognition result to the interpreter 720 to determine the intent of the Auto Speech Recognition (ASR) request 102 and generate a response. For example, an automatic speech recognition (ASR) request 102 requesting the current time might be satisfied by the query processing stack 700 determining the current time in the user 10's time zone and generating a response. The text-to-speech (TTS) module 730 may convert this response from text to speech and output the response in speech format to the user device 200, which is then output to the user 10 as synthesized speech, for example, through the speaker of the user device 200. Alternatively, the response may be output to the user device 200 in text format and sent to the user 10, for example, through the screen of the user device 200. In other implementations, the user device 200 may receive the response from the query processing stack 700a in the form of text or other data and use an on-device text-to-speech (TTS) module to convert the response to speech.

[0030] Figure 2 shows an example of a user device 200 that captures a voice input 104, collects information / data from one or more of several sources 220, 220a-220n, and generates content metadata 110 associated with the voice input 104 for inclusion in an automatic speech recognition ASR request 102 sent to a Quality of Service (QoS) manager 300. The user device 200 includes a screen 212 and runs a graphical user interface 214 for display on the screen 212. As shown in Figure 2, the voice input 104 responds to a hotword ("Hey Google") and subsequent voice query directed to the user device 200, for example, to a calendar application 220e running on the user device 200, in order to add an event (SkipFikany's birthday) on a given day (December 8) to the calendar application 200e. In some implementations, the user device 200 presents a transcription of the voice input 104 in the voice search / command window 216 displayed on the graphical user interface 214.

[0031] After the user device 200 receives the voice input 104, the user device 200 utilizes the information / data associated with the voice input 104 from multiple sources 220 and runs a signal generator 218 (on the data processing hardware 204) to generate content metadata 110 that can be used to provide context or hints about the voice input 104 for use by the Quality of Service QoS manager 300, without requiring the Quality of Service QoS manager 300 or query processing stack 700 running on the remote system 140 to start processing the automatic speech recognition ASR request 102. The content metadata 110 associated with the voice input 104 and generated by the signal generator 218 includes a login index indicating whether user 10 associated with user device 200 is logged into user device 200, a speaker identification score for voice input 104 indicating the likelihood that the voice input matches a speaker profile associated with user device 200, a broadcast voice score for voice input 104 indicating the likelihood that voice input 104 corresponds to broadcast or synthesized voice output from a non-human source (e.g., television), a hotword confidence score indicating the likelihood that one or more terms detected in voice input 104 prior to the voice query correspond to a hotword, and the user device 200 and query processing system The system includes at least one of the following: an activity indicator indicating whether a multi-turn interaction is in progress with Tack 700 (i.e., the query processing backend); an audio signal quality metric for the audio input 104; a spatial localization score indicating the distance and location of user 10 to user device 200; a transcript of the audio input 104 generated by the on-device automatic speech recognition ASR module 220n (e.g., a low-fidelity transcript); a user device behavior signal indicating the current behavior of user device 200; or an environmental condition signal indicating the current environmental conditions for user device 200.The multiple sources 220 include, but are not limited to, speaker classifiers 220, 220a, broadcast speech classifiers 220, 200b, hotword detectors 220, 220c, session activity logs 220, 220d, one or more applications 220, 220e running on the user device 200, speech quality scorers 220, 220f, one or more sensors 220, 220g on the user device 200, and on-device automatic speech recognition ASR 220, 220n. As will become apparent, the signal generator 218 is configured to leverage data / information from any of the sources 220 and other associated sources to generate content metadata 110 associated with the speech input 104. Furthermore, data / information obtained from two or more sources 220 is more overlapping, enabling the signal generator 218 to generate more robust content metadata 110.

[0032] The speaker classifier 220a may compare the audio data 103 of the speech input 104 (e.g., acoustic features associated with pronunciation, timing, etc.) with the acoustic features of a speaker profile associated with one or more users 10 of the user device 200. For example, the speaker profile may be learned and / or generated during the speaker enrollment process by one or more users 10 in a household authorized to use the user device 200, such as a smart speaker. If the audio data 103 of the speech input 104 matches the acoustic features of a speaker profile of one or more users 10 associated with the user device 200, the speaker classifier 220a may output a high speaker identification score indicating that the speech input 104 was likely spoken by a user 10 associated with the user device 200. In this example, the signal generator 218 may use the high speaker identification score to provide content metadata 110 indicating that the corresponding automatic speech recognition ASR request 102 is likely to be genuine. On the other hand, if the audio data 103 of the voice input 104 does not match the acoustic characteristics of the speaker profile of user 10 associated with the user device 200, the speaker classifier 220a may provide a low speaker identification score. Therefore, the speaker identification score may correspond to a confidence value or probability that the audio data 103 matches a known speaker profile.

[0033] The broadcast speech classifier 220b may analyze the audio data 103 of the speech input 104 and provide a broadcast speech score for the speech input indicating that the speech input 104 is likely to correspond to broadcast or synthesized speech output from a non-human source 12, such as a television, radio, computer, or other speech output device capable of outputting broadcast and / or synthesized speech. As used herein, broadcast speech (broadcast speech) refers to speech spoken by a human (e.g., a news anchor, actor, radio personality, etc.) but corresponding to speech content emitted / broadcast from a non-human source 12 during a media event such as a commercial, radio program, television program, and / or film. Synthesized speech (synthesized speech), on the other hand, refers to non-human speech produced by, for example, a text-to-speech (TTS) system. The broadcast audio classifier 220b may be capable of detecting watermarks or other features that may be added to audio content emitted from / broadcast from a non-human source 12, and / or may be capable of self-learning to distinguish between audio output from an actual human being in close proximity to the user device 200 and audio output from a non-human source 12 being broadcast during synthesized speech or a media event.

[0034] In some cases, the broadcast speech classifier 220b also analyzes the speaker identification score provided by the speaker classifier 220a as a basis for determining the broadcast speech score. For example, a high speaker identification score output from the speaker classifier strongly indicates that the speech input 104 is not coming from a non-human source 12, while a low speaker identification score output from the speaker classifier 220a opens up the possibility that the speech input 104 originated from a non-human source 12.

[0035] The hotword detector 220c calculates a hotword confidence score for the voice input 104 and compares the hotword confidence score to a hotword confidence score threshold. The hotword confidence score threshold represents the hotword confidence score that, when detected by the hotword detector 220c, triggers the user device 200 to wake up from sleep mode, capture the remainder of the voice input 104 corresponding to the voice query, and generate an automatic speech recognition ASR request 102 sent to the query processing stack 700. In an example where user 10 clearly speaks the specified hotword "OkGoogle" and is near the user device 200, the hotword confidence score may be high (e.g., greater than 0.9). In some cases, a user 10 near the user device 200 might speak a phrase that sounds similar to the designated hotword "OkGoogle," such as "Ok poodle," which would result in a lower confidence score (e.g., 0.7), but still meet the hotword confidence score threshold (e.g., 0.68). Furthermore, the hotword confidence score may decrease if the speaker is far from the user device 200 or if their speech is unclear. Therefore, providing content metadata 110 with a hotword confidence score for the speech input 104 (i.e., the initial portion of the speech input 104) may trigger the hotword confidence score threshold to be met and wake up the user device 200, but the hotword confidence score may be low enough to indicate that the speaker was far away and / or spoke other phrases that sounded similar to the hotword, and therefore did not intend to call the user device 200. Therefore, the hotword confidence score can contribute to content metadata 110 that indicates whether or not the automatic speech recognition ASR request is likely to be genuine.

[0036] The session activity log 220d may provide a log of interactions between the user device 200 and the query processing stack 700. For example, log 220d may include a timestamp of a recent Automated Speech Recognition (ASR) request 102 sent to the query processing stack 700 and a timestamp of a corresponding Text-to-Speech (TTS) response returned from the query processing stack 700. The signal generator 218 may access the session activity log 220d to determine content metadata 110 indicating whether a multi-turn interaction is in progress between the user device 200 and the query processing stack 700. For example, if the Automated Speech Recognition (ASR) request 102 is a voice query asking "Should I bring an umbrella?", the session activity log 220d may indicate that user device 200 had just provided a previous voice query asking "What will the temperature be in Detroit tomorrow morning?" and received a text-to-speech (TTS) response from the query processing stack 700 stating "The temperature in Detroit tomorrow will be 62 degrees Fahrenheit at 7:00 a.m. and is expected to reach 73 degrees Fahrenheit at 11:00 a.m.". Therefore, the signal generator 218 may generate content metadata 110 to include in the Automated Speech Recognition (ASR) request 102, indicating that user 10 is currently engaged in an ongoing discussion with user device 200 and therefore the Automated Speech Recognition (ASR) request 102 is likely to be genuine. In another example, user 10 may have recently submitted an Automated Speech Recognition (ASR) request 102 asking user device 200 to search for local restaurants. If a subsequent automated speech recognition (ASR) request 102 is submitted requesting the user device 200 to make a reservation at one of the local restaurants, the session activity log 220d may indicate that a multi-turn interaction is in progress between the user device 200 and the query processing stack 700. Furthermore, the signal generator 218 may further determine that a specific application 220e, such as a digital voice assistant, is currently running on the user device 200 in order to provide an activity indicator that an ongoing multi-turn interaction is in progress between the user device 200 and the query processing stack.In some examples, the digital voice assistant application 220e logs session activity in the session activity log 220d for use by the signal generator 218.

[0037] The voice quality scorer 220f may be configured to determine the voice quality score of the voice input. In some examples, the voice quality scorer 220f measures the loudness (e.g., decibels) of the audio data 103 associated with the voice input 104. The measured loudness may correspond to the portion of the audio data 103 corresponding to a hotword detected by the hotword detector 220c, the portion of the audio data 103 corresponding to a voice query following the hotword, or the entire audio data 103 captured by the user device 200. The voice quality score of the voice input 104 may further indicate the level of background noise present in the audio data 103. Thus, the voice quality score may simply be a confidence score of the voice quality of the voice input 104, i.e., an indication of how well the voice input 104 was captured by the microphone of the user device 200.

[0038] The signal generator 218 may, for example, utilize data / information from multiple sources 220 to determine content metadata 110 with a spatial localization score for the voice input 104, indicating the relative distance and location of user 10 to user device 200. In some examples, the hotword confidence score from the hotword detector 220c and / or the voice quality score from the voice quality scorer 220f may contribute to the spatial localization score. For example, a low hotword confidence score and / or a low voice quality score may be used to indicate that the source that spoke the voice input 104 (e.g., user 10) is not near user device 200. Furthermore, or alternatively, the spatial localization score may be based on data / information collected by one or more sensors 220, 200g of user device 200. The sensor 220g may include, but is not limited to, at least one of the following: a light detection sensor, an accelerometer, a microphone, a gyroscope, a magnetometer, a proximity sensor, a touchscreen sensor, a barometer, or a Global Positioning System (GPS) sensor. For example, if the user device 200 is equipped with a pair of smart headphones, one or more of the sensors 220g may be used to determine whether user 10 is currently wearing the headphones or whether the headphones are not being worn and therefore not being used. Whether user 10 is wearing smart headphones may be determined by one of the sensors 220g, for example, a proximity sensor, an accelerometer, etc. In this example, the signal generator 218 may generate a high spatial localization score (e.g., a binary value of 1) if user 10 is wearing smart headphones, and a low spatial localization score (e.g., a binary value of 0) if user 10 is not wearing smart headphones. As another example, one of the sensors 220g may be able to detect the light energy in the environment of the user device 200 located in a room in a house to indicate whether the room is dark. For example, a light energy level below a predetermined threshold may indicate that the user device 200 is located in a dark room.Thus, the signal generator 218 may determine a low spatial localization score to indicate that user 10 is unlikely to be in the same room as user device 200 when user device 200 is located in a dark room in the evening (for example, by further accessing the current time). Conversely, when determining the spatial localization score, the signal generator 218 may utilize information from session activity log 220d indicating that there was a recent Automated Speech Recognition ASR request 102 asking user device 200 to turn off the room lights, and therefore the spatial localization score may instead reflect that user 10 is in the dark room. In yet another example, user device 200, which is part of a vehicle's infotainment system, may use sensor 220g to determine whether user 10 is in the vehicle's cabin, whether the vehicle is running, etc. In this example, the fact that user device 200 has captured a voice input 104, but no one is in the vehicle's cabin and the vehicle is not running, may indicate that the voice input 104 is directed at another user device 200 or is background noise accidentally captured by the infotainment system.

[0039] Content metadata 110, which includes a login metric, may determine whether user 10 is logged into user device 200, for example, whether user 10 has entered the appropriate authentication information to access / unlock user device 200. The presence of user 10 logged into user device 200 may increase the likelihood that the automatic speech recognition ASR request 102 is genuine. Furthermore, the login metric may indicate whether user 10 is logged into a specific application 220e running on user device 200. For example, a signal generator 218 may generate a login metric by accessing a calendar application 220e running on user device 200 of logged-in user 10 and determining that logged-in user 10 has a meeting in 5 minutes. In this example, the login metric may provide content metadata 110 indicating that it is important to process the automatic speech recognition ASR request 102 for voice input 104 quickly because logged-in user 10 may need information for the meeting (e.g., directions, notes, etc.). As another example, the logged-in user 10 may be the homeowner, and the signal generator 218 may access the calendar application 220e to determine if the logged-in user 10 has appointments at a location other than the logged-in user 10's home and generate a login metric. If the user device 200 is, for example, a smart speaker located in the logged-in user 10's home, and the user device 200 is triggered when the calendar application 220e detects a hotword spoken at the same time as an appointment, the login metric may provide content metadata 110 indicating that the logged-in user 10 likely did not provide voice input 104, thereby rendering the corresponding Auto Speech Recognition ASR request 102 unlikely to be genuine. The user 10 may be required to explicitly consent to providing login information, application usage, and location information.

[0040] The signal generator 218 may, for example, utilize data / information from multiple sources 220 to determine the content metadata 110 containing the user device behavior signal, in order to indicate how likely it is that user 10 is interacting with user device 200 at the time the voice input 104 is captured. For example, information from one or more sensors 220g may indicate whether user device 200 is upside down or face down (e.g., if user device is a smartphone or tablet), or whether user device 200 is in a drawer / wallet. In these scenarios, if user device 200 is likely to have been triggered in error, it is determined that the corresponding automatic speech recognition ASR request 102 is unlikely to be genuine. Information from sensors 220g may further include the location of user device (e.g., obtained from GPS sensor 220g) and / or the signal strength of user device 200's network connection. For example, a user device behavior signal indicating that the current signal strength of user device 200 is low and that it is in a location notorious for having (e.g., based on past knowledge) spot cell coverage (cell range) is useful for the Quality of Service (QoS) manager 300 in prioritizing corresponding Auto Speech Recognition (ASR) requests 102, because even if the Auto Speech Recognition (ASR) request 102 is genuine, user device 200 is unlikely to receive the corresponding response (e.g., Auto Speech Recognition (ASR) output and / or Text-to-Speech (TTS) response) processed by the query processing stack. In this situation, when the query processing stack 700 is experiencing a high traffic spike, it may be beneficial to process other pending Auto Speech Recognition (ASR) requests 102 first, as these would generate corresponding responses that are more likely to be successfully returned to each user device 200.

[0041] The signal generator 218 may, for example, utilize data / information from multiple sources 220 to determine content metadata 110 that includes environmental condition signals in order to evaluate and analyze the circumstances surrounding the user device 200. Specifically, the environmental condition signals may utilize one or more sensors 220g to determine the environmental conditions surrounding the user device 200. For example, the environmental condition signals may indicate whether multiple user devices 200 are present in the vicinity of the user device 200, the conditions of the network to which the user device 200 is connected (e.g., whether the network is overloaded), the GPS coordinates of the user device 200, whether the user device 200 is outdoors, whether it is currently moving, or whether it is approaching an area with poor cellular or data reception.

[0042] As noted above, the user device 200 may have the capability to run an on-device automatic speech recognition (ASR) module 220n (e.g., local query processing stack 700b) for rapidly generating a low-fidelity transcription without requiring a network connection. Advantageously, the content metadata 110 generated by the signal generator 218 may include a low-fidelity transcription of the voice input 104 generated by the on-device automatic speech recognition (ASR) module 220n to provide the Quality of Service (QoS) manager 300 with potentially associated information or details regarding the automatic speech recognition (ASR) request 102. Here, the low-fidelity transcription can reveal that the automatic speech recognition (ASR) request 102 includes a time-sensitive voice query (e.g., a command to set a timer for one minute), thereby informing the Quality of Service (QoS) manager 300 to assign a high priority score 311 to the automatic speech recognition (ASR) request 102 so that it may be processed immediately. In this same scenario, if the query processing stack 700a is overloaded and has full processing capacity when the Automated Speech Recognition ASR request 102 is cached in the backlog 320, the query processing stack 700a may not be able to process the Automated Speech Recognition ASR request 102 immediately (for example, by sending a command to set a 1-minute timer) and may simply drop the Automated Speech Recognition ASR request 102 and optionally notify user 10 that the request cannot be completed at that time. This is preferable for user 10, as setting a short timer would be of little use after a considerable amount of time has passed since the Automated Speech Recognition ASR request 102 was provided. However, if the command sets a timer for a longer duration, such as 10 minutes, it may be acceptable for the Automated Speech Recognition ASR request 102 to remain pending and cached in the preprocessing backlog 320 until the query processing stack 700a is able to process it. This allows the command to set the timer to compensate for the latency caused by increased traffic while the automatic speech recognition ASR request 102 is pending in the pre-processing backlog 320.

[0043] In the example shown, after the signal generator 218 compiles and generates content metadata 110 associated with the voice input 104, the user device 200 embeds the content metadata 110, along with the corresponding voice input 104 and audio data 103, for inclusion in the automatic speech recognition ASR request 102. The user device 200 then sends the automatic speech recognition ASR request 102 to the quality of service (QoS) manager 300.

[0044] Figures 3A to 3C are schematic diagrams illustrating how the Voice Query Quality of Service (QoS) manager 300 assigns a ranking 312 to the Auto Speech Recognition (ASR) requests 102 received from the user device 200, and provides the ASR requests 102 to the query processing stack 700 in descending order of ranking 312, based on the processing availability of the query processing stack 700. The query processing stack 700 may comprise, but is not limited to, at least one of the Auto Speech Recognition (ASR) module 710 (Figure 1), the interpreter module 720 (Figure 1), or the Text-to-Speech (TTS) module 730 (Figure 1). In the illustrated example, the query processing stack 700 may currently be experiencing a large traffic spike of ASR requests 102 that is causing an overload on the query processing stack 700. For example, a television commercial aired during a major media event (e.g., the Super Bowl) may contain spoken hotwords that, when output from a user's household television (e.g., a non-human source 12), trigger and generate a false automatic speech recognition (ASR) request 102 in the voice-enabled user device 200 of that household, which includes audio data 103 captured after the detection of the hotwords output from the television.

[0045] Referring to Figure 3A, the pre-processing backlog 320 includes pending Auto Speech Recognition ASR requests A102a, B102b, and C102c that are waiting to be processed by the query processing stack 700 when the ranker 310 of the voice query quality of service (QoS) manager 300 receives a new Auto Speech Recognition ASR request D102d from the user device 200. The ranker 310 may determine the priority score 311 corresponding to the new Auto Speech Recognition ASR request D102d based on the content metadata 110 associated with the new Auto Speech Recognition ASR request D102d. When the new Automated Speech Recognition (ASR) request D102d is received, the pending Automated Speech Recognition (ASR) requests 102a, 102b, and 102c in the preprocessing backlog 320 are ranked in order of priority score 311, such that Automated Speech Recognition (ASR) request A102a is associated with the highest ranking 312, Automated Speech Recognition (ASR) request C102b is associated with the next highest ranking 312, and Automated Speech Recognition (ASR) request B102b is associated with the lowest ranking 312. Therefore, after the Quality of Service (QoS) manager 300 receives Automated Speech Recognition (ASR) request B102b, Automated Speech Recognition (ASR) request C102c may have been received by the Quality of Service (QoS) manager 300 from the corresponding user device 200, but ranker 310 assigns a rank 312 higher to Automated Speech Recognition (ASR) request C102c than the rank 312 assigned to Automated Speech Recognition (ASR) request B102b. The preprocessing backlog 320 may contain thousands to millions of pending Auto Speech Recognition ASR requests 102, ranked in order of priority score 311, waiting to be processed by the query processing stack 700. Continuing to refer to Figure 3A, the preprocessing backlog 320 provides the ranker 310 with a list 322 of pending Auto Speech Recognition ASR requests A102a, C102c, and B102b, which the ranker 310 then re-ranks along with new Auto Speech Recognition ASR requests D102d based on their priority scores.

[0046] In some implementations, ranker 310 refuses to allow the query processing stack 700 (e.g., backend-side Auto Speech Recognition ASR module 710) to process pending Auto Speech Recognition ASR requests 102 present in the preprocessing backlog 320 for a period of time that meets the timeout threshold. Figure 3B shows how ranker 310 refuses to allow pending Auto Speech Recognition ASR request B102b to be processed by the query processing stack 700 because the pending Auto Speech Recognition ASR request B102b meets the timeout threshold. For example, pending Auto Speech Recognition ASR request B102b had a low priority score 311, resulting in it remaining at the bottom of list 322. Therefore, even if a new Auto Speech Recognition ASR request 102 was received later in time, Auto Speech Recognition ASR request B102b might not have been processed. Consequently, Auto Speech Recognition ASR request B102b is dropped from the preprocessing backlog 320. Figure 3B further shows how ranker 310 determines that the new Automated Speech Recognition ASR request D102d has a priority score 311 that is higher than the priority score 311 of the pending Automated Speech Recognition ASR request C102c but lower than the priority score of the pending Automated Speech Recognition ASR request A102a. In this way, ranker 310 provides the preprocessing backlog 320 with a reranked list 322 of pending Automated Speech Recognition ASR requests A102a, D102d, and C102c, such that Automated Speech Recognition ASR request A102a is still associated with the highest ranking 312, Automated Speech Recognition ASR request D102d is associated with the next highest ranking 312, and Automated Speech Recognition ASR request C102c is associated with the lowest ranking 312. Therefore, the new Auto Speech Recognition ASR request D102d, which is ranked higher than Auto Speech Recognition ASR request C102c in the list 322 of pending Auto Speech Recognition ASR requests 102, will take precedence over Auto Speech Recognition ASR request C102c in the backlog 320 for processing by the query processing stack 700. However, Auto Speech Recognition ASR request A102a will take precedence over the new Auto Speech Recognition ASR request D102d for processing by the query processing stack 700.

[0047] Referring to Figure 3C, the query processing stack 700 is available to process the next pending Auto Speech Recognition ASR request 102 cached in the preprocessing backlog 320. Since Auto Speech Recognition ASR request A102a is associated with the highest rank 312 in the list 322 of pending Auto Speech Recognition ASR requests 102 awaiting processing in the preprocessing backlog 320, the preprocessing backlog 320 provides Auto Speech Recognition ASR request A102a to the query processing stack 700 for processing. Thus, Auto Speech Recognition ASR request A102a is removed from the backlog 320 and the list 322 of pending Auto Speech Recognition ASR requests 102.

[0048] Simultaneously with the provision of the Automated Speech Recognition (ASR) request A102a to the query processing stack 700 for processing, the ranker 310 of the Voice Query Quality of Service (QoS) Manager 300 receives the new Automated Speech Recognition (ASR) request E102e from the corresponding user device 200 and, as feedback, receives a list 322 of pending Automated Speech Recognition (ASR) requests D102d and C102c from the preprocessing backlog 320. Here, the ranker 310 determines the priority score 311 corresponding to the new Automated Speech Recognition (ASR) request E102e based on the content metadata 110 associated with the new Automated Speech Recognition (ASR) request E102e, and may re-rank the pending Automated Speech Recognition (ASR) requests D102d and C102c together with the new Automated Speech Recognition (ASR) request E102e based on the priority score. When a new Automated Speech Recognition (ASR) request 102 is received, the sequential reprioritization of pending ASR requests 102 in the preprocessing backlog 320 is an iterative process and depends on the processing availability of the query processing stack 700.

[0049] Figure 4 shows a schematic diagram (400) of the Quality of Service (QoS) manager 300 communicating an on-device processing command 420 to a user device 200, which allows the user device 200 to determine whether to send an Auto Speech Recognition (ASR) request 102 to the query processing stack 700 (e.g., the query processing backend) for processing if a high-load condition exists in the query processing stack. A high-load condition may indicate that the query processing stack 700a is overloaded due to a large traffic spike caused by a large number of ASR requests 102 sent to the query processing stack 700 for processing. The Quality of Service (QoS) manager 300 may provide the on-device processing command 420 to all or a selected subset of the group of voice-enabled user devices 200 whose processing is handled by the query processing stack 700a. A user device 200 associated with one device type (e.g., a smart speaker) may receive a different ASR request command 420 than a user device 200 associated with another device type (e.g., a smartphone). The on-device processing instruction 420 may provide one or more criteria for processing at least a portion of any new audio inputs 104 captured by the user device 200 on-device (for example, in the on-device query processing stack 700b) if the user device 200 determines that the query processing stack 700a is overloaded.

[0050] The on-device processing instruction 420 may provide criteria for sending the Auto Speech Recognition ASR request 102 to the query processing stack 700a when the query processing stack 700a is overloaded, based on the content metadata 110 associated with the Auto Speech Recognition ASR request 102. In some implementations, the on-device processing instruction 420 provides one or more thresholds that the corresponding portion of the content metadata 110 must satisfy in order for the user device 200 to send the Auto Speech Recognition ASR request 102 to the query processing stack 700a during high-load conditions. For example, the on-device processing instruction 420 may provide a hotword confidence score threshold that the hotword confidence score must satisfy, and / or a speech quality score threshold that the speech quality score of the speech input 104 must satisfy. The user device 200 typically applies default thresholds, but the thresholds provided by the on-device processing instruction 420 may be more conservative so that only genuine (or highly impactful to the user) Auto Speech Recognition ASR requests 102 with high confidence are sent to the query processing stack 700a for processing. For example, user device 200 may send an Auto Speech Recognition ASR request 102, which is typically associated with a hotword confidence score greater than 0.68, to the query processing stack 700a for processing. However, if the query processing stack 700a is overloaded, the on-device processing instruction 420 may indicate that the Auto Speech Recognition ASR request 102 must be associated with a hotword confidence score of at least 0.8 in order to be sent to the query processing stack 700a for processing. The on-device processing instruction 420 may further instruct user device 200 to drop the Auto Speech Recognition ASR request 102 if at least one of the thresholds is unsatisfied. The Quality of Service (QoS) manager 300 may send the on-device processing instruction 420 on the fly whenever a high-load condition exists, or it may send the on-device processing instruction 420 to user device 200 at any time so that user device 200 can apply / execute the on-device processing instruction 420 when a high-load condition occurs at a later time.Similarly, a speech quality threshold can be provided for use by the user device 200 when filtering out automatic speech recognition ASR requests 102 that have a speech quality that does not meet the threshold defined by the on-device processing instruction 420, when the query processing stack 700a is overloaded.

[0051] In the example shown, user device 200 captures the voice input 104 and generates content metadata 110 associated with the voice input 104. For example, user device 200 runs a signal generator 218 configured to generate content metadata 110 based on information / data obtained from one or more sources 220. The content metadata 110 generated by user device 200 is as described above with reference to Figure 2. Before sending (or processing locally) the corresponding Auto Speech Recognition ASR request 102 with the voice input 104 and associated content metadata 110, user device 200 may determine whether a high-load condition exists in the query processing stack 700a. In some examples, user device 200 receives an on-the-fly notification 410 (e.g., an overload condition status notification) from the Quality of Service (QoS) manager 300 indicating the presence of an overload condition in the query processing stack 700a. Alternatively, user device 200 may receive a notification 410 with a schedule of past and / or predicted overload conditions in the query processing stack 700a. The user device 200 may store this schedule in the memory hardware 206.

[0052] In another example, user device 200 determines that an overload condition exists in the query processing stack 700a by retrieving history data 250 (e.g., an Automated Speech Recognition ASR request history) associated with previous Automated Speech Recognition ASR requests 102 communicated by user device 200 to the query processing stack 700a. The history data 250 may be stored in the memory hardware 206 of user device 200 (or stored remotely). The history data 250 may indicate specific dates, days of the week, times, etc., when user device 200 and / or other user devices 200 experienced a scenario in which the query processing stack 700a became overloaded. For example, user device 200 has experienced an overload condition in the query processing stack around 7:36 PM every weekday over the past two weeks. In this example, a television commercial during the show "Jeopardy" may have contained a phrase ("Hey poodle") spoken by an actor with an accent that sounded substantially similar to the designated hotword ("Hey Google"), resulting in voice-enabled devices being mistakenly triggered in a large number of households (thousands).

[0053] Furthermore, the on-device processing instruction 420 may provide one or more criteria for processing at least a portion of any new voice input 104 captured by the user device 200 locally on-device if the user device 200 determines that the query processing stack 700a is overloaded. For example, one or more criteria for processing at least a portion of any new voice input 104 locally may include instructing the user device 200 to transcribe the new voice input 104 using a local automatic speech recognition ASR module 200n (e.g., if available) residing in the user device 200; interpret the transcription of the new voice input 104 to determine the voice query corresponding to the new voice input 104; determine whether the user device 200 can perform the action associated with the voice query corresponding to the new voice input 104; or, if the user device 200 cannot perform the action associated with the voice query, send the transcription of the voice input 104 to the query processing stack 700a. In some implementations, one or more criteria provided by the on-device processing instruction 420 delegate a portion of the Auto Speech Recognition ASR request 102 for local processing by the user device 200 while the query processing stack 700a processes the rest. For example, the user device 200 may have a client-side text-to-speech synthesis (TTS) module so that the query processing stack 700a can provide the Auto Speech Recognition ASR response to the user device 200 in text, and the user device 200 may use the client-side text-to-speech synthesis (TTS) module to generate the corresponding synthesized speech. This scenario would alleviate the need for the server-side query processing stack 700a to generate the text-to-speech synthesis (TTS) response during overload conditions.

[0054] Figure 5 is a flowchart illustrating an exemplary configuration of the operation of method 500 for processing a pending Auto Speech Recognition ASR request 102 in the query processing stack 700a (e.g., the backend Auto Speech Recognition ASR module 710a of the query processing stack 700a) based on the availability of processing in the query processing stack 700a. In operation 502, method 500 comprises receiving the Auto Speech Recognition ASR request 102 from the user device 200 in the data processing hardware 144 of the query processing stack 700a (e.g., the query processing backend). The Auto Speech Recognition ASR request 102 comprises a voice input 104 captured by the user device 200 with a voice query and content metadata 110 associated with the voice input 104. The content metadata 110 is generated by the user device 200 as described above with reference to Figure 2. In operation 504, method 500 comprises the data processing hardware 144 determining a priority score 311 of the Auto Speech Recognition ASR request 102 based on the content metadata 110 associated with the voice input.

[0055] In operation 506, method 500 includes the step of having the data processing hardware 144 cache the Auto Speech Recognition ASR requests 102 in a preprocessing backlog 320 of pending Auto Speech Recognition ASR requests 102, each having a corresponding priority score 311. The pending Auto Speech Recognition ASR requests 102 in the preprocessing backlog 320 are ranked in order of priority score 311, as described above with reference to Figures 3A to 3C. The preprocessing backlog 320 may reside on a storage resource (e.g., memory hardware) 146 of the remote system 140. In operation 508, method 500 includes having the data processing hardware 144 provide one or more of the pending Auto Speech Recognition ASR requests 102 from the preprocessing backlog 320 to the backend Auto Speech Recognition ASR module 710 (or another module in the query processing stack 700a) based on the processing availability of the backend Auto Speech Recognition ASR module 710. As described above with reference to Figures 3A to 3C, pending Auto Speech Recognition ASR requests 102 in the backlog 320 associated with a higher priority score 311 are processed by the backend Auto Speech Recognition ASR module 710 before pending Auto Speech Recognition ASR requests 102 associated with a lower priority score 311.

[0056] Figure 6 is a flowchart illustrating an exemplary configuration of the operation of method 600, which executes an on-device processing instruction when the server-based query processing stack 700a is overloaded (for example, when an overload condition exists in stack 700a). Method 600 may be executed on the data processing hardware 204 of the user device 200. In operation 602, method 600 comprises generating an automatic speech recognition ASR request 102 in the user device 200, where the automatic speech recognition ASR request 102 comprises a voice input 104 captured by the user device 200 with a voice query, and content metadata 110 generated by the user device 200 and associated with the voice input 104. Generating the content metadata 110 associated with the voice input 104 is as described above with reference to Figure 2. In operation 604, the method comprises receiving an on-device processing instruction 420 from the server-side query processing stack 700a in the user device 200. For example, Figure 4 shows the user device 200 receiving the on-device processing instruction 420. The on-device processing instruction 420 may provide criteria for sending the Auto Speech Recognition ASR request 102 to the query processing stack 700a when the query processing stack 700a is overloaded, based on the content metadata 110 associated with the Auto Speech Recognition ASR request 102. In some implementations, the on-device processing instruction 420 provides one or more thresholds that the corresponding portion of the content metadata 110 must satisfy in order for the user device 200 to send the Auto Speech Recognition ASR request 102 to the query processing stack 700a during an overloaded state.

[0057] In operation 606, method 600 also includes determining whether the server-side query processing stack 700a is overloaded by the user device 200. As described in more detail with reference to Figure 4, the user device 200 may determine the overload condition based on at least one of the historical data 250 (e.g., prediction-based) associated with previous automatic speech recognition ASR requests communicated to the query processing stack 700a by the user device 200 (and / or other user devices), or when it receives a notification 410 from the query processing stack 700a. The notification 410a may include an overload condition status notification sent on the fly by the query processing stack 700a to indicate a schedule of past and / or predicted overload conditions in the query processing stack 700a, and / or the current overload condition. In operation 608, if the user device 200 determines that the query processing stack 700a is overloaded, method 600 includes the user device 200 executing an on-device processing instruction 420. The execution of the on-device processing command 420 by the user device 200 is as described above with reference to Figure 4.

[0058] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform a task. In some cases, a software application may be called an “application,” “app,” or “program.” Examples of applications include, but are not limited to, system diagnostic applications, system administration applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0059] Non-temporary memory may be a physical device used to temporarily or permanently store programs (e.g., a set of instructions) or data (e.g., program state information) for use in a computing device. Non-temporary memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory, read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), phase-change memory (PCM), as well as disks and tapes.

[0060] Figure 7 is a schematic diagram of an exemplary computing device 700 that can be used to implement the systems and methods described herein. The computing device 700 is intended to represent various forms of digital computers, including laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be illustrative and are not intended to limit the implementation of the inventions described and / or claimed herein.

[0061] The computing device 700 comprises a processor 711 (e.g., data processing hardware 144), a memory 721 (e.g., memory hardware 146), a storage device 731, a high-speed interface / controller 740 connected to the memory 721 and a high-speed expansion port 750, and a low-speed interface / controller 760 connected to a low-speed bus 770 and the storage device 731. Each of the components 711, 721, 731, 740, 750, and 760 is interconnected using various buses and may be mounted on a common motherboard or in other configurations as appropriate. The processor 711 can process instructions to be executed within the computing device 700, which include instructions stored in the memory 721 or storage device 731 for displaying graphical information for a graphical user interface (GUI) on an external input / output device such as a display 780 coupled to the high-speed interface 740. In other implementations, multiple processors and / or multiple buses may be used as appropriate, along with multiple memories and memory types. Furthermore, multiple computing devices 700 may be connected, and each device may provide some of the necessary functions (for example, as a server bank, a group of blade servers, or a multiprocessor system).

[0062] Memory 721 stores information within the computing device 700 on a non-temporary basis. Memory 721 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). Non-temporary memory 721 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, RAM (Random Access Memory), DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), PCM (Phase Change Memory), as well as disks and tapes.

[0063] The storage device 731 can provide the computing device 700 with large-capacity storage. In some embodiments, the storage device 731 is a computer-readable medium. In various different implementations, the storage device 731 may be a floppy disk® device, a hard disk device, an optical disk device, or a tape device, flash memory or other similar solid-state memory device, or an array of devices comprising a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied on an information carrier. The computer program product comprises instructions that, when executed, perform one or more of the methods described above. The information carrier is a computer-readable medium or a machine-readable medium, such as memory 721, storage device 731, or memory on the processor 711.

[0064] The high-speed controller 740 manages bandwidth-intensive operations for the computing unit 700, and the low-speed controller 760 manages less bandwidth-intensive operations. Such job assignments are illustrative only. In some implementations, the high-speed controller 740 is coupled to memory 721, to the display 780 (e.g., via a graphics processor or accelerator), and to a high-speed expansion port 750 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to storage device 731 and the low-speed expansion port 790. The low-speed expansion port 790, which may have various communication ports (e.g., USB, Bluetooth®, Ethernet®, Wireless Ethernet®), may be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, or network devices such as switches or routers, for example, via a network adapter.

[0065] The computing device 700 may be implemented in several different forms, as shown in the figure. For example, it may be implemented as a standard server 700a or multiple such servers 701 in a group, as a laptop computer 703, or as part of a rack server system 705.

[0066] Various implementations of the systems and technologies described herein can be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system having at least one special-purpose or general-purpose programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from a storage system and transmit data and instructions to a storage system.

[0067] These computer programs (also known as programs, software, software applications, or code) have machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. In this specification, the terms “machine-readable medium” and “computer-readable medium” mean any computer program product, non-temporary computer-readable medium, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, having a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” means any signal used to provide machine instructions and / or data to a programmable processor.

[0068] The processing and logical flows described herein can be performed by one or more programmable processors, also called data processing hardware, executing one or more computer programs to manipulate input data and produce outputs. Processing and logical flows can also be performed by specialized logic circuits such as FPGAs (Field Programmable Gate Arrays) and ASICs (Application Specific Integrated Circuits). Examples of processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors in any type of digital computer. Generally, a processor receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer has one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is coupled to those mass storage devices in a manner that allows it to receive and transfer data, or both. However, a computer is not required to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, examples of which include semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory can be complemented by or incorporated into special-purpose logic circuits.

[0069] To provide user interaction, one or more aspects of this disclosure may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, and optionally a keyboard and pointing device, such as a mouse or trackball, to which the user can provide input to the computer. Other types of devices may also be used to provide user interaction, for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, such as acoustic input, voice input, or tactile input. Furthermore, the computer may interact with the user by sending documents to or receiving documents from a device used by the user. For example, the computer may interact with the user by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0070] Numerous embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer implementation method that is executed on data processing hardware on an audio-enabled device, wherein the computer implementation method includes causing the data processing hardware to perform an operation, and the operation is The process of receiving the initial portion of the audio input captured by the aforementioned audio-enabled device, A step of detecting the presence of a keyword in the initial portion of the voice input using a hot word detector, In response to detecting the presence of the keyword in the initial portion of the voice input, The steps include: triggering the voice-enabled device so that it wakes up from sleep mode and captures the remaining portion of the voice input; A step of processing audio data corresponding to the audio input captured by the audio-enabled device in order to determine the content metadata of the audio input, and Based on the content metadata, the process includes dropping the automatic speech recognition request corresponding to the voice input before sending it to the query processing backend, A computer implementation method comprising, The content metadata determined by processing the audio data includes a confidence score for the audio quality associated with the audio input. Computer implementation method.

2. The operation further includes a step of determining that the confidence score of the voice quality associated with the voice input is unsatisfactory of a threshold. The step of dropping the automatic speech recognition request before sending the automatic speech recognition request to the query processing backend is based on determining that the confidence score associated with the voice input does not satisfy the threshold. The computer implementation method according to claim 1.

3. A computer implementation method that is performed on data processing hardware on an audio-enabled device, wherein the computer implementation method comprises causing the data processing hardware to perform an operation, and the operation is The process of receiving the initial portion of the audio input captured by the aforementioned audio-enabled device, A step of detecting the presence of a keyword in the initial portion of the voice input using a hot word detector, In response to detecting the presence of the keyword in the initial portion of the voice input, The steps include: triggering the voice-enabled device so that it wakes up from sleep mode and captures the remaining portion of the voice input; A step of processing audio data corresponding to the audio input captured by the audio-enabled device in order to determine the content metadata of the audio input, and Based on the content metadata, the process includes dropping the automatic speech recognition request corresponding to the voice input before sending it to the query processing backend, A computer implementation method comprising, The content metadata determined by processing the aforementioned audio data includes an audio quality score associated with the audio input. Computer implementation method.

4. The aforementioned audio quality score indicates the level of background noise present in the audio data. The computer implementation method according to claim 3.

5. The operation further includes a step of processing the audio data in response to detecting the presence of the keyword in the initial portion of the audio input, in order to determine a speaker identification score for the audio input indicating the possibility that the audio input matches a speaker profile associated with the user device. The computer implementation method according to claim 1.

6. A computer implementation method that is performed on data processing hardware on an audio-enabled device, wherein the computer implementation method comprises causing the data processing hardware to perform an operation, and the operation is The process of receiving the initial portion of the audio input captured by the aforementioned audio-enabled device, A step of detecting the presence of a keyword in the initial portion of the voice input using a hot word detector, In response to detecting the presence of the keyword in the initial portion of the voice input, The steps include: triggering the voice-enabled device so that it wakes up from sleep mode and captures the remaining portion of the voice input; A step of processing audio data corresponding to the audio input captured by the audio-enabled device in order to determine the content metadata of the audio input, and Based on the content metadata, the process includes dropping the automatic speech recognition request corresponding to the voice input before sending it to the query processing backend, A computer implementation method comprising, The operation includes a step of transcribing the voice input by using a local automatic speech recognition (ASR) module present on the voice-enabled device in response to detecting the presence of the keyword in the initial portion of the voice input. The step of dropping the automatic speech recognition request before sending the automatic speech recognition request to the query processing backend includes the step of dropping the automatic speech recognition request before sending the automatic speech recognition request based on the transcription to the query processing backend. Computer implementation method.

7. The aforementioned voice-enabled device is equipped with a smartphone. The computer implementation method according to claim 1.

8. The aforementioned voice-enabled device is equipped with a smart speaker. The computer implementation method according to claim 1.

9. The aforementioned voice-enabled device includes a tablet. The computer implementation method according to claim 1.

10. A system, wherein the system is The user device's data processing hardware, Memory hardware communicating with the data processing hardware, the memory hardware storing instructions that cause the data processing hardware to perform an action when executed on the data processing hardware, It is equipped with, and the operation is, A process of receiving the initial portion of the audio input captured by the audio-enabled device, A step of detecting the presence of a keyword in the initial portion of the voice input using a hot word detector, In response to detecting the presence of the keyword in the initial portion of the voice input, The steps include: triggering the voice-enabled device so that it wakes up from sleep mode and captures the remaining portion of the voice input; A step of processing audio data corresponding to the audio input captured by the audio-enabled device in order to determine the content metadata of the audio input, and Based on the content metadata, the process includes dropping the automatic speech recognition request corresponding to the voice input before sending it to the query processing backend, A system that is equipped with, The content metadata determined by processing the audio data includes a confidence score for the audio quality associated with the audio input. system.

11. The operation further includes a step of determining that the confidence score of the voice quality associated with the voice input is unsatisfactory of a threshold. The step of dropping the automatic speech recognition request before sending the automatic speech recognition request to the query processing backend is based on determining that the confidence score associated with the voice input does not satisfy the threshold. The system according to claim 10.

12. A system, wherein the system is The user device's data processing hardware, Memory hardware communicating with the data processing hardware, the memory hardware storing instructions that cause the data processing hardware to perform an action when executed on the data processing hardware, It is equipped with, and the operation is, A process of receiving the initial portion of the audio input captured by the audio-enabled device, A step of detecting the presence of a keyword in the initial portion of the voice input using a hot word detector, In response to detecting the presence of the keyword in the initial portion of the voice input, The steps include: triggering the voice-enabled device so that it wakes up from sleep mode and captures the remaining portion of the voice input; A step of processing audio data corresponding to the audio input captured by the audio-enabled device in order to determine the content metadata of the audio input, and Based on the content metadata, the process includes dropping the automatic speech recognition request corresponding to the voice input before sending it to the query processing backend, A system that is equipped with, The content metadata determined by processing the aforementioned audio data includes an audio quality score associated with the audio input. system.

13. The aforementioned audio quality score indicates the level of background noise present in the audio data. The system according to claim 12.

14. The operation further includes a step of processing the audio data in response to detecting the presence of the keyword in the initial portion of the audio input, in order to determine a speaker identification score for the audio input indicating the possibility that the audio input matches a speaker profile associated with the user device. The system according to claim 10.

15. A system, wherein the system is The user device's data processing hardware, Memory hardware communicating with the data processing hardware, the memory hardware storing instructions that cause the data processing hardware to perform an action when executed on the data processing hardware, It is equipped with, and the operation is, A process of receiving the initial portion of the audio input captured by the audio-enabled device, A step of detecting the presence of a keyword in the initial portion of the voice input using a hot word detector, In response to detecting the presence of the keyword in the initial portion of the voice input, The steps include: triggering the voice-enabled device so that it wakes up from sleep mode and captures the remaining portion of the voice input; A step of processing audio data corresponding to the audio input captured by the audio-enabled device in order to determine the content metadata of the audio input, and Based on the content metadata, the process includes dropping the automatic speech recognition request corresponding to the voice input before sending it to the query processing backend, A system that is equipped with, The operation includes a step of transcribing the voice input by using a local automatic speech recognition (ASR) module present on the voice-enabled device in response to detecting the presence of the keyword in the initial portion of the voice input. The step of dropping the automatic speech recognition request before sending the automatic speech recognition request to the query processing backend includes the step of dropping the automatic speech recognition request before sending the automatic speech recognition request based on the transcription to the query processing backend. system.

16. The aforementioned voice-enabled device is equipped with a smartphone. The system according to claim 10.

17. The aforementioned voice-enabled device is equipped with a smart speaker. The system according to claim 10.

18. The aforementioned voice-enabled device includes a tablet. The system according to claim 10.

Images