Dynamically adapting speech processing over duration of audible rendering

Abstract

Implementations set forth herein relate to an automated assistant that may bias speech processing toward certain requests according to whether or not the requests are related to content being rendered or expected to be rendered at a computing device. In this manner, speech processing may be dynamically biased according to characteristics of content that may be rendered by a particular application and / or a particular device. The biasing may be performed during rendering of a portion of the content determined to be related to a particular request by adjusting a score threshold for determining whether the particular request is received. When the portion of the content is no longer rendered, the threshold may be returned to a particular value, or adjusted again according to a subsequent portion of the content.

Description

Background Art

[0001] A human may engage in a human-machine conversation with an interactive software application referred to herein as an "automated assistant" (also referred to as a "digital agent," "chatbot," "interactive personal assistant," "intelligent personal assistant," "assistant application," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when they interact with an automated assistant) may provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances) and / or by providing textual (e.g., typed) natural language input, which in some cases may be converted to text and then processed.

[0002] As described above, the automated assistant can process audio data that captures the user's spoken utterances when determining whether and / or how to perform one or more assistant actions. For example, audio data can be generated based on detecting the user's spoken utterances via one or more microphones of a client device, and the client device includes an assistant interface for enabling the user to interact with the automated assistant. The automated assistant may include a speech recognition engine that attempts to recognize various characteristics of the spoken utterances captured in the audio data, such as the sounds (e.g., phonemes) produced by the spoken utterances, the order of the sounds produced, the rhythm of the voice, the intonation, etc. In addition, the speech recognition engine can recognize text words or phrases represented by such characteristics. The text can then be further processed by the automated assistant (e.g., using a natural language understanding (NLU) engine and / or a dialog state engine) to determine the response content for the spoken utterance. The speech recognition engine can be implemented by a client device and / or one or more automated assistant components that are away from the client device but communicate with the client device over a network.

[0003] The automated assistant may include an additional or alternative speech processing engine. For example, some speech processing engines may utilize a machine learning (ML) model (also referred to herein as a "warm word model") on a certain (certain) device, which is trained to at least selectively monitor specific words and / or phrases in the audio data captured by the microphone of the client device. Such warm word models may be trained to process audio data to generate an output indicating whether the specific words and / or phrases (also referred to herein as "warm words") for which the model is trained exist in the audio data. If the output indicates that there are specific words and / or phrases, the corresponding fulfillment may be performed. Optionally, the fulfillment may be performed without the need to perform speech recognition and / or without the need to perform NLU. For example, a given warm word model may be trained to generate an output indicating whether there is a voice of "volume up" in the audio data, and the speech processing engine may at least selectively (e.g., at least when the audio output is being rendered) use a given warm word model to process the audio data. If the output generated using a given warm word model indicates the presence of "volume up" speech in the audio data, then an implementation may be executed that, for example, increases the volume by X amount (eg, X% or X levels).

[0004] Processing audio data to recognize speech and / or monitor the presence of warm words may be inhibited by, for example, the audio output being rendered by the device receiving the speech, the nearby device rendering the audio output, and / or other background noise. For example, a user may request an automated assistant to play music via a specific residence device (e.g., an independent speaker device), and then - while the music is playing - provide a verbal request for the automated assistant to perform some other operation (e.g., set a reminder, pause the music, increase the volume, or skip to the next song). However, the audio data capturing the verbal request may also capture the music being played, capture background noise (e.g., from other devices and / or people), and / or otherwise fail to clearly capture the verbal request (e.g., due to the user's low speaking volume, the microphone far from the audio data being captured, etc.). This may have a negative impact on the processing of audio data by the automated assistant. For example, any voice embodied in the audio data captured by the residence device may be muffled and / or distorted due to combination with other audio signals (e.g., music being played). Therefore, the speech processing engine responsible for processing audio data may not be able to process accurately and / or robustly in various situations. This may result in misrecognition of speech, or lack of recognition of speech, which may be frustrating and also waste computing resources. For example, a speech processing engine that is processing audio data using a warm word model to detect specific words and / or phrases may determine that the specific words and / or phrases are not present in the audio data based on output generated based on the processing - even though the audio data includes one of the specific words or phrases. This may result in a failure to perform the corresponding fulfillment, thereby inhibiting the interaction between the automated assistant and the user and prolonging the interaction (e.g., forcing the user to repeat the specific word or phrase or utilize alternative input to cause the corresponding fulfillment to occur). Summary of the invention

[0005] Implementations described herein involve dynamically adapting speech processing based on one or more characteristics of content that is expected to be rendered and / or is currently being rendered at a device associated with an automated assistant. The content can be, for example, audio and / or visual media being rendered at one or more interfaces of a computing device and that the automated assistant can access via the computing device and / or a separate computing device.

[0006] In some implementations, speech processing adaptation can include biasing toward and / or away from determining the presence of certain (certain) spoken terms in audio data captured via a microphone during rendering of the content. For example, automatic speech recognition (ASR) based on audio data can be biased toward certain (certain) terms and / or away from certain other (certain) terms during rendering of at least a portion of the content. As another example, speech processing of audio data using a warm word model to monitor occurrences of uttering terms for which the warm word model is trained can be biased toward finding those terms that were present during rendering of at least a portion of the content.

[0007] In some versions of those implementations, speech processing adaptations may vary over the duration of the content. For example, processing may be biased toward certain term(s) during a first portion of the content, and not biased (or even biased away from certain term(s)) during a second portion of the content. For example, processing may be biased toward "volume down" and / or "down" during the first portion in response to determining that the first portion has certain loudness characteristic(s), and not biased toward "volume down" or "down" during the second portion in response to determining that the second portion lacks certain loudness characteristic(s). Also, for example, processing may be biased toward "volume down" and / or "down" during the first portion in response to determining that the first portion is the initial X seconds or Y% of the content, and not biased toward "volume down" and / or "down" during the second portion in response to determining that the second portion is after the initial X seconds or Y% of the content (and optionally, before the last X seconds or Y% of the content). As yet another example, processing may be biased toward "slow down playback" and / or "slow down" during a first portion in response to determining that the first portion includes rapid speech, but not so biased during a second portion in response to determining that the second portion lacks rapid speech.

[0008] In these and other ways, the robustness and / or accuracy of speech processing is improved by dynamic speech processing adaptation over the duration of the content. For example, the adaptation may cause a term to be more likely to be determined to be spoken during the rendering of a certain (certain) portion of the content that is relevant to the term, while also preventing the term from being more likely to be determined to be spoken during the rendering of a certain (certain) other portion of the content that is not relevant (or at least less relevant) to the term. Therefore, the performance performed based on the determination that those terms are spoken is more likely to occur during the portion without the user having to say those terms again and / or utilizing non-voice input to cause the performance to be performed. For example, biasing toward "volume up" during a quiet portion of the content may eliminate the need for the user to have to say "volume up" multiple times to cause the performance to actually turn up the volume and / or eliminate the need for the user to touch-interact with a software or hardware interface element instead to cause the performance. In addition, the performance is less likely to occur during the other portions, thereby mitigating the occurrence of unintentional occurrences of the performance, which may result in the processing of additional user input provided to correct the occurrence of the unintentional occurrence of the performance. For example, not biasing towards "volume up" (or even biasing away from "volume up") during loud portions of content may eliminate unintentional occurrences of turning the volume up during the loud portions, which may result in processing additional input from the user seeking to turn the volume down after the unintentional occurrence.

[0009] In some additional or alternative versions of those implementations, speech processing adaptations applied during rendering of all or part of the content may be based on features of the content (e.g., features of a portion of the content or the entire content), and optionally based on historical usage data indicating whether a term corresponding to such adaptation was determined to be spoken during a previous rendering of content (the same and / or additional content) having those features. As one example, in response to determining that a song carries features with a low match score and / or with a low overall popularity score, processing may be biased toward "next" and / or "skip" during at least a portion of the rendering of the song. On the other hand, in response to determining that an alternative song carries features with a high match score and / or with a high overall popularity score, processing may not be biased toward (or at least less toward) "next" and / or "skip" during rendering of the alternative song. In some implementations, a match score for a song (or other content) may be based on a consistency between the (inferred or explicit) preferences of a user requesting the song and the characteristics of the song. For example, for a user who often listens to folk music but not often listens to rock music, folk music songs may have a lower match score for the user than rock songs would have for the user. In some implementations, the match score of a song (or other content) rendered in response to a request may be additionally or alternatively based on a confidence metric reflecting whether the song is in response to the request. For example, suppose a request is made to "play hypothetical song". If there are three songs named "hypothetical song" each from a different artist, the selected song may have a lower confidence score than if there is only one song named "hypothetical song" selected. Optionally, applying such a bias may be based on historical usage data indicating a statistically significantly higher rate of determining "next" and / or "skip" (and / or corresponding fulfillment) during songs with low match scores and / or low overall popularity scores compared to the rate during songs with higher match scores and / or higher overall popularity scores. As an example, in response to determining that audio during at least a portion of a video has certain acoustic characteristic(s), such as a loudness characteristic (e.g., a root mean square (RMS) metric and / or a loudness metric full scale (LMFS) metric) indicating a perceived loudness above a threshold, processing may be biased toward “down” and / or “volume down” during the portion of rendering of the video.On the other hand, when the corresponding loudness feature does not indicate a perceived loudness above a threshold, the processing may not be biased toward (or at least less toward) "down" and / or "volume down" during rendering of the alternative portion of the video (or alternative video). Optionally, applying such a bias may be based on historical usage data indicating a statistically significantly higher rate of determining "down" and / or "volume down" (and / or corresponding performance) during rendering of portions of the audio having such loudness features. As yet another example, the processing may be dynamically biased toward "volume down" and / or "volume up" in relation to the loudness feature. For example, as the RMS measure increases, the likelihood of identifying "volume down" may increase and / or the likelihood of identifying "volume up" may decrease. Conversely, as the RMS measure decreases, the likelihood of identifying "volume up" may increase and / or the likelihood of identifying "volume down" may decrease.

[0010] In these and other ways, the robustness and / or accuracy of speech processing is improved by dynamic speech processing adaptation that depends on features of the content being rendered. As described herein, such features may include features that are different from general type classifications of content, and features may include those for the entirety of the corresponding content (e.g., an entire song, an entire album, an entire show, an entire series) and / or those for only a portion of the corresponding content. Non-limiting examples of such features include loudness features (e.g., RMS and / or LUFS metrics), speech rate features (e.g., slow speech, fast speech, normal speech), beats per minute (bpm) features (e.g., the tempo of a song), time features (e.g., first X seconds or first Y% and / or last X seconds or last Y%), and / or other features. Such features may additionally or alternatively include features that cannot be derived directly from processing of the audio and / or video of such content, such as a match score and / or popularity metric for the content. Thus, speech processing adaptations may be performed in response to determining the presence and / or absence of granular features, which may enable fine-tuned adaptations toward and / or away from certain term(s). As also described herein, the adaptations performed for certain feature(s) may be based on historical usage data, thereby ensuring that the adaptations will at least generally result in improved speech processing performance. Thus, performance that is performed based on determining that terms were spoken during the rendering of content is more likely to occur when relevant, and without the need for the user to speak those terms again and / or utilize non-voice input to cause the performance to be performed. Furthermore, performance is less likely to occur when irrelevant, thereby mitigating the occurrence of unintentional performance.

[0011] As an example of some implementations disclosed herein, playback of media (e.g., a movie, a song, an audiobook) may be rendered in response to a user providing a spoken utterance to an automated assistant. For example, the spoken utterance may be "Assistant, continue playing the movie I was watching last night", and in response to the spoken utterance, the automated assistant may cause a particular computing device (e.g., a smart TV) to continue rendering the movie from where the user left off the previous night. The content of the remaining portion of the movie may be processed to determine whether certain requests to the automated assistant may be related to one or more remaining portions of the movie. For example, the processing of the content may indicate that the amplitude of the movie audio increases (relative to the previous portion) during the first portion of the movie when an "explosion" occurs and / or satisfies an amplitude threshold, and thus the volume change request may be determined to be related to the first portion of the movie. For example, RMS processing of the audio of the content may be performed to determine the average loudness of the first portion, and determine that the determined average loudness satisfies the amplitude threshold. Alternatively or additionally, the visual features of the movie may change during the second portion of the movie when an intermission occurs, and thus the pause request may be determined to be associated with the second portion of the movie (e.g., when the user wants to take a break). As another example, the stop request may be determined to be associated with the ending portion of the movie additionally or alternatively based on a temporal feature (e.g., it is the last X minutes of the movie) and / or a visual feature (e.g., it indicates that the credits are being rendered).

[0012] As playback of the movie continues, speech processing can be dynamically biased based on the identified related requests. For example, the ASR can be biased towards terms associated with the first request (e.g., "volume", "turn", "down", "lower", etc.) during the first portion of the movie. Additionally or alternatively, the ASR can be biased towards terms associated with the second request (e.g., "pause", "stop", "break", etc.) during the second portion of the movie. Thus, during those specific portions of playback, the automated assistant will have a higher probability and / or a lower threshold to respond to spoken utterances that include those terms. For example, speech processing of audio data representing a spoken utterance can optionally generate candidate interpretations of the spoken utterance. When a portion of the content being rendered is related to a request associated with one or more terms, the priority of a specific candidate interpretation that includes one or more terms can be adjusted based on the spoken utterance being received. Alternatively or additionally, the threshold for initiating the execution of a specific operation in response to a spoken utterance can be adjusted based on the content being rendered or expected to be rendered at the computing device.

[0013] In some implementations, speech processing for one or more requests can be adjusted based on content being rendered or expected to be rendered at a computing device. For example, parameters associated with a speech model (e.g., the warm word model described above) utilized when processing a particular request (e.g., a volume level change request, such as "volume up", "volume down", etc.) can be adjusted based on the content to be rendered at the computing device. For example, a parameter can be a threshold for determining whether a user has submitted a request to an automated assistant. For example, the content being rendered at a computing device may include music set to an output volume level of 9 out of 10. When this volume setting for the rendered content is applied, the automated assistant can determine that the user is estimated to be more likely to provide a request to lower the volume setting than other requests. Based on this determination, a threshold for detecting whether the processed audio data includes a request to lower the volume can be adjusted so that the speech input captured in the audio data can be more easily interpreted as a request to lower the volume. For example, when the volume is set to 9 out of 10, and the user provides a spoken utterance such as "down," the automated assistant can respond to the spoken utterance as a volume "down" request (e.g., respond by lowering the volume level to perform an operation that modifies playback). In some implementations, the response can be based on a threshold for volume down requests that is set to 0.9 when the current volume x is less than or equal to 7 and is set to 0.5 when the current volume x is greater than 7. In other words, when audio data is processed using a speech model (e.g., a warm word model trained to detect occurrences of "volume down") and an output is generated indicating whether the audio data includes certain word(s) and / or phrase(s) (e.g., a probability from 0.0 to 1.0), the output can be compared to a threshold that is dynamically adjusted based on content being rendered or expected to be rendered at the computing device.

[0014] As a specific example, it is assumed that the client device includes a volume increase warm word model, which can be used to process audio data and generate a probability indicating whether there is "volume up" in the audio data, wherein a value closer to 1.0 indicates a greater probability of "volume up" than a value closer to 0.0. It is further assumed that the client device includes a volume reduction warm word model, which can be used to process audio data and generate a probability indicating whether there is "volume down" in the audio data, wherein a value closer to 1.0 indicates a greater probability of "volume down" than a value closer to 0.0. The volume increase warm word model and the volume reduction warm word model can be used at least selectively to process audio data captured by a microphone of the client device. For example, they can be used when music is being rendered by the client device. For example, a volume increase probability can be generated by processing audio data using a volume increase warm word model, and the volume increase probability can be compared with a volume increase threshold to determine whether there is "volume up" in the audio data (e.g., it is determined to exist only when the volume increase probability is greater than the volume increase threshold). When it is determined that "volume up" exists in the audio data (for example, the probability satisfies the volume increase threshold), the corresponding volume increase execution can be executed. Further, the volume reduction probability can be generated by processing the audio data using the volume reduction warm word model, and the volume reduction probability can be compared with the volume reduction threshold to determine whether "volume down" exists in the audio data. When it is determined that "volume down" exists in the audio data (for example, the probability satisfies the volume reduction threshold), the corresponding volume reduction execution can be executed.

[0015] Continuing with the specific example and in accordance with implementations disclosed herein, the volume up threshold and / or the volume down threshold may be dynamically adapted based on the content being rendered. For example, when the client device is rendering music at a volume setting of 4 out of 10, 5 out of 10, or 6 out of 10, the volume down threshold and the volume up threshold may be the same (e.g., 0.85). However, when the client device is rendering music at a volume setting of 7 out of 10 or greater, the volume down threshold may be reduced and / or the volume up threshold may be increased, thereby increasing the likelihood that "volume down" will be determined to be present in such cases and / or reducing the likelihood that "volume up" will be determined to be present in such cases. Additionally or alternatively, when the client device is rendering music at a volume setting of 3 out of 10 or less, the volume down threshold may be increased and / or the volume up threshold may be reduced, thereby increasing the likelihood that "volume up" will be determined to be present in such cases and / or reducing the likelihood that "volume down" will be determined to be present in such cases. As another example, when the client device is rendering music, the volume down threshold and the volume up threshold may be the same (e.g., 0.85). However, when the client device is rendering advertisements between songs, the volume down threshold may be decreased and / or the volume up threshold may be increased, thereby increasing the likelihood that "volume down" will be determined to be present in such cases and / or decreasing the likelihood that "volume up" will be determined to be present in such cases. As yet another example, when the client device is rendering certain portions of a movie, the volume down threshold and the volume up threshold may be the same (e.g., 0.85). However, when the client device is rendering alternate portions of a movie (e.g., those portions indicated by historical data as frequently causing volume increases—e.g., quiet conversation scenes), the volume down threshold may be increased and / or the volume up threshold may be decreased, thereby increasing the likelihood that "volume up" will be determined to be present in such cases and / or decreasing the likelihood that "volume down" will be determined to be present in such cases.

[0016] In some implementations, adjusting the threshold of the warm word model may include: not adjusting the threshold itself, but keeping the threshold static, but selectively applying a boost or reduction to the output generated using the warm word model. For example, a static threshold of 0.80 may be used for the volume up warm word model and may be kept static. However, during portions of the time when "volume up" is determined to be content related thereto, a boost (e.g., an increase of 10%) may be applied to the output generated using the volume up warm word model, and / or during portions of the time when "volume up" is determined to be content unrelated thereto, a reduction (e.g., a decrease of 10%) may be applied to the output generated using the volume up warm word model. Additionally or alternatively, during portions of the time when "volume up" has not yet been determined to be content related thereto, boosts and reductions may not be applied to the output generated using the volume up warm word model.

[0017] As an additional specific example, it is assumed that the client device includes a next warm word model, which can be used to process audio data and generate a probability indicating whether "next" exists in the audio data, where a value closer to 1.0 indicates a greater probability of "next" than a value closer to 0.0. The next warm word model can be used at least selectively to process audio data captured via a microphone of the client device. For example, it can be used at least when music is being rendered by the client device, and when the next warm word model is used to determine that "next" exists, it is determined that "next" exists and can result in the execution of jumping to the next song. As another example, it can be used when a list of items is being rendered by the client device, and it is determined that "next" exists and can result in the execution of jumping to the next item in the list (and pausing the rendering of the current item). For example, the next probability can be generated by processing audio data using the next warm word model, and the next probability can be compared with the next threshold to determine whether "next" exists in the audio data (for example, it is determined to exist only when the next probability is greater than the next threshold).

[0018] Continuing with the additional specific examples and in accordance with implementations disclosed herein, the next threshold can be dynamically adjusted based on the content being rendered. For example, when a client device is rendering a song, the next threshold can be a lower threshold during an initial portion of the song (e.g., the first 10 seconds) and / or an ending portion of the song (e.g., the last 10 seconds), and can be a higher threshold during other portions of the song. Thus, the likelihood of determining that "next" exists can increase during the initial portion and / or the ending portion of the song, which can correspond to portions during which the user is more likely to say "next" (e.g., "next" is more likely to be said at the beginning of playback of a song that the user does not like). In these and other approaches, the likelihood of determining that "next" is said in the audio data is dynamically adjusted during the duration of playback of the song and depending on characteristics of the rendering of the song (e.g., the amount of songs played so far and / or the amount of songs remaining to be played). As another example, when a client device is rendering a list of candidate restaurants and corresponding brief descriptions (e.g., in response to a request for “restaurants nearby”), the next threshold may be a lower threshold during an initial portion (e.g., the first 40%) of the rendering of each item in the list, and may be a higher threshold during the remainder of the rendering of each item in the list. For example, if the list is being rendered audibly and the item in the list is “Hypothetical Cafe - 1 mile away and serving staples in a contemporary setting”, the next threshold may be a lower threshold during the audible rendering of “Hypothetical Cafe” and may be a higher threshold during the audible rendering of “1 mile away and serving staple in a contemporary setting”. Thus, the likelihood of determining that “next” exists may increase during an initial portion of the rendering of the items in the list, which may correspond to a portion during which a user is more likely to say “next”.

[0019] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of those implementations and other implementations are described in more detail below.

[0020] Other implementations may include a non-transitory computer-readable storage medium storing instructions that are executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include a system of one or more computers including one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and / or elsewhere herein.

[0021] It should be understood that all combinations of the aforementioned concepts and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1A , Figure 1B and Figure 1C A diagram showing a user interacting with an automated assistant that can bias speech processing based on characteristics of rendered content that are determined to be relevant to certain assistant requests is shown.

[0023] Figure 2 A system is shown for biasing speech processing (or other assistant input) toward certain requests based on whether those requests are related to content being rendered or expected to be rendered at a computing device.

[0024] Figure 3 A method is shown for biasing speech processing based on whether content being rendered or to be rendered at a computing device is associated with one or more requests that may be submitted via spoken utterance.

[0025] Figure 4 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0026] Figure 1A , Figure 1B and Figure 1CViews 100, 120, and 140 are shown of a user 102 interacting with an automated assistant that can bias speech processing based on features of rendered content that are determined to be relevant to certain assistant requests. For example, user 102 can be viewing a portion of content rendered at a display interface 114 of a computing device 106 (e.g., a television). The portion of the content can be a streaming media program that can have a duration of playback as indicated by a progress bar 112, and one or more current frames being rendered at the display interface 114 can correspond to a progress element 110 of the progress bar. In some implementations, an automated assistant accessible via computing device 106 and / or another computing device 104 can process data associated with the content to determine whether certain assistant operations may be relevant to certain portions of the content.

[0027] For example, when the user 102 is viewing the content when the progress element 110 of the progress bar 112 is close to the middle of the duration of the playback, the automated assistant can determine that the subsequent part of the content may be related to a specific assistant request. The subsequent part of the content can correspond to the content 116 related to the request, which can be characterized by content data. The content data can be processed by the automated assistant to determine that the subsequent part of the content is related to a specific assistant request, such as a request to view the "menu" of the application that is rendering the content. In some implementations, one or more heuristic processes and / or one or more trained machine learning models can be used to determine the degree of relevance of the content being rendered or to be rendered by the computing device. For example, one or more trained machine learning models can be used to process the content data to generate one or more embeddings that can be mapped to a latent space. The distance between the embedding and another existing embedding corresponding to a specific assistant request and / or action in the latent space can be determined. When the distance meets a distance threshold, it can be determined that the specific assistant request is related to the subsequent part of the content. In some implementations, features of the content, such as text, images, audio, volume, brightness, and / or other features, can be processed to determine the relevance of the content to certain requests and / or operations. For example, text 122 rendered during the last portion of the content can be used as a basis for inferring that a particular request (e.g., a request to "skip" to view the next episode element 142 and / or view the menu 144) is relevant during the last portion of the duration of the content.

[0028] In some implementations, the historical usage data may indicate when a user has historically interacted with their automated assistant during certain portions of content playback, with prior permission from the user. The historical usage data may be processed to determine whether a portion of the content that the user 102 is currently viewing and / or anticipates viewing is associated with a certain assistant request(s). When it is determined that the assistant request is relevant and / or otherwise associated with the content, the automated assistant may generate bias data 118 according to one or more processes. The bias data 118 may indicate that a threshold for determining whether a particular request is received should be adjusted during rendering of a corresponding portion of the content. In some implementations, the bias data 118 may indicate that when a first portion of the content (e.g., associated with Figure 1A The speech processing (e.g., ASR processing and / or warm word processing) for the first request (e.g., a request to "skip" to view the next episode element 142) is biased when the content corresponding to the progress element 110 at the same time is rendered, but when the second part of the content (e.g., Figure 1B Alternatively or additionally, the bias data 118 may indicate that speech processing for a second request (e.g., a request to view menu 144) is not biased when the first portion of the content is rendered, but speech processing for the second request is biased when the second portion of the content is rendered.

[0029] For example, when computing device 106 is rendering the second portion of the content, such as Figure 1B As shown in view 120 of , user 102 may provide a spoken utterance 124 such as "Menu". The spoken utterance 124 may or may not include an invocation phrase for invoking the automated assistant. However, audio data representing the spoken utterance 124 may be processed with prior permission from the user 102 to determine whether the user 102 is requesting the automated assistant to perform a specific operation. For example, a speech processing model may be utilized when processing audio data to determine whether the user 102 is requesting to perform certain operations - despite not leading the request with an invocation phrase. The automated assistant may process the audio data corresponding to the spoken utterance 124 to generate biased speech processing data 126. The biased speech processing data 126 may indicate whether the score of the spoken utterance 124 meets a threshold for a specific request (e.g., a request to view "menu"). For example, the bias data 118 may indicate an adjusted threshold to which the score should be compared, and when the score meets the adjusted threshold, the automated assistant may initiate the execution of one or more operations to facilitate satisfying the specific request.

[0030] according to Figure 1BIn the example shown, when the bias data 118 indicates to bias the speech processing toward a “menu” command, the automated assistant can determine that the user 102 provided a specific request during the duration of playback. In some implementations, based on historical usage data indicating that multiple other users requested to view the menu of the application during the rendering of the second portion of the content, the speech processing can be biased toward the “menu” command during the second portion of the content. Alternatively or additionally, the context data can be processed together with the content data to determine that the user 102 previously issued a request to the automated assistant to render a “menu” when the duration of playback had an end within a threshold duration. Therefore, when the progress element 110 reaches the threshold duration (e.g., a remaining time length of X% of the total duration), the speech processing used to determine whether a “menu” request is received can be biased toward the “menu” request.

[0031] In other words, the score for whether the spoken utterance 124 includes a “menu” request may be a “9” out of 10, and the adjusted threshold may be “>6” over the threshold duration. Thus, the score may more easily satisfy the adjusted threshold than an unadjusted or different threshold that may be enforced during other portions of the playback of the content. For example, during the first half of the duration of playback, the bias data 118 may indicate that the threshold for a “menu” request may be “>=9”. Thus, while the spoken utterance 124 would satisfy the threshold during the first half of the duration of playback, the automated assistant may be more likely to respond during the final duration when the adjusted threshold is enforced. Additionally, the spoken utterance may not be scored as a “9” when other audible distractions occur during the first half of the duration of playback. Biasing based on content may prevent false positives for other requests that the user 102 may make to the automated assistant while viewing certain portions of the content. Thus, by reducing the number of instances of false positives, computing resources such as network bandwidth and processing may be preserved at devices that may provide access to or otherwise be controllable via the automated assistant.

[0032] Figure 2A system 200 is shown for biasing speech processing (or other input processing) toward and / or away from certain requests based on whether those requests are relevant or irrelevant to content being rendered or expected to be rendered at a computing device. An automated assistant 204 can operate as part of an assistant application provided at one or more computing devices such as a computing device 202 and / or a server device. A user can interact with the automated assistant 204 via an assistant interface 220, which can be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user can initialize the automated assistant 204 by providing speech, text, and / or graphical input to the assistant interface 220 to cause the automated assistant 204 to initiate one or more actions (e.g., provide data, control peripheral devices, access an agent, generate input and / or output, etc.). Alternatively, the automated assistant 204 can be initialized based on processing context data 236 using one or more trained machine learning models. Context data 236 may characterize one or more features of an environment in which automated assistant 204 may access, and / or one or more features of a user predicted to be intent on interacting with automated assistant 204. Computing device 202 may include a display device, which may be a display panel, including a touch interface for receiving touch input and / or gestures to allow a user to control applications 234 of computing device 202 via the touch interface. In some implementations, computing device 202 may lack a display device, thereby providing audible user interface outputs without providing graphical user interface outputs. In addition, computing device 202 may provide a user interface for receiving oral natural language input from a user, such as a microphone. In some implementations, computing device 202 may include a touch interface and may not contain a camera, but may optionally include one or more other sensors.

[0033] Computing device 202 and / or other third-party client devices may communicate with the server device via a network such as the Internet. Additionally, computing device 202 and any other computing devices may communicate with each other via a local area network (LAN) such as a Wi-Fi network. Computing device 202 may offload computing tasks to a server device in order to conserve computing resources at computing device 202. For example, a server device may host automated assistant 204, and / or computing device 202 may transmit input received at one or more assistant interfaces 220 to a server device. However, in some implementations, automated assistant 204 may be hosted at computing device 202, and various processes that may be associated with automated assistant operations may be performed at computing device 202.

[0034] In various implementations, all or less than all aspects of automated assistant 204 may be implemented on computing device 202. In some of those implementations, aspects of automated assistant 204 are implemented via computing device 202, and may interface with a server device, which may implement other aspects of automated assistant 204. The server device may optionally serve multiple users and their associated assistant applications via multiple threads. In implementations in which all or less than all aspects of automated assistant 204 are implemented via computing device 202, automated assistant 204 may be an application separate from (e.g., installed “on top of”) the operating system of computing device 202—or may alternatively be implemented directly by (e.g., treated as an application of, but integrated with) the operating system of computing device 202.

[0035] In some implementations, automated assistant 204 may include an input processing engine 206 that may employ a plurality of different modules to process input and / or output of computing device 202 and / or server devices. For example, input processing engine 206 may include speech processing engine 208 that may process audio data received at assistant interface 220 to identify text embodied in the audio data. The audio data may be transmitted from, for example, computing device 202 to a server device in order to preserve computing resources at computing device 202. Additionally or alternatively, the audio data may be processed exclusively at computing device 202.

[0036] The process for converting audio data to text may include a speech recognition algorithm that can use a neural network and / or a statistical model to identify grouped audio data corresponding to a word or phrase. The text converted from the audio data can be parsed by the data parsing engine 210 and used as text data for the automated assistant 204, which can be used to generate and / or identify command phrases, intentions, actions, slot values, and / or any other content specified by the user. In some implementations, the output data provided by the data parsing engine 210 can be provided to the parameter engine 212 to determine whether the user has provided input corresponding to a specific intention, action, and / or routine that can be performed by the automated assistant 204 and / or an application or agent that can be accessed via the automated assistant 204. For example, the assistant data 238 can be stored at a server device and / or a computing device 202, and can include data defining one or more actions that can be performed by the automated assistant 204, and parameters necessary for performing the action. The parameter engine 212 can generate one or more parameters for the intention, action, and / or slot value, and provide one or more parameters to the output generation engine 214. Output generation engine 214 may use the one or more parameters to communicate with assistant interface 220 for providing output to a user, and / or to communicate with one or more applications 234 for providing output to one or more applications 234 .

[0037] In some implementations, automated assistant 204 may be an application that may be installed “on top of” an operating system of computing device 202, and / or may itself constitute part (or the entirety) of an operating system of computing device 202. The automated assistant application includes and / or has access to on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, on-device speech recognition may be performed using an on-device speech recognition module that processes audio data (detected by a microphone) using an end-to-end speech recognition machine learning model stored locally at computing device 202. On-device speech recognition generates recognized text for spoken utterances (if any) present in the audio data. Additionally, for example, on-device natural language understanding (NLU) may be performed using an on-device NLU module that processes recognized text generated using on-device speech recognition and optional context data to generate NLU data.

[0038] The NLU data may include an intent corresponding to the spoken utterance and optionally parameters for the intent (e.g., slot values). On-device fulfillment may be performed using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU) and optionally other local data to determine actions to be taken to resolve the intent (and optionally parameters for the intent) of the spoken utterance. This may include determining local and / or remote responses (e.g., replies) to the spoken utterance, interactions with locally installed applications performed based on the spoken utterance, commands transmitted to an Internet of Things (IoT) device (directly or via a corresponding remote system) based on the spoken utterance, and / or other resolution actions performed based on the spoken utterance. The on-device fulfillment may then initiate local and / or remote execution / execution of the determined actions to resolve the spoken utterance.

[0039] In various implementations, remote speech processing, remote NLU, and / or remote fulfillment may be at least selectively utilized. For example, the recognized text may be at least selectively transmitted to a remote automated assistant component for remote NLU and / or remote fulfillment. For example, the recognized text may be optionally transmitted for remote execution in parallel with on-device execution or in response to failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be prioritized at least due to the reduced latency they provide when resolving spoken utterances (because resolving spoken utterances does not require a client-server round trip). Further, on-device functionality may be the only functionality available in the absence of a network connection or in the absence of a limited network connection.

[0040] In some implementations, computing device 202 may include one or more applications 234, which may be provided by a third-party entity that is different from the entity that provided computing device 202 and / or automated assistant 204. Automated assistant 204 and / or an application state engine of computing device 202 may access application data 230 to determine one or more actions that can be performed by one or more applications 234, as well as a state of each of one or more applications 234 and / or a state of a corresponding device associated with computing device 202. Automated assistant 204 and / or a device state engine of computing device 202 may access device data 232 to determine one or more actions that can be performed by computing device 202 and / or one or more devices associated with computing device 202. Additionally, application data 230 and / or any other data (e.g., device data 232) can be accessed by automated assistant 204 to generate context data 236, which can characterize the context in which a particular application 234 and / or device is executing, and / or the context in which a particular user is accessing computing device 202, accessing application 234, and / or any other device or module.

[0041] While one or more applications 234 are executing at computing device 202, device data 232 may characterize the current operating state of each application 234 being executed at computing device 202. Additionally, application data 230 may characterize one or more features of the executing applications 234, such as the content of one or more graphical user interfaces being rendered under the direction of one or more applications 234. Alternatively or additionally, application data 230 may characterize action paradigms based on the current operating state of the respective applications, which action paradigms may be updated by the respective applications and / or by automated assistant 204. Alternatively or additionally, one or more action paradigms of one or more applications 234 may remain static, but may be accessed by the application state engine in order to determine appropriate actions to be initiated via automated assistant 204.

[0042] The computing device 202 may further include an assistant invocation engine 222 that can process the application data 230, the device data 232, the context data 236, and / or any other data accessible to the computing device 202 using one or more trained machine learning models. The assistant invocation engine 222 can process this data to determine whether to wait for the user to explicitly say the invocation phrase to invoke the automated assistant 204, or to treat the data as indicating the user's intention to invoke the automated assistant - instead of requiring the user to explicitly say the invocation phrase. For example, one or more trained machine learning models can be trained using examples of training data based on scenarios in which the user is in an environment where multiple devices and / or applications are exhibiting various operating states. Instances of training data can be generated to capture training data that characterizes contexts in which the user invokes the automated assistant and other contexts in which the user does not invoke the automated assistant. When one or more trained machine learning models are trained based on these instances of training data, the assistant invocation engine 222 can cause the automated assistant 204 to detect or limit detection of verbal invocation phrases from the user based on features of the context and / or environment.

[0043] In some implementations, the system 200 may include a content feature engine 216 that can process content data characterizing content to facilitate identifying features of the content. The content data may characterize content currently being rendered at the interface, content expected to be rendered, and / or content previously rendered at the interface. In some implementations, the content feature engine 216 may access the content data by performing a screenshot or other operation for capturing content being rendered by a computing device with prior permission from the user. Alternatively or additionally, the content feature engine 216 may access the content data by retrieving application data 230 from one or more applications that may be assisting in the rendering of the content. In some implementations, features of the content may be identified by processing the content data using one or more heuristic processes and / or one or more trained machine learning models. For example, one or more trained machine learning models may be utilized to identify objects in a video frame that may be rendered at a computing device. Alternatively or additionally, one or more trained machine learning models may be utilized to identify context features that may be apparent when a computing device is rendering content.

[0044] In some implementations, the system 200 may include a request relevance engine 218 that can determine the degree of relevance of content features to a particular request that can be fulfilled by an automated assistant and / or an operation that can be performed by an automated assistant. In some implementations, the degree of relevance can be determined using one or more heuristic processes and / or one or more trained machine learning models. For example, when a particular feature (e.g., a countdown timer) of content (e.g., a movie) is identified, historical usage data can be processed to determine whether one or more users issued certain assistant requests (e.g., a request to "pause" playback) when the particular feature (e.g., the countdown timer) was previously rendered during the rendering of the content (e.g., a different movie). In some implementations, the degree of relevance of the particular request can be determined based on how many users previously issued the particular request during the rendering of the particular feature. For example, it can be determined that "next" is related to the initial duration of a song (generally, or certain songs) based on historical usage data indicating that a user explicitly invoked the automated assistant (e.g., by saying an invocation utterance such as "OK Assistant") and said a threshold number and / or rate of occurrences of "next". Based on such a determination of relevance, the threshold of the next warm word model during the initial duration of the song can be lowered, making it more likely that "next" is determined to exist based on processing the audio data using the next warm word model. Notably, by processing the audio data using the next warm word model, there is no need to explicitly call the automated assistant to detect "next" in the corresponding audio data and perform a corresponding fulfillment (e.g., skip to the next song). As another example, it can be determined that "volume down" is associated with a volume setting of the device from 6 to 10 based on historical usage data indicating a threshold number of occurrences and / or ratios at which the user adjusted the volume to "down" on the device when the volume was set from 6 to 10. For example, it can be determined that it is relevant based on the number of occurrences and / or ratio of volume down adjustments exceeding the number of occurrences of volume up adjustments (e.g., exceeding a threshold amount). Adjustment of the volume setting indicated by the historical usage data may be in response to touch input (e.g., interaction with a software or hardware volume adjustment button) and / or verbal input (e.g., a user saying “volume down” or “volume 5” (or other lower volume setting)). Based on such a determination of relevance, when audio is rendered at a volume setting from 6 to 10, the threshold of the volume down warm word model may be lowered, thereby making it more likely that “volume down” is present based on processing the audio data using the volume down warm word model.

[0045] In some implementations, the request correlation engine 218 may determine which (or which) bias adaptations to apply based on processing content features using a trained machine learning model. For example, the trained machine learning model may be trained based on supervised training examples from historical usage data. Each supervised training instance may include a training input that includes features of all or part of the content rendered when a fulfillment occurs, such as a discretization of loudness features, temporal features, matching features, and / or other features described herein. In addition, each supervised training instance may include a training output that includes an indication of a fulfillment. For example, the training output may include a one-hot vector having a hot value indicating which fulfillment (of the multiple fulfillments indicated by the vector) occurred. For example, if the machine learning model is trained to predict the probability of each of the multiple fulfillments based on the input features, the first value in the output vector may correspond to the first fulfillment (e.g., "volume up"), the second value may correspond to the second fulfillment (e.g., "volume down"), the third value may correspond to the third fulfillment (e.g., "skip"), and so on. Continuing with the example, for content being rendered, the request correlation engine 218 can use a trained machine learning model to process discretization of features of the content to generate a predicted probability for each of a plurality of fulfillments. A bias can then be performed for terms associated with fulfillments (if any) that meet a threshold. For example, if the "volume up" probability meets an upper threshold (i.e., has a high probability), the volume increase threshold can be adapted to make it more likely that "volume up" is detected in the processing of the audio data. Moreover, for example, if the "volume down" probability additionally or alternatively meets a lower threshold (i.e., has a low probability), the volume reduction threshold can be adapted to make it less likely that "volume down" is detected in the processing of the audio data. Additional and / or alternative machine learning models can be trained for processing input features to generate an output indicating which (which) speech processing bias adaptations (if any) are to be applied during the rendering of content having those input features. For example, an additional or alternative machine learning model may include a machine learning model that is trained only to determine whether a “volume up” bias should be applied (e.g., the output may be a single value indicating whether a “volume up” bias should be applied).

[0046] In some implementations, the system 200 may include a bias determination engine 226 that may determine whether to bias a speech process based on whether certain features of portions of the content are being rendered at the computing device 202. For example, when certain features are temporarily rendered at the interface, thresholds for various different requests may be dynamically adjusted by the bias determination engine 226. In some implementations, the bias may affect a model used to determine whether a request for an operation in a limited number of operations has been submitted. For example, a model may be used to determine whether a volume change request has been received, and the sensitivity of the determination may be adjusted based on whether one or more relevant content features are being rendered or have been rendered for a threshold duration.

[0047] In some implementations, the system 200 may include a biased execution engine 224 that may cause the automated assistant 204 to initiate one or more operations in response to receiving a request and according to a speech processing bias. For example, a user may provide a spoken utterance that embodies a particular request, and the computing device 202 may capture the spoken utterance as audio data. The input processing engine 206 may process the audio data, and certain aspects of the processing may be biased according to content that may be rendered at the computing device 202 or a separate computing device. The audio data may be processed to generate a score indicating a probability and / or likelihood that the user has provided a particular request. The score may be compared to a threshold value, which may have been adjusted according to features being rendered at the interface when the spoken utterance is received or within a threshold duration of receiving the spoken utterance. When the score meets the threshold value, the automated assistant 204 may initiate the execution of one or more operations to facilitate fulfilling the particular request.

[0048] Figure 3A method 300 for biasing speech processing based on whether content being rendered or to be rendered at a computing device is associated with one or more requests that can be submitted by a user to an automated assistant is shown. The method 300 can be performed by one or more computing devices, applications, and / or any other device or module that can be associated with an automated assistant. The method 300 can include an operation 302, which determines whether content related to one or more assistant operations (e.g., operations that can be initialized by an automated assistant) is being rendered or expected to be rendered at an interface of the computing device. The content can be, for example, audio content, visual content, tactile content, and / or any content rendered through one or more modalities that can be utilized by the user. In some implementations, the content can be audio content and video content rendered at a computing device, and the content can include a portion of the content that can be associated with one or more assistant operations. When it is determined that the portion of the content is associated with one or more assistant operations, the method 300 can proceed from operation 302 to operation 304. Otherwise, the automated assistant can continue to determine whether certain assistant operations may be related to a certain content that the user is accessing or expects to access.

[0049] Operation 304 may include generating bias data indicating that one or more specific assistant operations are associated with one or more features of the content. For example, content data characterizing content being or to be rendered at a computing device may be processed to determine whether the content is associated with and / or otherwise associated with one or more assistant operations that can be initialized by an automated assistant. In some implementations, one or more heuristic processes and / or one or more trained machine learning models may be used to identify one or more features of the content. One or more features may then be compared with one or more assistant operations to determine whether one or more features are associated with an assistant operation. In some implementations, historical usage data may also be processed with prior permission from the user to determine whether the user has ever provided certain requests during the rendering of certain content features. Based on the historical usage data, the automated assistant may determine operations that the user has historically associated with certain rendered content, and utilize the determined associations to bias speech processing. For example, a portion of the content that includes intermission features (e.g., a blank screen, no audio, etc.) may be associated with an operation to jump to other content, and / or stop rendering content and / or stop playback progress (e.g., a user request to stop video playback). Thus, bias data generated based on such portions of the content may indicate that speech received during such portions of the content should be biased towards requests for “skip” and / or “stop”.

[0050] Alternatively or additionally, the context associated with the content to be rendered and / or being rendered can be processed to determine whether certain operations are related to the content or otherwise associated. For example, the setting of the device can be determined as the context of certain content so that voice processing can be biased towards a request to control such settings when certain content is rendered. In some implementations, the volume of the device can be a context feature of the content including audio of a specific amplitude (e.g., the audio amplitude can be estimated and / or determined by one or more sensors). For example, when the volume setting is set to "8" out of "10", the automated assistant can determine to bias voice processing when the audio content is rendered. Alternatively or additionally, when the volume setting is set to "8" out of "10", the automated assistant can determine to bias voice processing when the audio content is rendered at a specific threshold (e.g., 100 decibels as measured using a microphone of a computing device) and / or above the specific threshold.

[0051] From operation 304, method 300 may proceed to operation 306, which may include determining whether a spoken utterance has been received from a user. When a spoken utterance has been received, method 300 may proceed to operation 308 from operation 306. Otherwise, method 300 may return to operation 302 to determine whether the content is being rendered or expected to be rendered via a computing device. Operation 308 may include determining whether a spoken utterance is received while the content is being rendered. In other words, the automated assistant may determine whether a user has provided a spoken utterance while a particular portion of the content corresponding to the bias data is being rendered. When a spoken utterance is received during the rendering of the content and / or within a threshold duration while the content is being rendered, method 300 may proceed to operation 310 from operation 308. Otherwise, method 300 may proceed to operation 312 from operation 308.

[0052] Operation 310 may include processing audio data according to bias data and voice processing to facilitate initialization of one or more operations to satisfy a request determined to be embodied in a spoken utterance. When a request corresponds to one or more operations identified as being related to the rendered content, the bias data may indicate that the voice processing should be biased toward requests that may be directed to those operations. For example, it may be determined that a pause operation is associated with a rendered portion of the content because the rendered portion of the content is within a threshold duration from the end of the content. Therefore, the bias data may characterize an adjusted threshold for determining whether to initialize a pause operation in response to a spoken utterance. For example, in some implementations, processing a spoken utterance may include generating scores for certain operations that the spoken utterance may be directed to. Each score may be compared to a corresponding operation threshold for each corresponding operation. However, when it is determined that an operation is associated with a portion of the content being rendered, the operation threshold may be adjusted during the rendering of the portion of the content and / or within a threshold duration of the rendering of the portion of the content.

[0053] As an example, audio data corresponding to the spoken utterance may be processed to determine a score indicating the likelihood and / or probability that the user is requesting a “pause” operation for the content. Additionally or alternatively, the audio data may be processed to determine another score indicating the likelihood and / or probability that the user is requesting a “pause” operation for another device or application (e.g., a countdown timer rendered by a kitchen computing device) or that the user is otherwise not directing the spoken utterance to the automated assistant. The score may be compared to an adjusted threshold (e.g., adjusted according to bias data) to determine whether the user provides a request for a “pause” operation to affect the content. When the score satisfies the adjusted threshold, the method 300 may proceed from operation 310 to operation 314, which causes one or more operations to be initialized to facilitate fulfilling the identified request. For example, when it is determined that a “pause” operation has been requested to affect the content, the automated assistant may communicate with the application that is rendering the content to cause the playback of the content to be paused or otherwise stopped.

[0054] When it is determined that the spoken utterance has been received when the portion of the content is not rendered, the method 300 may proceed from operation 308 to operation 312. Operation 312 may include processing the audio data according to other bias data, and / or speech processing that would otherwise be utilized when no bias data is available. In some implementations, other bias data may have been generated based on other factors that are unrelated to the content being rendered. For example, other bias data may have been generated based on the user providing the spoken utterance, the device capturing the spoken utterance, the location of the user providing the spoken utterance, the state of one or more applications when the spoken utterance was received, and / or any other factor that may provide a basis for biasing speech processing. For example, although the score generated for the "pause" operation may not meet the threshold (e.g., an unadjusted value) for determining whether the user has provided a request to "pause" the content, other scores may be generated in response to the spoken utterance. For example, when a spoken utterance (e.g., "pause") is received outside the threshold duration of playback of the portion of the content that is occurring, a separate threshold for pausing a kitchen device (e.g., an automatic vacuum cleaner) may be met in certain contexts.

[0055] From operation 312 and / or operation 310, method 300 may proceed to operation 314, which may include causing one or more operations to be initialized to facilitate fulfillment of the request. For example, when the score of the "pause" operation satisfies the adjusted threshold, the automated assistant may initialize the execution of one or more operations to facilitate fulfillment of the request to pause the playback of the content. Alternatively, when the score of another request for the operation of the "pause" device (e.g., not the playback of the content) satisfies another threshold (e.g., a threshold associated with an operation for pausing the vacuuming operation of the automatic vacuum cleaner), the automated assistant may initialize the execution of one or more operations to facilitate fulfillment of the other request. Method 300 may then return to operation 302 and / or another operation to facilitate fulfillment of the request from the user.

[0056] Figure 4 4 is a block diagram 400 of an example computer system 410. The computer system 410 generally includes at least one processor 414 that communicates with a number of peripheral devices via a bus subsystem 412. These peripheral devices may include a storage subsystem 424 (including, for example, a memory 425 and a file storage subsystem 426), a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices allow a user to interact with the computer system 410. The network interface subsystem 416 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0057] The user interface input device 422 may include a keyboard, a pointing device (such as a mouse, a trackball, a touch pad or a graphics tablet, a scanner, a touch screen incorporated into a display), an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computer system 410 or over a communication network.

[0058] User interface output device 420 can include display subsystem, printer, fax machine or non-visual display such as audio output device.Display subsystem can include cathode ray tube (CRT), flat panel device such as liquid crystal display (LCD), projection device or some other mechanisms for creating visible image.Display subsystem can also provide non-visual display such as via audio output device.In general, the use of term "output device" is intended to include all possible types of devices and modes for outputting information from computer system 410 to user or another machine or computer system.

[0059] Storage subsystem 424 stores programming and data constructs that provide some or all of the functionality of the modules described herein. For example, storage subsystem 424 may include logic for performing selected aspects of method 300 and / or implementing one or more of system 200, computing device 104, computing device 106, automated assistant, and / or any other application, device, apparatus, and / or module discussed herein.

[0060] These software modules are typically executed by processor 414 alone or in combination with other processors. The memory 425 used in storage subsystem 424 may include multiple memories, including a main random access memory (RAM) 430 for storing instructions and data during program execution and a read-only memory (ROM) 432 in which fixed instructions are stored. File storage subsystem 426 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored in storage subsystem 424 by file storage subsystem 426, or in other machines accessible to processor 414.

[0061] The bus subsystem 412 provides a mechanism for the various components and subsystems of the computer system 410 to communicate with each other as intended. Although the bus subsystem 412 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0062] Computer system 410 can be of different types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 4 The description of the computer system 410 depicted in FIG. 4 is intended only as a specific example for illustrating some implementations. Many other configurations of the computer system 410 are possible, with similar configurations. Figure 4 The computer systems may have more or fewer components than those depicted.

[0063] In cases where the systems described herein collect or make available personal information about users (or "participants" as they are often referred to herein), users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about a user's social network, social behavior or activities, occupation, a user's preferences, or a user's current geographic location), or to control whether and / or how content is received from a content server that may be more relevant to the user. In addition, certain data may be processed in one or more ways prior to its storage or use so that personally identifiable information is removed. For example, the identity of a user may be processed so that personally identifiable information about the user cannot be determined, or, in the case of geographic location information, the geographic location of the user may be generalized (such as to a city, zip code, or state level) so that the specific geographic location of the user cannot be determined. Thus, users may control how information about the user is collected and / or used.

[0064] Although several implementations have been described and illustrated herein, a variety of other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize or confirm many equivalents of the implementations described herein using only routine experiments. Therefore, it should be understood that the aforementioned implementations are presented only by way of example, and it should be understood that within the scope of the attached claims and their equivalents, implementations may be practiced in a manner other than specifically described and claimed. The implementations disclosed herein relate to the features, systems, products, materials, kits, and / or methods of each individual described herein. In addition, any combination of two or more such features, systems, products, materials, kits, and / or methods is included within the scope of the present disclosure if such features, systems, products, materials, kits, and / or methods do not contradict each other.

[0065] In some implementations, a method implemented by a processor is provided, and the method includes: during audible rendering of a first portion of content by one or more hardware speakers of a client device: processing audio data captured by one or more microphones of the client device or an additional client device using a warm word model to generate an output indicating whether the audio data includes speaking one or more specific words and / or phrases; identifying a first threshold based on one or more features of the first portion of the content and / or the audible rendering of the first portion; and in response to identifying the first threshold: determining whether the audio data includes speaking one or more specific words and / or phrases based on the output and the first threshold, and when it is determined that the audio data includes speaking one or more specific words and / or phrases: causing execution corresponding to the warm word model. The warm word model is trained to generate an output indicating whether the one or more specific words and / or phrases are present in the audio data. The method also includes, during audible rendering of a second portion of the content by one or more hardware speakers: processing additional audio data captured by one or more microphones using a warm word model to generate an additional output indicating whether the additional audio data includes speaking one or more specific words and / or phrases; identifying a second threshold based on one or more alternative features of the second portion of the content and / or the audible rendering of the second portion; and in response to identifying the second threshold: determining whether the additional audio data includes speaking one or more specific words and / or phrases based on the output and the second threshold, and when it is determined that the additional audio data includes speaking one or more specific words and / or phrases: causing execution corresponding to the warm word model.

[0066] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0067] In some implementations, identifying the first threshold is based on one or more features of a first portion of the content, and identifying the second threshold is based on one or more alternate features of a second portion of the content. In some versions of those implementations, the one or more features of the first portion include the first portion being an initial portion of the content, and the one or more alternate features of the second portion include the second portion being a separate portion of the content, the separate portion being a non-initial portion of the content. In some of those versions, the method further includes, prior to audible rendering of the content: generating a first threshold based on a number and / or ratio of past occurrences of fulfillment during past audible renderings with the one or more features in historical usage data; and assigning the first threshold to the one or more features. In some of those versions: the content is a song, and fulfillment includes causing rendering of the song to stop and causing rendering of an alternate song; the content is an item in a list of items, and fulfillment includes causing rendering of the item to stop and causing rendering of a next item in the list of items; or the content is a song, and fulfillment includes adjusting a volume of the audible rendering of the content. In some other versions of those implementations, one or more characteristics of the first portion include the first portion being an ending portion of the content, and one or more alternative characteristics of the second portion include the second portion being a separate portion of the content that is a non-ending portion of the content.

[0068] In some implementations, the method further includes, prior to audible rendering of the content: generating a first threshold based on a number and / or ratio of past occurrences fulfilled during past audible renderings having the one or more features in the historical usage data; and assigning the first threshold to the one or more features.

[0069] In some implementations, identifying the first threshold is based on one or more features of the audible rendering of the first portion of the content, and identifying the second threshold is based on one or more alternate features of the audible rendering of the second portion of the content. In some versions of those implementations, the one or more features of the audible rendering of the first portion of the content include a first volume of the audible rendering of the first portion of the content, and the one or more alternate features of the audible rendering of the second portion of the content include a second volume of the audible rendering of the second portion of the content. In some of those versions, performing includes reducing the volume of the audible rendering of the content, and / or the one or more microphones are of the client device and processing the audio data and processing the additional audio data are performed at the client device.

[0070] In some implementations, the one or more characteristics of the first portion of the content include a loudness metric for the first portion of the content, and the one or more alternative characteristics of the second portion of the content include a second loudness metric different from the first loudness metric.

[0071] In some implementations, a method implemented by a processor is provided, and the method includes: during audible rendering of content by one or more hardware speakers of a client device: processing audio data captured by one or more microphones of the client device using a warm word model to generate an output indicating whether the audio data includes speaking one or more specific words and / or phrases; selecting a specific threshold from a plurality of candidate thresholds and based on features of the content and / or the audible rendering of the content; and in response to selecting the specific threshold: determining whether the audio data includes speaking one or more specific words and / or phrases based on comparing the output to the specific threshold, and when determining that the audio data includes speaking one or more specific words and / or phrases: causing execution corresponding to the warm word model. The warm word model is trained to generate an output indicating whether the one or more specific words and / or phrases are present in the audio data.

[0072] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0073] In some implementations, selecting a particular threshold is based on one or more features of the content. In some versions of those implementations, the audible rendering of the content is in response to a user input received in association with a user account, and the one or more features of the content include a match score for the content, such as a match score generated based on a degree of match between the content and the user input and / or a degree of match between the content and a user account feature of the user account. In some additional or alternative versions of those implementations, the one or more features of the content include an overall popularity metric for the content. In some additional or alternative versions of those implementations, the one or more features of the content include one or more acoustic features of an audio portion of the content, such as a root mean square (RMS) metric and / or a loudness unit full scale (LUFS) metric.

[0074] In some implementations, a method implemented by a processor is provided, and the method includes: during audible rendering of content by one or more hardware speakers of a client device: processing audio data captured by one or more microphones of the client device using a warm word model to generate an output indicating whether the audio data includes speaking one or more specific words and / or phrases; determining based on one or more features of the content and / or the audible rendering of the content, applying a boost to the output to generate a boosted output; and determining based on comparing the boosted output to a threshold whether the audio data includes speaking one or more specific words and / or phrases; and when it is determined that the audio data includes speaking one or more specific words and / or phrases: causing execution corresponding to the warm word model. The warm word model is trained to generate an output indicating whether the one or more specific words and / or phrases are present in the audio data.

[0075] In some implementations, a method implemented by a processor is provided, and the method includes determining, by an automated assistant at a computing device, one or more features of content that can be rendered via an interface of the computing device or a separate computing device. The method also includes generating bias data indicating that a particular assistant operation is more relevant to the one or more features of the content than other assistant operations based on the one or more features of the content. The method also includes receiving, by the automated assistant, a spoken utterance from a user when rendering the content via the computing device or the interface of the separate computing device. The method also includes processing audio data using the bias data and in response to receiving the spoken utterance to facilitate determining a request embodied in the spoken utterance. The audio data embodies at least a portion of the spoken utterance from the user. The method also includes causing, by the automated assistant and in response to receiving the spoken utterance, one or more operations to be performed to facilitate fulfilling a specific request determined to be embodied in the spoken utterance from the user.

[0076] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0077] In some implementations, the content rendered via the interface is audible, the one or more characteristics of the content include changes in audio amplitude of the content, and generating the bias data includes determining that adjusting a volume level of the content is associated with the one or more characteristics of the content.

[0078] In some implementations, the content rendered via the interface includes video playback, the one or more characteristics of the content include an end of video playback, and generating the bias data includes determining that an operation that modifies playback of the content is associated with the one or more characteristics of the content.

[0079] In some implementations, generating bias data indicating that a particular assistant operation is more relevant to one or more characteristics of content than other assistant operations includes assigning an adjusted threshold to a particular threshold utilized in determining whether a user request directs the automated assistant to perform the particular assistant operation.

[0080] In some implementations, generating bias data indicating that a particular assistant operation is more relevant to one or more features of the content than other assistant operations includes assigning an adjusted threshold to the particular threshold when the content is being rendered at an interface of a computing device or a separate computing device, and assigning a different threshold to the particular threshold when the content is not being rendered at the interface. The particular threshold is utilized when determining whether a user request directs the automated assistant to perform the particular assistant operation.

[0081] In some implementations, generating bias data indicating that a particular assistant operation is more relevant to one or more features of the content than other assistant operations includes determining that at least one other user has submitted a separate request to at least one other instance of the automated assistant to perform the particular assistant operation when the content was previously rendered via at least one other corresponding computing device.

[0082] In some implementations, the method further includes determining one or more settings of the computing device or a separate computing device that is rendering the content. In those implementations, generating the bias data is also based on one or more settings of the computing device or the separate computing device.

[0083] In some implementations, the one or more characteristics of the content include an output volume of the content, and the one or more settings include a volume setting of the computing device or a separate computing device.

[0084] In some implementations, a method implemented by a processor is provided, and the method includes determining, by an automated assistant at a computing device, that content is being rendered via an interface of the computing device or a separate computing device. The content has a duration of playback, and one or more features of the content change during the duration of playback. The method also includes determining, based on the content being rendered, that a first request is associated with a first portion of playback, and determining that a second request is associated with a second portion of playback, the second portion being after the first portion of playback. The method also includes generating bias data based on the first request and the second request, the bias data indicating that speech processing of the automated assistant will be biased toward the first request during the first portion of playback, and will be biased toward the second request during the second portion of playback. The method also includes receiving, by the automated assistant, a spoken utterance from a user that embodies a specific request to be fulfilled by the automated assistant. The method also includes processing audio data representing at least a portion of the spoken utterance from the user in response to receiving the spoken utterance and based on the bias data. When the spoken utterance is received during the first portion of playback, the processing of the audio data is biased toward the first request, and when the spoken utterance is received during the second portion of playback, the processing of the audio data is biased toward the second request. The method also includes, in response to receiving the spoken utterance and based on processing the audio data, causing performance of one or more operations to facilitate fulfillment of the identified request embodied in the spoken utterance.

[0085] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0086] In some implementations, the first request and / or the second request is a volume change request, and determining that the first request is associated with the first portion of the playback and the second request is associated with the second portion of the playback includes determining that the volume change request is associated with the first portion of the playback and / or the second portion of the playback.

[0087] In some implementations, the first request and / or the second request is a stop request, and determining that the first request is associated with the first portion of the playback and the second request is associated with the second portion of the playback includes determining that the stop request is associated with the first portion of the playback and / or the second portion of the playback.

[0088] In some implementations, generating bias data indicating that speech processing of the automated assistant is to be biased toward the first request during the first portion of the playback includes assigning an adjusted threshold to a particular threshold for determining whether the first request is provided to the automated assistant.

[0089] In some implementations, generating bias data indicating that speech processing of the automated assistant is to be biased toward the first request during a first portion of playback includes assigning an adjusted threshold to a particular threshold during the first portion of playback, and assigning a different threshold to the particular threshold during a second portion of playback. The particular threshold is utilized when determining whether the user request directs the automated assistant to perform a particular assistant operation.

[0090] In some implementations, the method further includes determining one or more settings of the computing device or a separate computing device that is rendering the content. In those implementations, generating the bias data is also based on one or more settings of the computing device or the separate computing device.

[0091] In some implementations, determining that the first request is associated with the first portion of the playback includes: processing content data using one or more trained machine learning models to facilitate determining whether the first portion of the playback is associated with the first request. The content data represents a portion of the content rendered during the first portion of the playback, and the one or more trained machine learning models are trained using data indicating one or more requests submitted by other users during rendering of the portion of the content at other respective devices.

[0092] In some implementations, a method implemented by a processor is provided, the method comprising determining, by an automated assistant at a computing device, that a portion of content that can be rendered via an interface of the computing device or a separate computing device is associated with a specific request that can be acted upon by the automated assistant. The content has a duration of playback, and one or more features of the content change during the duration of playback. The method also includes causing a threshold request value to be adjusted based on the degree of relevance of the specific request to the portion of the content. The threshold request value is used by the automated assistant as a basis for determining whether the assistant input includes a threshold for a specific request. The method also includes receiving, by the automated assistant, a spoken utterance from a user. The spoken utterance includes a request that can be acted upon by the automated assistant. The method also includes, when a spoken utterance is received during the duration of playback including the content: causing the automated assistant to perform one or more operations to facilitate fulfillment of the specific request or another request based on whether the threshold is met; and when a spoken utterance is not received during the duration of playback including the content: causing the automated assistant to perform one or more other operations to facilitate fulfillment of the specific request or another request based on whether a separate threshold is met.

[0093] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0094] In some implementations, determining that the portion of the content is associated with a particular request that can be acted upon by the automated assistant includes determining that at least one other user submitted the particular request to at least one other instance of the automated assistant when the portion of the content was previously rendered via at least one other respective computing device.

[0095] In some implementations, the method also includes determining one or more settings of a computing device or a separate computing device that can render the portion of the content. In those implementations, causing the threshold request value to be adjusted is also based on one or more settings of the computing device or a separate computing device.

[0096] In some implementations, determining that the portion of the content is related to a particular request that is actionable by the automated assistant includes determining that the particular request was submitted by the other user to another instance of the automated assistant while the portion of the content is being rendered to the other user.

[0097] In some implementations, the particular request corresponds to a particular operation for modifying settings of an application and / or device that does not control playback of the portion of the content.

Claims

1. A method implemented by one or more processors, the method comprising: During audible rendering of a first portion of the content by one or more hardware speakers of the client device: processing audio data captured by one or more microphones of the client device or an additional client device using a warm word model to generate an output indicating whether the audio data includes utterance of one or more specific words and / or phrases, wherein the warm word model is trained to generate an output indicating whether the one or more specific words and / or phrases are present in the audio data; identifying a first threshold based on one or more characteristics of the first portion of the content and / or the audible rendering of the first portion; and In response to identifying the first threshold: determining, based on the output and the first threshold, whether the audio data includes utterance of the one or more particular words and / or phrases; When it is determined that the audio data includes speaking the one or more specific words and / or phrases: causing execution corresponding to the warm word model to be performed; During audible rendering of a second portion of the content by the one or more hardware speakers: processing additional audio data captured by the one or more microphones using the warm word model to generate additional output indicating whether the additional audio data includes utterance of the one or more specific words and / or phrases, identifying a second threshold based on one or more alternative features of the second portion of the content and / or the audible rendering of the second portion; and In response to identifying the second threshold: determining, based on the output and the second threshold, whether the additional audio data includes speaking the one or more particular words and / or phrases; When it is determined that the additional audio data includes speaking the one or more specific words and / or phrases: The execution corresponding to the warm word model is caused to be executed.

2. The method according to claim 1, wherein: Identifying the first threshold is based on the one or more characteristics of the first portion of the content, and wherein identifying the second threshold is based on the one or more alternative characteristics of the second portion of the content.

3. The method according to claim 2, wherein: The one or more characteristics of the first portion include that the first portion is an initial portion of the content, and wherein the one or more alternative characteristics of the second portion include that the second portion is a separate portion of the content, the separate portion being a non-initial portion of the content.

4. The method according to claim 3, further comprising: Before the audible rendering of said content: generating the first threshold based on a number and / or ratio of past occurrences of the fulfillment during past audible renderings having the one or more characteristics in historical usage data; and The first threshold is assigned to the one or more features.

5. The method according to claim 3 or claim 4, wherein: The content is a song, and the fulfillment includes causing rendering of the song to cease and causing rendering of a replacement song.

6. The method according to claim 3 or claim 4, wherein: The content is an item in a list of items, and the fulfilling includes causing rendering of the item to cease and causing rendering of a next item in the list of items.

7. The method according to claim 3 or claim 4, wherein: The content is a song, and the performing includes adjusting a volume of the audible rendering of the content.

8. The method according to claim 2, wherein: The one or more characteristics of the first part include that the first part is an ending part of the content, and wherein the one or more alternative characteristics of the second part include that the second part is a separate part of the content, the separate part being a non-ending part of the content.

9. The method according to claim 1, further comprising: Before the audible rendering of said content: generating the first threshold based on a number and / or ratio of past occurrences of the fulfillment during past audible renderings having the one or more characteristics in historical usage data; and The first threshold is assigned to the one or more features.

10. The method according to claim 1, wherein: Identifying the first threshold is based on the one or more characteristics of the audible rendering of the first portion of the content, and wherein identifying the second threshold is based on the one or more alternate characteristics of the audible rendering of the second portion of the content.

11. The method according to claim 10, wherein: The one or more characteristics of the audible rendering of the first portion of the content include a first volume of the audible rendering of the first portion of the content, and the one or more alternative characteristics of the audible rendering of the second portion of the content include a second volume of the audible rendering of the second portion of the content.

12. The method according to claim 11, wherein: The performing includes reducing a volume of the audible rendering of the content.

13. The method according to claim 11 or claim 12, wherein: The one or more microphones are of the client device, and wherein processing the audio data and processing the additional audio data are performed at the client device.

14. A method according to any preceding claim, wherein: The one or more characteristics of the first portion of the content include a loudness metric for the first portion of the content, and the one or more alternative characteristics of the second portion of the content include a second loudness metric different from the first loudness metric.

15. A method implemented by one or more processors of a client device, the method comprising: During audible rendering of content by one or more hardware speakers of the client device: processing audio data captured by one or more microphones of the client device using a warm word model to generate an output indicating whether the audio data includes utterance of one or more specific words and / or phrases, wherein the warm word model is trained to generate an output indicating whether the one or more specific words and / or phrases are present in the audio data; selecting a particular threshold from a plurality of candidate thresholds, wherein selecting the particular threshold is based on one or more characteristics of the content and / or the audible rendering of the content; and In response to selecting the particular threshold: determining whether the audio data includes utterance of the one or more specific words and / or phrases based on comparing the output to the specific threshold; When it is determined that the audio data includes speaking the one or more specific words and / or phrases: Causes execution of the execution corresponding to the warm word model.

16. The method according to claim 15, wherein: Selecting the particular threshold is based on the one or more characteristics of the content.

17. The method according to claim 16, wherein: The audible rendering of the content is in response to user input received in association with a user account, and wherein the one or more characteristics of the content include a match score for the content, wherein the match score is generated based on a degree of match of the content with the user input and / or a degree of match with user account characteristics of the user account.

18. The method according to claim 16 or 17, wherein: The one or more characteristics of the content include an overall popularity metric of the content.

19. The method according to any one of claims 16 to 18, wherein: The one or more characteristics of the content include one or more acoustic characteristics of an audio portion of the content.

20. The method according to claim 19, wherein: The one or more acoustic features include a root mean square (RMS) metric and / or a loudness unit full scale (LUFS) metric.

21. A client device, comprising: at least one processor; as well as A memory storing instructions which, when executed, cause the at least one processor to perform operations corresponding to any one of claims 1 to 20.

22. At least one non-transitory computer-readable medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 20.

Images