Adaptive management of casting requests and / or user input in a rechargeable device

Abstract

To provide adaptive management of casting requests and / or user input in rechargeable devices.SOLUTION: A rechargeable device (client device) provides access to an automated assistant and renders data to be casted from a separate device. Casting requests are processed by the rechargeable device even though a device SoC of the rechargeable device is operating in a sleep mode. Utterances given by a user to invoke the automated assistant are also adaptively managed by the rechargeable device to reduce power consumption during idle by the device SoC. Such utterances can be first processed by a digital signal processor, and based on one or more features of the utterance (voice characteristics, match with a particular calling phrase, etc.), the device SoC is initialized for an amount of time selected based on the features of the utterance.SELECTED DRAWING: Figure 1A

Description

[Background technology]

[0001] The limited charging capacity of a battery built into a portable electronic device may affect the usefulness of such a portable electronic device, especially given that each subsystem of such a device relies on at least a certain amount of energy to function. Furthermore, when a battery-powered device provides access to an automated assistant, energy resources may be further limited if the device is tasked with constantly processing audio data and / or other data for detection of a call signal that invokes the automated assistant. If the battery-powered assistant device includes a device system-on-chip (SoC), the device SoC may consume a significant proportion of energy compared to other subsystems (e.g., network processor, digital signal processor (DSP), etc.) when a user is interacting with the automated assistant. For example, a significant battery charge may be spent on the processor performing audio processing, which may include removing various audio data artifacts such as echo, static, and / or other noise.

[0002] Providing or streaming content from one device for rendering at another device may be referred to as "casting." A battery-powered portable electronic device capable of responding to casting requests may expend a significant amount of battery charge when operating to constantly process casting requests from local network devices. For example, a battery-powered device that accepts casting-related pings and / or requests for "casting" media to a battery-powered device may use the device SoC to process the data encompassed by the arriving requests. However, as such requests become more frequent and / or redundant, using the device SoC to process the requests may tighten the limitations of the battery-powered device's charge capacity. As a result, while the battery-powered device can still render the casted data, the total amount of available casting time is reduced as a result of how frequently the device SoC is required to process requests related to casting. Summary of the Invention [Means for solving the problem]

[0003] Implementations described herein relate to a rechargeable device that adaptively manages casting requests and / or user input while providing access to an automated assistant and / or one or more interfaces for rendering casted data provided by a separate computing device. Rechargeable devices may generally be limited by having a finite power source, such as a battery, that may be depleted by operating the rechargeable device to frequently process casting requests and user input (e.g., utterances such as call phrases). To extend the time between charges and also to eliminate the waste of other computational resources, the rechargeable device may use a variety of different subsystem operating schemes adapted to manage such requests and input.

[0004] For example, in some implementations, the rechargeable device may include a first processor, such as a digital signal processor (DSP), and a second processor, such as a device system-on-chip (SoC), for processing various inputs depending on the operating mode of the rechargeable device. The operating mode may be one of a number of operating modes, such as a sleep mode in which the device SoC is powered down or otherwise consumes less power than if the device SoC were operating in another operating mode (e.g., an operating mode in which an automated assistant may actively interact with a user via the rechargeable device). While the rechargeable device is operating in the sleep mode, the DSP may be powered on to monitor user input to the rechargeable device under permission from the user. As an example, the rechargeable device may include one or more microphones, and when the rechargeable device is operating in the sleep mode, the DSP may monitor any output provided by one or more of the microphones (e.g., output characterizing speech from the user into the microphones). The DSP may operate a speech recognition model (e.g., an invocation phrase model) to determine whether the user has provided an utterance corresponding to an invocation phrase (e.g., "Assistant...") to invoke the automated assistant. When the DSP uses the speech recognition model to determine that the user has provided an invocation phrase to invoke the automated assistant, the DSP may cause the device SoC to initialize for further processing. For example, the device SoC may initialize for a particular period of "wake time" to wait for further instructions and / or input from the user.

[0005] The amount of time that the device SoC remains active may vary over time depending on various characteristics associated with the interaction between one or more users and the automated assistant. The amount of time may be adapted and / or determined to reduce waste of computational resources and power that may otherwise be spent on running a voice recognition model in the device SoC. For example, the device SoC may run another voice recognition model (e.g., a first call phrase model and / or a voice activity detector) that is different from the voice recognition model run by the DSP (e.g., a second call phrase model and / or other voice activity detector) and may require more computational resources and / or power than the voice recognition model run by the DSP. Thus, by adapting the amount of "wake time" of the device SoC, unnecessary consumption of battery power may be prevented while still ensuring that the rechargeable device can provide the functionality of the automated assistant in an effective manner.

[0006] In some implementations, another voice recognition model operated by the device SoC may determine the user's voice characteristics based on data provided by the DSP, such as audio data generated in response to the user giving an initial utterance. Based on the determined voice characteristics, the device SoC may select a wake time during which the device SoC remains operable to proceed with processing any subsequent input from the user. As an example, a first user may typically and / or on average delay several seconds (e.g., 3 seconds) between giving an invocation phrase (e.g., "Assistant...") and giving a command phrase (e.g., "...play my study playlist."). The device SoC may acknowledge this delay and select a wake time for the device SoC that does not significantly exceed the average delay for the user. For example, the selected wake time for the device SoC may be, but is not limited to, (wake time = (user's determined average delay) x (1 + N)), where "N" is any number, such as, but not limited to, 0.2, 0.5, etc. The same or different wake times may be selected for different users who typically and / or on average delay several seconds (e.g., 2 seconds) between providing the alert phrase and providing the command phrase. In this manner, a rechargeable device, including a DSP and device SoC, may adaptively manage "wake times" for each user to ensure responsiveness without wasting power and / or computational resources.

[0007] In some implementations, the speech recognition model operated by the DSP and the other speech recognition model operated by the device SoC may both be used to determine whether the user has given a particular invocation phrase to invoke the automated assistant. However, the speech recognition model operated by the DSP may apply a less stringent standard for determining whether the user has given an invocation phrase than the standard enforced by the other speech recognition model operated by the device SoC. In other words, the speech recognition model may be associated with a first accuracy threshold for determining whether a particular utterance corresponds to an invocation phrase, and the other speech recognition model may be associated with a second accuracy threshold for determining whether the particular utterance corresponds to an invocation phrase. As an example, the first accuracy threshold may be met when the user gives a particular utterance that is determined to include at least a portion of the invocation phrase but also includes some amount of background noise. However, the particular utterance may not meet the second accuracy threshold because the second accuracy threshold may require a higher degree of correlation between the particular utterance and the invocation phrase compared to the degree of correlation to meet at least the first accuracy threshold.

[0008] In some implementations, the DSP may operate the speech recognition model using less power, less data, fewer channels of audio, lower sampling rate audio, and / or lower quality audio than the device SoC would use with another speech recognition model. For example, the DSP may receive a single channel of audio data when a user speaks to the rechargeable device, and the device SoC may receive multiple channels of audio data when a user speaks to the rechargeable device. Additionally or alternatively, the DSP may operate using an average amount of power when using the speech recognition model, and the device SoC may operate using a greater than average amount of power when using another speech recognition model.

[0009] In some implementations, the degree of correlation determined by the DSP and / or device SoC may be used to select an amount of wake time that the device SoC remains active to process further input from the user. For example, when the device SoC determines a first degree of correlation between an utterance and an invocation phrase, a first wake time may be selected by the device SoC. However, when the device SoC determines a second degree of correlation between another utterance and the invocation phrase, where the second degree of correlation is greater than the first degree of correlation, the device SoC may select a second wake time that is longer than the first wake time. In this way, the amount of time that the device SoC remains active to wait for further input from the user may be adapted depending on the accuracy and / or degree of correlation between the utterance from the user and the invocation phrase used to invoke the automated assistant. This may conserve computational resources in a rechargeable device by avoiding standard "wake times" that do not distinguish any content and / or context of the user input.

[0010] This may be particularly beneficial when the correlation between the utterance from the user and the invocation phrase falls short of the correlation required to invoke the automated assistant. This is because a "near miss" (i.e., a correlation that is almost enough but not quite enough to invoke the assistant) is more likely to have resulted from an actual attempt to invoke the automated assistant (rather than an utterance having a correlation that is far from the correlation required to invoke the assistant) and is therefore more likely to be followed by the user attempting to invoke the assistant again. Keeping the SoC active longer when a "near miss" is detected may allow the device to process subsequent invocation attempts with less latency. In other words, the device SoC may determine that the utterance falls short of correlating with the invocation phrase to a particular degree, and therefore the device SoC may remain in place for an amount of time based on the particular degree (e.g., an amount of time in seconds selected based on and / or proportional to the particular degree). However, when the device SoC determines that another utterance is less likely to correlate with the invocation phrase (e.g., less likely to a greater extent), the device SoC can shut down much more quickly to conserve power and computational resources.

[0011] When speech is detected by the rechargeable device and the device SoC is initialized for further processing, there may be differences in clock settings of a clock running in the DSP and another clock running in the device SoC. To further eliminate the waste of computational resources involved in waiting for speech received at the rechargeable device and / or responding to such speech, time synchronization may be performed in the DSP and / or the device SoC in order for the device SoC to process the audio data generated and time-stamped in the DSP. Such time synchronization may be particularly useful, for example, when the SoC is outputting audio when speech is received. Indeed, without time synchronization, processing the captured audio data to remove the audio output by the SoC from the data corresponding to speech may be problematic.

[0012] In some implementations, time synchronization may be performed by the device SoC using one or more timestamps generated in the device SoC and one or more other timestamps generated in the DSP. As an example, the DSP may generate a first timestamp corresponding to a local time associated with the DSP using a first clock. In addition, the DSP may generate a second timestamp, for example, when the DSP causes the device SoC to initialize in response to the DSP determining that a user has given an alert phrase. Upon receiving a signal from the DSP (e.g., a wake and / or interrupt command), the device SoC may generate a third timestamp using a second clock, where the third timestamp may correspond to a local time associated with the device SoC.

[0013] To perform time synchronization, the device SoC may generate a time offset using the first timestamp, the second timestamp, and the third timestamp, and then use the time offset when processing the audio data generated in the DSP. In some implementations, the device SoC may determine an average value of the first timestamp and the second timestamp, and then determine a delta value corresponding to the difference between the average value and the third timestamp. The delta value may then be used when the device SoC is processing the audio data, such as when the device SoC is performing echo cancellation. During echo cancellation, the device SoC may use the delta value to remove instances of audio being output by the rechargeable device from audio recorded by the microphone. As an example, when the device SoC is generating an audio output corresponding to music playback and a user provides speech into the microphone while the music is playing, audio data characterizing the speech may be processed by the device SoC to remove instances of the music playing. Furthermore, this process of removing instances of music playback from the audio data can be performed accurately using difference values ​​determined by the device SoC and / or DSP, thereby allowing the "wake time" of the device SoC to be determined from the accurate data. In other words, timestamps generated by the DSP can be transformed to correlate with timestamps generated by the device SoC for purposes of performing certain audio processes, such as echo cancellation. Additionally or alternatively, timestamps generated by the device SoC can be transformed to correlate with timestamps generated by the DSP for purposes of performing those audio processes.

[0014] In some implementations, the rechargeable device may include one or more interfaces to render audio, visual, haptic, and / or any other type of output in response to a casting request from another computing device. However, while such casting requests may be provided by other rechargeable devices, such as a cell phone and / or a laptop computer, the computing device providing the casting request may provide such requests without considering the available power in the rechargeable device. To handle frequent casting requests while still eliminating the waste of rechargeable power, the rechargeable device may offload the processing of certain requests to a subsystem of the rechargeable device that is not the device SoC. For example, the WiFi chip of the rechargeable device may be tasked with processing certain requests received on a local area network (LAN) to which the rechargeable device and the casting device are connected. In some implementations, the WiFi chip may process certain casting requests while the device SoC remains in a sleep mode to eliminate the waste of power and computational resources. The requests tasked to the WiFi chip for processing without invoking the device SoC for further processing may be casting requests that specify one or more specific ports. Additionally or alternatively, the WiFi chip may be tasked with processing the mDNS broadcasted data without invoking the device SoC.

[0015] As an example, a user may operate a music application on the user's cellular device to stream music, and while the music is playing, the user may initiate a cast of the music to the rechargeable device. The cellular device may send a casting request, which may include mDNS broadcast data, to various different devices connected to the LAN on which the rechargeable device resides. The rechargeable device may receive the casting request when the rechargeable device is operating in a sleep mode, in which the device SoC is asleep, off, or otherwise in a lower power mode than if the rechargeable device were not operating in a sleep mode. The WiFi chip of the rechargeable device may first process the casting request to determine whether the casting request specifies a particular port and / or includes a particular property.

[0016] When the casting request specifies a particular port that corresponds to one or more predetermined ports, the WiFi chip may avoid invoking the device SoC to respond to the casting request. More precisely, the WiFi chip may rely on cached data stored in the memory of the WiFi chip to generate response data to send back to the cellular device over the LAN. Additionally or alternatively, the WiFi chip may avoid invoking the device SoC if the mDNS broadcasted data included with the casting request specifies particular parameters of the casting request. For example, the mDNS broadcasted data provided by the cellular device may indicate that an audio playback service is requested and / or that a particular application initiated the casting request. The WiFi chip's cached data may indicate that the rechargeable device supports audio playback services and / or a particular application based on previous interactions with one or more other devices. Thus, based on the available cached data, the WiFi chip may generate a response to the cellular device using the cached data without invoking the device SoC for further information. In this way, the rechargeable device can reduce the number of times that the device SoC would otherwise be initialized for processing, thereby eliminating waste of the rechargeable power source (e.g., one or more batteries and / or capacitors) and computational resources.

[0017] The above description has been provided as a summary of some implementations of the present disclosure. Further descriptions of those and other implementations are provided in more detail below.

[0018] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include one or more computers and / or one or more robotic systems including one or more processors operable to execute the stored instructions to perform methods such as one or more of the methods described above and / or elsewhere herein.

[0019] It should be understood that all combinations of the above concepts, and additional concepts described in more detail herein, are contemplated as part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. [Brief description of the drawings]

[0020] [Figure 1A] FIG. 1 illustrates a user controlling a first client device to broadcast media to separate client devices without transitioning all client devices on a local network from their sleep states. [Figure 1B] FIG. 1 illustrates a user controlling a first client device to broadcast media to separate client devices without transitioning all client devices on a local network from their sleep states. [Figure 2A] FIG. 2 is a diagram of a scenario in which a device SoC is left in sleep mode, according to implementations discussed herein. [Figure 2B]FIG. 2 is a diagram of a scenario in which a device SoC is transitioned out of a sleep mode, according to implementations discussed herein. [Figure 2C] FIG. 2 is a diagram of a scenario in which a device SoC is transitioned out of a sleep mode, according to implementations discussed herein. [Diagram 3] FIG. 1 illustrates a client device that can determine a wake time for a device SoC based on one or more properties associated with an interaction between a user and the client device. [Figure 4] FIG. 1 illustrates a system for using a WiFi chip to generate wake times for a device SoC, generate a time offset between a DSP's clock and a device SoC's clock, and / or respond to casting requests without transitioning the device SoC out of a sleep mode. [Diagram 5] FIG. 1 illustrates a method for initializing a particular processor of a computing device for an amount of time selected based on one or more characteristics of an interaction between a user and the computing device. [Figure 6] FIG. 1 illustrates a method for processing audio data using a determined time offset corresponding to differences in operation between a first processor and a second processor. [Figure 7] FIG. 1 illustrates a method for providing response data to a broadcasting device using a WiFi chip included in a battery-powered computing device. [Figure 8] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] FIG. 1A illustrates a diagram 100 in which a user 124 controls a first client device 134 to broadcast media to a separate client device without transitioning all client devices on a local network from their client device sleep states. The first client device 134 can be a computing device such as a cellular phone 126 and / or any other device capable of casting media to another device. The first client device 134 can provide access to an automated assistant 136 that can be invoked by input to an assistant interface 138. The first client device 134 can also include one or more applications 144 that can access media that can be cast from the first client device 134 to another client device. In some implementations, the first client device 134 can cast media to a separate client device in response to an input from the user 124 to the automated assistant 136. For example, in response to the user consulting other devices available for casting media associated with the application 144, the first client device 134 can transmit mDNS data over a local area network to which multiple different client devices are connected.

[0022] The mDNS data broadcast by the first client device 134 may be transmitted to the second client device 102 and / or the third client device 112 via a local area network such as a WiFi network. For example, the mDNS data 130 may be transmitted from the first client device 134 to the second client device 102, and the mDNS data 132 may be transmitted from the first client device 134 to the third client device 112. In some implementations, the second client device 102 may be a portable computing device powered by a portable power source 110. Furthermore, the third client device 112 may be powered by a portable power source and / or any other power source, such as a power source provided by a utility service. The second client device 102 and the third client device 112 may operate in a sleep mode when each device receives the respective mDNS data. In other words, since the devices are operating in a sleep mode, an available WiFi chip in each device may process the mDNS data without transitioning the respective device from a sleep state. For example, when WiFi chip 106 and WiFi chip 116 receive and respond to mDNS data, device SoC 108 and device SoC 118 may operate in a sleep mode (as indicated by the gradient fill pattern). In some implementations, a computing device may be considered to be in a "sleep mode" when at least the device SoC of the computing device is powered down or is otherwise consuming less power than if the device SoC were operating according to another mode of operation.

[0023] The WiFi chip 106 of the second client device 102 may process the mDNS data 130 using cached data available in the memory 140 of the WiFi chip 106. Additionally, the WiFi chip 116 of the third client device 112 may process the mDNS data 132 using cached data available in the memory 142 of the WiFi chip 116. The mDNS data broadcast by the first client device 134 may identify an application associated with the broadcast, a port for sending the broadcast, a service being requested by the first client device 134, and / or any other characteristics that a computing device may specify when initializing casting.

[0024] FIG. 1B illustrates a diagram 150 in which the second client device 102 and the third client device 112 respond to the mDNS data sent to each client device. During the generation and transmission of the response data, each of the device SoC 108 and the device SoC 118 can remain in a sleep mode, thereby eliminating waste of power and computational resources. The response data 148 can indicate whether the second client device 102 includes one or more features requested by the first client device 134, and the response data 146 can indicate whether the third client device 112 includes one or more features requested by the first client device 134. In response to the first client device 134 receiving the response data 148 and the response data 146 via the local network, the first client device 134 can provide a graphical interface that identifies one or more client devices that fulfill the request. The user 124 can then select one of the client devices for casting media. For example, when an application 144 requests casting to a client device and the user is presented with a list of client devices to select from for casting, the user may select the first client device 134 .

[0025] Depending on the selection, the application 144 may communicate directly with the first client device 134 over a local network, or the application may communicate with a separate server to have the separate server communicate instructions to the first client device 134 to render certain media data by the first client device 134. In some implementations, the first client device 134 may be a standalone speaker device 122 and / or a display device capable of rendering audio and / or visual data. Alternatively or additionally, the third client device 112 may be a display device such as a computer monitor and / or a television. The second client device 102 and the third client device 112 may each include a digital signal processor, which may monitor the respective device interfaces that access the automated assistant when the respective device SoCs are operating in a sleep mode. Furthermore, the digital signal processor (DSP), WiFi chip, device SoC, and / or any other subsystem of the client device may operate according to any of the implementations discussed herein.

[0026] 2A illustrates a diagram 200 in which a user 220 provides an utterance 218 to a client device 202 and causes the client device 202 to process the utterance 218 using a digital signal processor without transitioning the device SoC 208 from a sleep mode. The client device 202 can be a computing device 222 operated by a power source 210, which can include a rechargeable power source, such as a battery, a capacitor, and / or any other rechargeable energy source. When the client device 202 is operating such that the device SoC 208 is in a sleep mode, the user 220 can provide the utterance 218, which can be different from the invocation phrase that causes the client device 202 to transition the device SoC 208 from a sleep mode. For example, the user 220 can provide the utterance 218 "Hello...", which can be received at one or more microphones connected to the client device 202.

[0027] A microphone connected to the client device 202 can provide an output in response to the user 220 providing speech 218. Even though the device SoC 208 operates in a sleep mode, the digital signal processor DSP 204 of the client device 202 can monitor the microphone output to determine whether the user has provided one or more invocation phrases that can invoke the client device 202 to perform one or more different actions. In some implementations, the DSP 204 can process the audio data 212 characterizing the speech 218 by a process that utilizes a lower sampling rate than the sampling rate used by the device SoC 208 to process the audio data. Alternatively or additionally, the DSP 204 can process the audio data 212 generated based on output from a smaller number of microphones compared to the number of microphones used to generate the audio processed by the device SoC 208. In other words, the DSP 204 can utilize fewer channels of audio data compared to the amount of channels utilized by the device SoC 208. Utilizing a lower sampling rate and / or fewer channels may be computationally efficient and minimize power consumption (and resulting battery consumption). Alternatively or additionally, the DSP 204 may access the first model 214 to process the audio data 212 to determine whether the user 220 has provided the call phrase.

[0028] The first model 214 may be different from the second model 216 used by the device SoC 208 to determine whether the user 220 said the invocation phrase. For example, the first model 214 may be a model trained to determine whether the audio data characterizes the invocation phrase. The correspondence between the audio data and the invocation phrase may be characterized as one or more values, and a threshold degree of similarity between the audio data and the invocation phrase may be lower than another threshold degree corresponding to the second model 216. In other words, an utterance may be determined to meet a threshold of the first model 214 but not a threshold of the second model 216, but an utterance may not be determined to meet the second model 216 and not the first model 214.

[0029] In various implementations, the second model 216 may be larger (in terms of bits) than the first model 214, have a larger input dimension (e.g., for processing more channels of audio data) and / or a larger amount of trained nodes. As a result, processing the audio data using the second model 216 may be more computationally expensive than processing the audio data using the first model 214. However, in some implementations, processing the audio data using the second model 216 may result in a more accurate determination of whether the user 220 said the call phrase as a result of the second model 216 being larger, more channels of audio data being processed, higher precision samples, and / or a higher sampling rate of the audio data being processed. Thus, the DSP 204 may utilize the more efficient first model 214 to determine whether the audio data passes an "initial check" for the presence of the call phrase, and only if the "initial check" is passed, the SoC 208 and the less efficient (but more accurate) second model 216 may be utilized. This is more resource efficient than utilizing only the SoC 208 and the second model 216 .

[0030] In some implementations, the DSP 204 may process the audio data at a different bit depth compared to the bit depth at which the device SoC 208 processes the audio data. For example, the DSP 204 may capture the audio data as 24-bit audio, but convert the audio data to 16-bit audio data and then use the 16-bit audio data when determining whether the audio data characterizes an alert phrase provided by a user. When the DSP 204 determines that the 16-bit audio data characterizes the alert phrase, the DSP 204 may cause the captured 24-bit audio data to be transferred to the device SoC 208. The device SoC 208 may then process the 24-bit audio data rather than converting to a different bit depth for processing the transferred audio data.

[0031] In response to the user 220 providing the utterance 218, the DSP 204 may process the audio data 212 using the first model 214 and determine that the utterance 218 does not correspond to an invocation phrase of the one or more invocation phrases. In response, the DSP 204 may avoid waking the device SoC 208 from sleep for further processing. In this manner, the device SoC 208 does not need to be initialized frequently to further process the audio data 212 and can remain in a sleep mode. This allows the client device 202 to avoid waste of energy provided by the power source 210 and computational resources available at the client device 202.

[0032] 2B illustrates a diagram 230 in which a user provides speech 234 to a client device 202, causing the DSP 204 of the client device 202 to wake up the device SoC 208 of the client device 202. The speech 234 may be captured by one or more microphones of the client device 202, which may be a computing device 222 operating from a portable and / or rechargeable power source 210. Initially, the device SoC 208 may operate in a sleep mode to conserve power and computational resources. While the device SoC 208 is operating in the sleep mode, the DSP 204 may operate to detect when the user 220 provides speech corresponding to one or more invocation phrases.

[0033] As an example, the user 220 may provide an utterance 234, such as "Assistant," which may correspond to an alert phrase that, when detected by the DSP 204, may cause the DSP 204 to wake the device SoC 208 from sleep. To detect the alert phrase, the DSP 204 may convert output from one or more microphones of the client device 202 into audio data 232. The DSP 204 may use the first model 214 to process the audio data 232 to determine whether the utterance 234 corresponds to the alert phrase. When the DSP 204 determines that the utterance 234 corresponds to the alert phrase, the DSP 204 may send a command to the device SoC 208 to wake up or otherwise transition the device SoC 208 from a sleep mode.

[0034] When the DSP 204 transitions the device SoC 208 from a sleep mode to an operating mode, the DSP 204 may also send audio data to the device SoC 208 for further processing. The device SoC 208 may then process the audio data using the second model 216 to determine whether the utterance 234 corresponds to the alerting phrase. When the device SoC 208 determines that the utterance 234 did not correspond to the alerting phrase, the device SoC 208 may transition back to a sleep mode to conserve computational resources and power. Alternatively or additionally, when the device SoC 208 determines that the utterance 234 does not correspond to the alerting phrase but the DSP 204 determines that the utterance 234 corresponds to the alerting phrase, the device SoC 208 may remain active or awake for a period of time in anticipation of at least further input from the user 220. In some implementations, the wake time may be based on a degree of correlation between the speech 234 and the invocation phrase, an identification of the voice of the user 220, and / or any other implementation features discussed herein. As an example, the wake time may be determined based on a comparison between a degree of correlation detected by the device SoC 208 and a threshold degree of correlation. For example, when the degree of correlation detected by the device SoC 208 is 0.87 and the threshold degree of correlation is 0.9, the wake time of the device SoC 208 may be set to a period of time X. However, if the degree of correlation detected by the device SoC 208 is 0.79 and the threshold degree of correlation is 0.9, the wake time of the device SoC 208 may be set to a period of time Y, where Y is less than X.

[0035] FIG. 2C illustrates a diagram 240 in which an utterance 244 is provided by a user 220 to cause a DSP 204 of a computing device 202 to transition the device SoC 208 from a sleep mode and further initialize an automated assistant for further operation. The client device 202 can be a computing device 222 that includes one or more different interfaces for interacting with an automated assistant. To initialize an automated assistant, the user 220 can provide an awakening phrase that is detected by the DSP 204 when the device SoC 208 is in a sleep mode. The awakening phrase can be included in the utterance 244 that is processed using the audio data 242 and the first model 214 when detected by the DSP 204. When the DSP 204 determines using the first model that the audio data 242 characterizes the awakening phrase, the DSP 204 can provide a wake command to the device SoC 208.

[0036] In response to the device SoC 208 receiving the wake command, the device SoC 208 can process audio data corresponding to the utterance 244 using the second model 216. Based on processing the audio data using the second model 216, the device SoC 208 can determine that the utterance 244 included an invocation phrase. Thus, based on the device SoC 208 determining that the user 220 provided the invocation phrase, the device SoC 208 can initialize the automated assistant locally and / or provide a network request to initialize the automated assistant via a server device. For example, the device SoC 208 can send data to the WiFi chip 106 of the client device 202 to initialize the automated assistant. The data can be sent to the automated assistant server via a network, such as the Internet, such that a subsequent request from the user can be sent to the automated assistant server via the client device 202. In some implementations, the automated assistant may be hosted on the client device 202, such that requests from a user 220 for the automated assistant to perform a particular action may be processed at the client device 202. By allowing the device SoC 208 to sleep to conserve power and other resources and wake up to verify a particular utterance from the user 220, the client device 202 can conserve computational and power resources, which may be particularly advantageous for a client device 202 that operates using a rechargeable power source 210.

[0037] FIG. 3 illustrates a diagram 300 of a client device 302 that can determine a wake time of the device SoC 308 based on one or more properties associated with an interaction between a user 320 and the client device 302. The user 320 can interact with the client device 302 to invoke an automated assistant to perform one or more different functions. For example, the client device 302 can be a standalone speaker device 322 that can render audio, such as music, and / or control various other client devices connected to a common network with the client device 302. The client device 302 can be controlled by multiple different users who have different ways of speaking and / or interacting with the client device 302. To accommodate such differences between users while still avoiding waste of power and computational resources, the client device 302 can determine a wake time 324 of the device SoC 308 to limit the amount of time the device SoC 308 monitors input from the user 320.

[0038] As an example, the user 320 may give an utterance 318 such as "Assistant, could you..." and then briefly pause to consider how to continue the utterance. The user 320 may have a habit or history of exhibiting such pauses when interacting with the client device 302. Thus, data characterizing previous interactions between the user 320 and the client device may be used to determine how long to monitor for further input from the user 320 without wasting resources of the client device 302. For example, in response to the utterance 318, the DSP 304 of the client device 302 may process the audio data 312 characterizing the utterance 318 to determine whether the audio data 312 characterizes an invocation phrase, such as "Assistant." When the DSP 304 determines that the utterance 318 includes an invocation phrase, the DSP 304 may communicate with the device SoC 308 to cause the device SoC 308 to transition from a sleep mode to an operating mode. In some implementations, the DSP 304 may also send audio data 312 to the device SoC 308 to confirm that the user 320 provided the call phrase.

[0039] In some implementations, when the device SoC 308 determines that the user 320 did indeed provide the call phrase, the device SoC 308 may further process the audio data 312 to identify the user who provided the utterance 318. For example, the device SoC 308 may access a voice identification model under permission from the user to identify one or more voice features encompassed by the audio data 312. Based on the voice features encompassed by the audio data 312, the device SoC 308 may rank one or more different users according to whether the utterance 318 corresponds to the particular voice features of those users. The highest ranked user may then be selected as the user who provided the utterance 318, and the device SoC 308 may determine the wake time 324 based on identifying the highest ranked user. Alternatively or additionally, a user may be selected by the device SoC 308 using one or more models that may be used to generate a prediction of the source of the utterance 318. Alternatively or additionally, the audio data 312 may be processed using one or more models, which may also be used to generate the wake time 324.

[0040] In response to determining that the user 320 has provided the invocation phrase, the device SoC 308 may communicate with the WiFi chip 306 to initialize the automated assistant over a wide area network, such as the Internet. However, in some implementations, the device SoC 308 may initialize the automated assistant via a local device that communicates with the client device 302 over a local area network. While the automated assistant is initializing, the device SoC 308 may monitor one or more interfaces of the client device 302 for an amount of time at least equal to the wake time 324. When the wake time 324 has elapsed, the device SoC 308 may return to a sleep mode and the DSP 304 may take over monitoring output from one or more interfaces of the client device 302.

[0041] In some implementations, the wake time 324 may be based on a determined degree of correlation between the utterance 318 and the invocation phrase. For example, the device SoC 308 and / or the DSP 304 may generate a value characterizing the degree of correlation between the utterance 318 and the invocation phrase. The amount of wake time 324 may decrease as the degree of correlation increases, and the amount of wake time 324 may increase as the degree of correlation decreases. In other words, the device SoC 308 determines that the utterance 318 falls within a 10% tolerance range of a threshold for confirming that the utterance 318 includes the invocation phrase, and the wake time 324 may be one minute. However, when the device SoC 308 determines that the utterance 318 does include the invocation phrase and therefore meets the threshold, the wake time 324 may be set to five seconds. It should be noted that the wake time may be any amount of milliseconds, seconds, minutes, and / or any other time value upon which a processor's operation may be based. For example, an utterance that is more closely correlated with the invocation phrase may result in a wake time that has fewer total milliseconds than a wake time that results from a different utterance that is less closely correlated with the invocation phrase.

[0042] 4 illustrates a system 400 for operating a computing device 418 to conserve computing resources by generating a wake time for the device SoC 444, generating a time offset between the clock of the DSP 442 and the clock of the device SoC 444, and / or using the WiFi chip 434 to respond to casting requests without transitioning the device SoC 444 out of a sleep mode. The automated assistant 404 may operate as part of an assistant application provided in one or more computing devices, such as the computing device 418 and / or the server device 402. A user may interact with the automated assistant 404 via an assistant interface, which may be a microphone, a camera, a touchscreen display, a user interface, and / or any other device that may provide an interface between a user and an application.

[0043] For example, a user may initialize the automated assistant 404 by providing verbal, textual, and / or graphical input to the assistant interface to cause the automated assistant 404 to perform a function (e.g., provide data, control peripheral devices, access agents, generate input and / or output, etc.). The computing device 418 may include a display device, which may be a display panel including a touch interface for receiving touch input and / or gestures to enable a user to control applications of the computing device 418 via a touch interface. In some implementations, the computing device 418 may not have a display device, thereby providing an audible user interface output without providing a graphical user interface output. Additionally, the computing device 418 may provide a user interface, such as a microphone, for receiving spoken natural language input from a user. In some implementations, the computing device 418 may include a touch interface and may not have a camera, but may optionally include one or more other sensors.

[0044] The computing device 418 and / or other computing devices 434 can communicate with the server device 402 over a network 440, such as the Internet. Additionally, the computing device 418 and other computing devices 434 can communicate with each other over a local area network (LAN), such as a WiFi network. The computing device 418 can offload computational tasks to the server device 402 to conserve computational resources at the computing device 418. For example, the server device 402 can host an automated assistant 404, and the computing device 418 can send inputs received at one or more assistant interfaces 420 to the server device 402. However, in some implementations, the automated assistant 404 can be hosted at the computing device 418 as a client automated assistant 422.

[0045] In various implementations, all or some aspects of the automated assistant 404 may be implemented on the computing device 418. In some of those implementations, aspects of the automated assistant 404 are implemented by a client automated assistant 422 on the computing device 418, which interfaces with a server device 402 that implements other aspects of the automated assistant 404. The server device 402 may optionally serve multiple users and their associated assistant applications via multiple threads. In implementations in which all or some aspects of the automated assistant 404 are implemented by a client automated assistant 422 on the computing device 418, the client automated assistant 422 may be an application separate from (e.g., installed "on") the operating system of the computing device 418--or may alternatively be implemented directly by (e.g., considered an application of, but integral with, the operating system) the operating system of the computing device 418.

[0046] In some implementations, the automated assistant 404 and / or the client automated assistant 422 can include an input processing engine 406 that can use multiple different modules to process input and / or output for the computing device 418 and / or the server device 402. For example, the input processing engine 406 can include a voice processing module 408 that can process audio data received at the assistant interface 420 to identify text contained in the audio data. The audio data can be transmitted from the computing device 418 to the server device 402, for example, to conserve computational resources of the computing device 418.

[0047] The process for converting audio data to text may include speech recognition algorithms that may use neural networks and / or statistical models to identify groups of audio data that correspond to words or phrases. The text converted from the audio data may be analyzed by the data analysis module 410 and made available to the automated assistant as text data that may be used to generate and / or identify command phrases from the user. In some implementations, the output data provided by the data analysis module 410 may be provided to a parameter module 412 to determine whether the user has given input corresponding to a particular action and / or routine that may be performed by the automated assistant 404 and / or an application or agent that may be accessed by the automated assistant 404. For example, assistant data 416 may be stored on the server device 402 and / or on the computing device 418 as client data 432 and may include data defining one or more actions that may be performed by the automated assistant 404 and / or the client automated assistant 422, and parameters required to perform the action.

[0048] In some implementations, the computing device may include a WiFi chip 434 that may include at least one or more portions of the memory 436 and / or a broadcast engine 438. The broadcast engine 438 may receive broadcasted data from one or more other client devices over the network 440 and generate response data using cached data stored in the memory 436. The WiFi chip 434 may store data characterizing available services, applications, hardware features, and / or any other properties and / or functions that may be associated with the computing device 418. When the computing device 418 is operating in a sleep mode in which the device SoC 444 consumes less power and / or computational resources than when the computing device 418 is operating in a wake mode, the WiFi chip 434 may respond to casting requests from other client devices without transitioning the device SoC 444 out of the sleep mode.

[0049] For example, when a request from a client device is received at the WiFi chip 434 and the request identifies a target service that is also characterized by data stored in the memory 436, the broadcast engine 438 can generate response data using cached data from the memory 436 and provide the response data to the client device. If the client device selects the computing device 418 to use the target service, the client device can send a command to the computing device 418, and the WiFi chip 434 can process the command and transition the device SoC 444 from a wake mode to an operational mode. However, if the broadcast engine 438 determines that the memory 436 does not include sufficient data to determine whether the computing device 418 can provide a particular service, initialize a particular application, and / or otherwise provide a service to the requesting client device, the WiFi chip 434 can communicate with the device SoC 444 to process the request. In this case, the device SoC 444 can generate response data and provide the response data to the WiFi chip 434, and the WiFi chip 434 can transmit the response data to the client device.

[0050] In some implementations, the computing device 418 includes one or more assistant interfaces 420 that can provide access to the client's automated assistant 422 and / or the automated assistant 404. A user can provide one or more different types of input to invoke the client's automated assistant 422 and / or the automated assistant 404. Such input can include a spoken input, which can be processed by the digital signal processor 442 when the device SoC 444 is operating in a sleep mode. One or more speech recognition models 440 available in the computing device 418 can be used to determine whether audio data characterizing the spoken input includes an invocation phrase for initializing the automated assistant. Additionally, the one or more speech recognition models 440 can be used by the wake time engine 448 to determine an amount of time the device SoC 444 should remain awake to detect subsequent input from the user. In some implementations, the amount of wake time can be based on a degree of similarity between the user's speech and the invocation phrase for invoking the automated assistant. Alternatively or additionally, the amount of wake time may be based on the audio processing engine 430 processing audio data corresponding to the utterance and identifying the user who gave the utterance. For example, the audio processing engine 430 may use the client data 432 and / or the assistant data 416 to determine characteristics of an interaction between a user and an automated assistant, such as how long a user typically pauses during an interaction with the automated assistant. The wake time engine 448 can use this information to generate a wake time for the device SoC 444 during a particular interaction between a user and an automated assistant.

[0051] Additionally or alternatively, the power engine 426 of the computing device 418 may determine an estimated charge of the power source 446 and communicate the estimated charge and / or amount of operating time to the wake time engine 448. The amount of charge and / or amount of operating time estimated by the power engine may be used by the wake time engine 448 to determine a wake time for the device SoC 444. For example, when a user generally takes longer than an average user to interact with an automated assistant and the power source 446 is fully charged, the wake time engine 448 may allocate an extended wake time at least as compared to the wake time that would otherwise be allocated if the estimated charge were less than 50%. Alternatively or additionally, when a user generally takes shorter than an average user to interact with the computing device 418 and the power source 446 is fully charged, the wake time engine 448 may allocate a shorter wake time to conserve power compared to the extended wake time based at least on the user's historical interactions.

[0052] In some implementations, the computing device 418 may include a time offset engine 424 to determine an offset between clocks used by the computing device 418. For example, the DSP 442 may operate a first clock and the device SoC 444 may operate a second clock that may be offset from the first clock during operation of the computing device 418. This offset may affect operations at the audio processing engine 430, particularly when the audio processing engine 430 is performing echo cancellation on spoken input to the assistant interface 420.

[0053] In some implementations, the offset between a first clock on which DSP 442 operates and a second clock on which device SoC 444 operates may be determined using a timestamp. The timestamp may correspond to a pair of clock values ​​including a clock value captured using the first clock and another clock value captured with the second clock. When DSP 442 is operating to determine whether an alert phrase is detected and device SoC 444 is in a sleep mode, DSP 442 may record a clock value corresponding to a “wake” time when the alert phrase is detected. When DSP 442 transitions device SoC 444 out of a sleep mode, a timestamp may be recorded using the first clock and the second clock. However, to determine a “wake” time expressed relative to the second clock, the second clock value of the timestamp may be “scaled” and / or otherwise adjusted according to the determined time offset between the first and second clocks.

[0054] The time offset may be determined using a first timestamp and a second timestamp that may be recorded when both the device SoC 444 and the DSP 442 are not both in a sleep mode. The first timestamp may correspond to a first pair of clock values, and the second timestamp may correspond to a second pair of clock values. The clock value of the first DSP of the first pair of clock values ​​may be subtracted from the clock value of the second DSP of the second pair of clock values ​​to generate a first clock difference value. Additionally, the clock value of the first SoC of the first pair of clock values ​​may be subtracted from the clock value of the second SoC of the second pair of clock values ​​to generate a second clock difference value. The mapping between the first clock difference value and the second clock difference value may then be used when the DSP 442 wakes the device SoC 444 from sleep to determine when the alert phrase was received. For example, a ratio of the second clock difference value to the first clock difference value can be determined, and the ratio can be multiplied by the DSP clock value to determine a corresponding device SoC clock value. For example, when the DSP 442 wakes the device SoC 444 from sleep, a DSP clock value corresponding to a time when the user gave the wake-up phrase can be provided to the device SoC 444. The device SoC 444 can then map the DSP clock value to the device SoC clock value to determine when the wake-up phrase was given by the user relative to the device SoC clock. This value can then be used during processing of the audio data, such as during echo cancellation, to analyze the content of the audio data (e.g., to identify the natural language content of an utterance from the user).

[0055] FIG. 5 illustrates a method 500 for initializing a particular processor of a computing device for an amount of time selected based on one or more characteristics of an interaction between a user and the computing device. The method 500 may be performed by one or more processors, applications, and / or any other devices and / or modules that may provide an interface between a user and an automated assistant. The method 500 may include an operation 502 of determining whether the first processor detects speech from the user. The first processor may be operable when the second processor is operating in a sleep mode. The sleep mode may be a mode in which the second processor consumes less power and / or fewer computational resources compared to an operating mode in which one or more applications, such as an automated assistant, are actively running by the second processor. In some implementations, the first processor may be a digital signal processor and the second processor may be a device SoC. Both the first processor and the second processor may be incorporated into a computing device operating using a rechargeable power source, such as a battery, a power storage, and / or any other rechargeable power source.

[0056] The method 500 may proceed to operation 504 when the first processor detects speech. Otherwise, when speech is not detected by the first processor, the first processor may continue to monitor one or more microphones of the computing device to determine whether the user has given speech. Operation 504 may include determining by the first processor whether the speech includes a particular invocation phrase. The computing device may be operative to transition the second processor from a sleep mode when a particular invocation phrase of one or more different invocation phrases is given to the computing device by the user. The invocation phrase may be, for example, "Assistant" and / or any other phrase that may be used to initialize an application. When the first processor determines that the speech includes an invocation phrase, the method 500 may proceed from operation 504 to operation 508.

[0057] Operation 508 may include transitioning the second processor from a sleep mode to an operating mode. Operation 508 may be performed by the first processor in response to the first processor identifying the invocation phrase. However, when the first processor determines that the utterance does not include the invocation phrase, the method 500 may proceed from operation 504 to operation 506. Operation 506 may include avoiding transitioning the second processor from a sleep mode to an operating mode. In other words, because the first processor does not detect the invocation phrase in the utterance, the first processor returns to operation 502 to determine whether another utterance has been detected.

[0058] The method 500 may proceed from operation 508 to operation 510, which may include providing audio data from the first processor to the second processor. The audio data may correspond to an utterance given by a user to the computing device. In some implementations, the first processor may operate a first invocation phrase model to determine whether the utterance includes an invocation phrase, while the second processor may operate a second invocation phrase model to determine whether the utterance included the invocation phrase. The first model may correspond to a lower threshold for identifying a correspondence between the utterance and the invocation phrase, while the second invocation phrase model may correspond to a higher threshold for determining a correspondence between the utterance and the invocation phrase compared to the threshold of the first model. Thus, when the second processor receives the audio data, the second processor may use the second invocation phrase model to determine whether the utterance includes the invocation phrase.

[0059] Method 500 may include an optional operation 512 of determining, by the second processor, the degree to which the audio data characterizes the call phrase. The degree to which the audio data characterizes the call phrase may be one or more metrics that quantify one or more similarities between the audio data and the call phrase. In this manner, the one or more metrics may be later used to make a decision about how to subsequently operate the computing device. For example, a value characterizing the degree to which the audio data characterizes the call phrase may be used to determine an amount of time to operate the second processor in an operational mode before transitioning back to a sleep mode (if no other audio data is passed to the second processor for processing).

[0060] In some implementations, the method 500 may include an optional operation 514 of determining, by the second processor, voice characteristics encompassed by the audio data. The second processor may operate a voice identification model that may be used to identify a user who provided the utterance (e.g., a corresponding user profile of the user) with permission from one or more users. For example, each user of a computing device may speak with a different and / or unique voiceprint, and based on these differences, the voice identification model may determine a ranking corresponding to a prediction of which user provided the utterance. The user corresponding to the highest ranking may be selected as the user who provided the utterance to the computing device. The identification of the user using the voice identification model may be used with the user's permission to determine an amount of time to operate the second processor in an operating mode rather than a sleep mode. The amount of time may be based on previous interactions between one or more users and the automated assistant. For example, a greater amount of time may be selected for a user who generally has a delay between providing an invocation phrase and providing a subsequent command, while a lesser amount of time may be selected for another user who generally has no delay between providing an invocation phrase and providing another subsequent command.

[0061] The method 500 may further include an operation 516 of operating in an operational mode by the second processor for an amount of time based on one or more characteristics of the interaction between at least the user and the computing device. For example, in some implementations, the amount of time may be based on the degree to which the audio data characterizes the invocation phrase. Alternatively or additionally, the amount of time may be based on an identification of a user who provided one or more voice characteristics and / or utterances embodying the audio data. Alternatively or additionally, the amount of time may be based on one or more contextual characteristics corresponding to the interaction between the user and the computing device, such as a time of day, a number of available computing devices, a network strength, a number of users within a certain proximity of the computing device, and / or any other characteristics that may be associated with the interaction between the user and the computing device.

[0062] FIG. 6 illustrates a method 600 for processing audio data using a determined time offset corresponding to a difference in operation between a first processor and a second processor. The method 600 may be executed by one or more processors, applications, and / or any other device or module capable of processing audio data. The first processor and the second processor identified in the method 600 may be incorporated in a computing device that is powered by a portable power source, such as a battery and / or a power storage device, and provides access to an automated assistant. The method 600 may include an operation 602 of determining, by the first processor, whether an utterance has been received at the computing device and / or another device in communication with the computing device. In particular, the first processor may process the output from the one or more microphones to determine whether a user has given an utterance to the one or more microphones. When the first processor determines that an utterance has not been received, the first processor may continue to monitor the output of the one or more microphones to determine whether an utterance has been received by one or more users.

[0063] When the first processor determines that the speech was detected, the method 600 may proceed from operation 602 to operation 604. Operation 604 may include determining whether the speech included an invocation phrase. The first processor may determine whether the speech included an invocation phrase by using a first invocation phrase model that may be executed by the first processor. In particular, the first invocation phrase model may be used to analyze the output of one or more microphones to determine whether the speech included an invocation phrase. When it is determined that the speech included an invocation phrase, the method 600 may proceed from operation 604 to operation 608.

[0064] Operation 608 may include transitioning the second processor from a sleep mode to an operating mode. Operation 608 may be initiated by the first processor in response to determining that the utterance included the call-up phrase. When the first processor determines that the utterance did not include the call-up phrase, method 600 may proceed from operation 604 to operation 606. Operation 606 may include avoiding transitioning the second processor from a sleep mode to an operating mode, and instead of transitioning the second processor from a sleep mode, method 600 may return to operation 602 to detect whether a subsequent utterance has been provided to one or more microphones.

[0065] The method 600 may further include an operation 610 of causing the computing device to render an audio output using the audio output data by the second processor. The audio output may be provided by one or more interfaces connected to the computing device. For example, the computing device may include one or more speakers for emitting audio and / or the computing device may communicate with another computing device including one or more speakers. The audio output data may be based on data received over a network to which the computing device is connected and / or based on data stored in a memory of the computing device. For example, the audio output may be music rendered using audio data corresponding to music stored in a memory device of the computing device. The audio output data may include or be associated with time data indicating a time when a portion of the audio was rendered by the computing device and / or output by one or more speakers.

[0066] The method 600 may proceed to operation 612 of determining whether an invocation phrase is detected using a second processor. In some implementations, the first processor may be a digital signal processor and the second processor may be a device SoC. The first processor may operate a first speech recognition model and the second processor may operate a second speech recognition model. The first speech recognition model may have a lower threshold for determining whether an utterance includes an invocation phrase and the second speech recognition model may have a higher threshold for determining whether an utterance includes an invocation phrase. In some implementations, the first processor may process audio data of lower quality than the audio data processed by the second processor. For example, the first processor may monitor the output of one or more microphones of the computing device at a lower sampling rate compared to the sampling rate at which the second processor monitors the one or more microphones. Alternatively or additionally, the first processor may monitor a smaller total number of audio channels compared to the number of audio channels monitored by the second processor. For example, a first processor may monitor a single microphone to determine whether speech has been provided by the user, and a second processor may monitor two or more microphones to determine whether speech and / or an invocation phrase has been provided by the user.

[0067] The second processor may monitor the output of one or more microphones while the audio output is rendered by the computing device. When the second processor determines that the call phrase was provided by the user, the method 600 may proceed from operation 612 to operation 614. When the second processor does not determine that the call phrase was provided by the user, the second processor may continue to monitor the output of one or more microphones of the computing device. Operation 614 may include determining, by the second processor, a time offset between the time data and the audio input data that characterizes the call phrase detected by the second processor. In some implementations, the time offset may be based on a difference between clock operation characteristics of a clock of the first processor and another clock of the second processor. However, in some implementations, the first processor and the second processor may operate by a single clock.

[0068] The method 600 may further include an operation 616 of processing the audio input data by the second processor using a time offset to advance removal of one or more features of the audio input data. In particular, the time offset may be used during echo cancellation to remove features of the rendered audio output from the audio input provided to one or more microphones. By considering the time offset between the first processor and the second processor, errors that would otherwise be evident during echo cancellation may be removed. This may lead to a shorter latency between the user providing an utterance and the automated assistant responding to the utterance. Furthermore, because the computing device operates from a rechargeable power source, the operating time of each full charge of the power source may be extended by reducing the latency and total operating time with respect to at least the second processor.

[0069] FIG. 7 illustrates a method 700 for providing response data to a broadcasting device using a WiFi chip included in a battery-powered computing device. The method may be executed by one or more applications, processors, and / or any other devices or modules capable of processing network data. The method 700 may include an operation 702 of determining whether mDNS broadcast data is received at the WiFi chip. When it is determined that the mDNS broadcast data is received at the WiFi chip, the method 700 may proceed to an operation 704. The operation 704 may include determining whether a specific target port is identified by the mDNS broadcast data. When the mDNS broadcast data is not received at the WiFi chip at operation 702, the WiFi chip may continue to monitor network traffic to determine whether any packets of data received at the WiFi chip correspond to the mDNS broadcast data.

[0070] When the mDNS broadcast data identifies a particular target port, such as a port designated for casting media among client devices, method 700 may proceed from operation 704 to operation 706. Operation 706 may include determining whether cached data stored in a memory of the WiFi chip characterizes one or more characteristics of the mDNS broadcast data. When the mDNS broadcast data does not identify a particular target port, method 700 may proceed from operation 704 to operation 702, where the WiFi chip may continue to monitor network traffic.

[0071] In some implementations, in operation 706, the WiFi chip may compare the mDNS broadcast data to cached data stored in the memory of the WiFi chip. For example, the WiFi chip may store packets of data previously provided over the network and / or data generated in response to packets received over the network. For example, the WiFi chip may have previously responded to a cast request from another broadcast device by indicating that the computing device including the WiFi chip includes an application that is also included in the other broadcast device. Alternatively or additionally, the data stored in the memory of the WiFi chip may indicate whether one or more services may be used by the computing device by the broadcast request. Alternatively or additionally, the data stored in the memory of the WiFi chip may indicate one or more hardware characteristics of the computing device. Alternatively or additionally, the WiFi chip may determine whether the cached data stored by the WiFi chip characterizes one or more characteristics associated with the mDNS broadcast data. In this manner, the WiFi chip may respond to requests broadcast over the network without waking up another processor of the computing device, such as the device SoC.

[0072] Method 700 may proceed from operation 706 to operation 708, which may include generating response data based on the cached data. Operation 708 may be executed when the WiFi chip has cached data that characterizes one or more features associated with the mDNS broadcast data. For example, when the cached data identifies an application that is the subject of the mDNS broadcast, the WiFi chip may generate response data to indicate to the broadcasting device that the computing device does include that particular application. In this manner, the computing device does not need to wake up another processor to respond to the broadcast data, thereby eliminating waste of computational and / or power resources that may be limited with respect to battery-powered devices.

[0073] When the cached data of the WiFi chip does not characterize one or more features associated with the mDNS broadcast data, the method 700 may proceed from operation 706 to operation 710. Operation 710 may include transitioning a device SoC of the computing device from a first mode of operation to a second mode of operation. In some implementations, the first mode of operation can be a mode in which the device SoC is running fewer processes compared to the second mode of operation. Alternatively or additionally, the first mode of operation may correspond to less power consumption by the device SoC compared to the power consumption of the device SoC when operating in the second mode of operation.

[0074] Method 700 may proceed from operation 708 and / or operation 710 to operation 712. Operation 712 may include causing the computing device to transmit the broadcasted response data and / or other response data. The other response data may be generated at least in part by the device SoC when operation 712 is performed. For example, when the cached data does not identify a particular feature, such as a service, associated with the mDNS broadcast data, the device SoC may be used to generate other response data that may identify one or more features associated with the mDNS broadcast data using data accessible to the device SoC. In some implementations, the cached data may be updated by the WiFi chip and / or the device SoC when the WiFi chip is tasked with transmitting data that was otherwise inaccessible via the memory of the WiFi chip. In this manner, subsequent queries or requests from other client devices over the network may be responded to by the WiFi chip without waking up the device SoC, thereby eliminating waste of power and computational resources.

[0075] 8 is a block diagram of an exemplary computer system 810. Generally, the computer system 810 includes at least one processor 814 that communicates with several peripheral devices via a bus subsystem 812. These peripheral devices may include, for example, a storage subsystem 824 including a memory 825 and a file storage subsystem 826, a user interface output device 820, a user interface input device 822, and a network interface subsystem 816. The input and output devices enable user interaction with the computer system 810. The network interface subsystem 816 provides an interface to external networks and is coupled to corresponding interface devices of other computer systems.

[0076] The user interface input devices 822 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 810 or a communications network.

[0077] The user interface output devices 820 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for generating a visible image. The display subsystem may also provide a non-visual display, such as through an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from computer system 810 to a user or to another machine or computer system.

[0078] The storage subsystem 824 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 824 may include logic for performing selected aspects of the methods 500, 600, 700, and / or for implementing one or more of the first client device 134, the second client device 102, the third client device 112, the client device 202, the client device 302, the server device 402, the computing device 418, and / or any other engines, modules, chips, processors, applications, etc. discussed herein.

[0079] These software modules are generally executed by the processor 814 alone or in combination with other processors. The memory 825 used in the storage subsystem 824 may include several memories including a main random access memory (RAM) 830 for storing instructions and data during execution of the program, and a read-only memory (ROM) 832 in which certain instructions are stored. The file storage subsystem 826 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular implementation may be stored by the file storage subsystem 826 in the storage subsystem 824 or other machines that may be accessed by the processor 814.

[0080] Bus subsystem 812 provides a mechanism for allowing the various components and subsystems of computer system 810 to communicate with each other as intended. Although bus subsystem 812 is shown diagrammatically as a single bus, alternative implementations of the bus subsystem could use multiple buses.

[0081] Computer system 810 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system 810 shown in Figure 8 is intended only as a specific example intended to illustrate some implementations. Many other configurations of computer system 810 are possible having more or fewer components than the computer system shown in Figure 8.

[0082] In situations where the systems described herein may collect or utilize personal information about users (or as is often referred to herein as "participants"), the user may be given the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how content that may be more relevant to the user should be received from a content server. Also, certain data may be processed in one or more ways before the data is stored or used, such that personally identifiable information is removed. For example, the user's identity may be processed such that personally identifiable information cannot be determined about the user, or if geographic location information is obtained, the user's geographic location may be generalized (such as to the level of a city, zip code, or state) and thus the user's specific geographic location cannot be determined. Thus, the user may be able to control how information is collected and / or used about the user.

[0083] Although several implementations have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining one or more of the results and / or advantages described herein may be utilized, and each such change and / or modification is deemed to be within the scope of the implementations described herein. More broadly, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications in which the teachings are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation many equivalents to the specific implementations described herein. Thus, the implementations described above are presented merely as examples, and it will be understood that within the scope of the appended claims and their equivalents, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is within the scope of the disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

[0084] In some implementations, the method is described as including operations such as processing an output of a microphone in a first processor of a computing device, where the output corresponds to an utterance provided by a user into the microphone, and the computing device includes a second processor operating in a sleep mode when the utterance is provided by the user. The method may further include determining in the first processor whether the output corresponds at least in part to an invocation phrase for invoking an automated assistant that may be accessed via the computing device. The method may further include transitioning the second processor from a sleep mode to an operating mode by the first processor when the first processor determines that the output at least partially corresponds to the call phrase; providing by the first processor to the second processor data characterizing the microphone output; determining by the second processor an extent to which the data characterizes the call phrase based on the data received from the first processor; determining by the second processor an amount of wake time for which the second processor will remain in the operating mode based on the extent to which the data characterizes the call phrase; and operating the second processor in the operating mode for at least the amount of wake time in response to determining the amount of wake time for the second processor.

[0085] In some implementations, the method may further include receiving, at the second processor, additional data from the first processor characterizing a distinct utterance from the user or another user when the second processor is operating in the operational mode for at least the amount of wake time, and causing the automated assistant to respond to the distinct utterance based on the additional data by the second processor. In some implementations, the first processor operates a first speech recognition model, and the second processor operates a second speech recognition model different from the first speech recognition model. In some implementations, the first speech recognition model is associated with a first accuracy threshold for determining a different degree to which the data characterizes the invocation phrase, and the second speech recognition model is associated with a second accuracy threshold for determining a different degree to which the data characterizes the invocation phrase, different from the first accuracy threshold. In some implementations, the second accuracy threshold is satisfied by a greater degree of correlation between the spoken input and the invocation phrase, the greater degree of correlation being relative to the degree of correlation to satisfy the first accuracy threshold.

[0086] In some implementations, the first processor is a digital signal processor (DSP) and the second processor is a device system-on-chip (SoC), and the computing device includes one or more batteries that provide power to the first processor and the second processor when the device SoC is in the operating mode. In some implementations, determining an amount of wake time for which the second processor remains in the operating mode includes identifying a previously determined amount of wake time designated for the second processor, the previously determined amount of wake time being based on one or more interactions between the user and the automated assistant prior to the user providing an utterance. In some implementations, the method may further include avoiding, by the first processor, transitioning the second processor from the sleep mode to the operating mode when the first processor determines that the output does not at least partially correspond to the invocation phrase.

[0087] In some implementations, the method may further include, when the first processor determines that the output at least partially corresponds to the call phrase, determining by the second processor a user's voice characteristic characterized by the microphone output based on the data characterizing the microphone output, wherein determining an amount of wake time for the second processor to remain in the operating mode is further based on the user's voice characteristic characterized by the microphone output.

[0088] In other implementations, the method is described as including operations such as processing the microphone output in a first processor of a computing device, where the output corresponds to an utterance provided by a user to the microphone, and the computing device includes a second processor operating in a sleep mode when the utterance is provided by the user. In some implementations, the method may further include determining in the first processor whether the output corresponds at least in part to an invocation phrase for invoking an automated assistant that may be accessed via the computing device. In some implementations, the method may further include transitioning the second processor from a sleep mode to an operating mode by the first processor when the first processor determines that the output corresponds at least in part to the invocation phrase, determining by the second processor voice features characterized by the microphone output, determining by the second processor an amount of wake time for which the second processor will remain in the operating mode based on the voice features characterized by the output, and operating the second processor in the operating mode for at least the amount of wake time based on determining the amount of wake time for the second processor.

[0089] In some implementations, the method may further include: determining, at a first processor of the computing device, that after the second processor operates in the operational mode, when the second processor is thereafter operating in the sleep mode, that a further output from the microphone corresponds at least in part to an invocation phrase for invoking the automated assistant, where the other input is provided in response to a separate user providing a separate utterance into the microphone; transitioning the second processor from the sleep mode to the operational mode by the first processor; determining, by the second processor, a further voice characteristic characterized by the further output from the microphone based on the further output; determining, by the second processor, a further amount of wake time for which the second processor will remain in the operational mode based on the voice characteristic characterized by the further output, where the further amount of wake time is different from the amount of wake time; and operating the second processor in the operational mode for at least the further amount of wake time based on determining the amount of wake time for the second processor.

[0090] In some implementations, the second processor operates the voice feature model when determining whether speech is provided to the microphone by the user and / or a separate user. In some implementations, the computing device includes one or more batteries that provide power to the first processor and the second processor when the second processor is operating according to the operating mode. In some implementations, the amount of wake time is based on one or more interactions between the user and the automated assistant before the user provides the utterance.

[0091] In yet other implementations, a method is described as including an operation such as determining, by a processor of the computing device, that input to a microphone of the computing device corresponds at least in part to an invocation phrase for invoking an automated assistant that may be accessed via the computing device. The method may further include transitioning, by the processor, another processor of the computing device from a sleep mode to an operating mode based on the input to the microphone. The method may further include generating, by the other processor, first data characterizing an audio output provided by the computing device through one or more speakers in communication with the computing device after the other processor transitions from a sleep mode to an operating mode, the first data including first time data characterizing a time at which the other processor generated the first data; determining, by the processor, that a other input was provided to a microphone of the computing device; generating, by the processor, second data characterizing the other input to the microphone of the computing device, the second data including second time data characterizing another time at which the processor generated the second data; determining, by the other processor, a time offset between the time at which the other processor generated the first data and the other time at which the processor generated the second data; processing, by the other processor, the second data using the time offset to proceed removing one or more features of the audio output provided by the one or more speakers; and determining, by the other processor, whether the other input to the microphone corresponds to an utterance to invoke an automated assistant that may be accessed via the computing device based on processing the second data using the time offset.The method may further include, when the other input to the microphone is determined to correspond to an utterance to invoke the automated assistant, causing the other processor to provide a response output via an interface in communication with the computing device.

[0092] In some implementations, processing the second data using the time offset to advance removal of one or more characteristics of the audio output includes performing an acoustic echo cancellation (AEC) process using the second data and the audio data. In some implementations, the time offset corresponds to a difference in a characteristic of a clock operation of a clock of the processor and another clock of the other processor. In some implementations, the time offset is based on a difference between a first clock value determined using the clock and a second clock value determined using the other clock. In some implementations, the first clock value and the second clock value are determined when the other processor is in an operating mode. In some implementations, the time offset is determined by multiplying a ratio of a difference between the clock values ​​by a time value corresponding to the other time. In some implementations, the computing device includes one or more batteries that provide power to the processor and the other processor when the other processor is operating according to the operating mode. In some implementations, the processor is a digital signal processor (DSP) and the other processor is a device system on chip (SoC).

[0093] In yet other implementations, a method is described as including acts such as receiving multicast Domain Name System (mDNS) broadcasted data from a broadcasting device at a WiFi chip of a computing device, where the computing device includes a device system-on-chip (SoC) operating in a first operating mode when the WiFi chip of the computing device receives the mDNS broadcasted data. The method may further include determining, by the WiFi chip based on the mDNS broadcasted data, whether a target port identified by the mDNS broadcasted data corresponds to a particular port that may be accessed via the computing device. The method may further include, when the target port identified by the mDNS broadcasted data corresponds to a particular port that can be accessed via the computing device, accessing cached broadcast device data stored in a memory device accessible to the WiFi chip when the device SoC is operating in a first operating mode, based on the target port corresponding to the particular port; determining based on the cached broadcast device data stored in the memory whether the cached broadcast device data characterizes one or more characteristics of the broadcast device specified by the mDNS broadcasted data; and, when the cached broadcast device data characterizes one or more characteristics of the broadcast device, generating response data based on the cached broadcast device data; and transmitting the response data to the broadcast device.

[0094] In some implementations, the method may further include transitioning the device SoC from a first operating mode to a second operating mode based on the cached broadcast device data not characterizing the one or more features when the target port identified by the mDNS broadcasted data corresponds to a particular port that can be accessed via the computing device and when the cached broadcast device data does not characterize the one or more features of the broadcast device, where the second operating mode is associated with a higher power consumption by the device SoC compared to the power consumption of the device SoC when operating in the first operating mode.

[0095] In some implementations, the computing device includes one or more batteries that provide power to the WiFi chip and the device SoC when the device SoC operates according to the second mode of operation. In some implementations, determining whether the cached broadcast device data characterizes one or more characteristics of the broadcast device includes determining whether the cached broadcast device data identifies an application that initiated transmission of the mDNS broadcast data from the broadcast device. In some implementations, determining whether the cached broadcast device data characterizes one or more characteristics of the broadcast device includes determining whether the cached broadcast device data identifies a service requested by the broadcast device. In some implementations, the method may further include causing the device SoC to generate other response data based on the mDNS broadcast data and transmitting the other response data to the broadcast device by the WiFi chip when a target port identified by the mDNS broadcast data corresponds to a specific port that may be accessed via the computing device and when the cached broadcast device data does not characterize one or more characteristics of the broadcast device. [Explanation of symbols]

[0096] 100 Figures 102 Second client device 106 WiFi chips 108 Device SoC 110 Portable Power Source 112 3rd Client Device 116 WiFi chip 118 Device SoC 122 Standalone speaker device 124 users 126 Cellular Phone 130 mDNS Data 132 mDNS Data 134 First Client Device 136 Automated Assistants 138 Assistant Interface 140 Memory 142 Memory 144 Applications 146 Response Data 148 Response Data 150 Figures 200 Figures 202 Client Devices 204 Digital Signal Processor DSP 208 Device SoC 210 Power supply 212 Audio Data 214 First Model 216 Second Model 218 utterances 220 users 222 Computing Devices 230 Figure 232 Audio Data 234 utterances 240 Figures 242 Audio Data 244 utterances 300 Figures 302 Client Device 306 WiFi Chip 308 Device SoC 312 Audio Data 318 utterances 320 users 322 Standalone speaker device 324 Wake Time 400 Systems 402 Server Device 404 Automated Assistant 406 Input Processing Engine 408 Audio Processing Module 410 Data Analysis Module 412 Parameter Module 416 Assistant Data 418 Computing Devices 420 Assistant Interface 422 Client Automated Assistant 424 Time Offset Engine 426 Electric Engine 430 Audio Processing Engine 432 Client Data 434 WiFi chips and other computing devices 436 Memory 438 Broadcast Engine 440 Network, speech recognition model 444 Device SoC 446 Power supply 448 Wake Time Engine 500 ways 600 ways 810 Computer Systems 812 Bus Subsystem 814 Processor 816 Network Interface Subsystem 820 User Interface Output Device 822 User Interface Input Devices 824 Storage Subsystem 825 Memory 826 File Storage Subsystem 830 Main Random Access Memory (RAM) 832 Read-Only Memory (ROM)

Claims

1. determining, by a processor of the computing device, that an input to a microphone of the computing device corresponds at least in part to an invocation phrase for invoking an automated assistant that can be accessed via the computing device; transitioning, by the processor, another processor of the computing device from a sleep mode to an operating mode based on the input to the microphone; After the other processor transitions from the sleep mode to the operating mode, generating, by the other processor, first data characterizing an audio output provided by the computing device through one or more speakers in communication with the computing device; the first data including first time data characterizing a time at which the other processor generated the first data; determining, by the other processor, that another input including a call phrase is provided to the microphone of the computing device; generating, by the other processor, second data characterizing the other input to the microphone of the computing device; the second data including second time data characterizing a different time at which the different processor generated the second data; determining, by the other processor, a time offset between the time that the other processor generated the first data and the other time that the other processor generated the second data; processing the second data by the further processor using the time offset to facilitate removal of one or more characteristics of the audio output provided by the one or more speakers; determining, by the other processor, whether the other input to the microphone corresponds to an utterance for invoking the automated assistant that may be accessed via the computing device based on processing the second data using the time offset; When it is determined that the other input to the microphone corresponds to the utterance for invoking the automated assistant, and causing the other processor to provide a response output via an interface in communication with the computing device.

2. 2. The method of claim 1, wherein processing the second data using the time offset to facilitate removal of one or more features of the audio output includes performing an acoustic echo cancellation (AEC) process using the second data and audio data.

3. 3. A method according to claim 1 or 2, wherein the time offset corresponds to a difference in clock operation characteristics of a clock of the processor and another clock of the other processor.

4. 4. The method of claim 3, wherein the time offset is based on a difference between a first clock value determined using the clock and a second clock value determined using the other clock.

5. 5. The method of claim 4, wherein the first clock value and the second clock value are determined when the other processor is in the operating mode.

6. 5. The method of claim 4, wherein the time offset is determined by multiplying a ratio of a difference between clock values ​​by a time value corresponding to the other time.

7. 7. The method of claim 1, wherein the computing device includes one or more batteries that provide power to the processor and the other processor when the processor is operating in the operating mode.

8. 8. The method of claim 1, wherein the processor is a digital signal processor (DSP) and the further processor is a device system on chip (SoC).

9. one or more speakers; A microphone and one or more batteries; At least two processors for carrying out the method according to any one of claims 1 to 8; A portable computing device comprising:

10. A computer-readable storage medium comprising instructions that, when executed by two or more processors, cause the two or more processors to perform the method of any one of claims 1 to 8.

Images