Efficient, low-latency automated assistant control for smart devices

Abstract

To provide methods, devices and programs for efficiently and with low latency controlling automatic assistants on smart devices.SOLUTION: A method includes: processing audio data that captures speech so that an assistant client device generates a recognized text of the speech; determining that the text matches the text of the cache entry; selecting a locally interpretable semantic representation based on the client device being mapped to the text of the cache entry; determining whether a state change of the locally interpretable semantic representation is the state change that depends on the current state; using the state change and optionally using an adapter identified based on the cache entry, to generate a local control command, and sending a local command through an optionally selected local channel based on the cache entry.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to efficient, low latency automated assistant control of smart devices. [Background technology]

[0002] Humans can engage in human-computer interactions using interactive software applications referred to herein as "automated assistants." For example, a human (who may be referred to as a "user" when interacting with an automated assistant) provides input to the automated assistant that can cause the automated assistant to generate and provide a response output to control one or more smart devices and / or perform one or more other functions. The input provided by the user may be, for example, touch input (e.g., via a touchscreen), gestures (e.g., detected via a camera), and / or spoken natural language input (i.e., utterances detected via a microphone), which may optionally be converted to text (or other semantic representation) and then further processed.

[0003] In many cases, an automated assistant includes an automated assistant client that runs locally on an assistant client device and is utilized directly by a user, and a cloud counterpart that leverages the more robust resources of the cloud to assist the automated assistant client in responding to user input. For example, the automated assistant client can provide its cloud counterpart with an audio recording of the user's speech (or a transcription thereof) and, optionally, data indicating the user's identity (e.g., credentials). The cloud counterpart can perform various processing on the query to return results to the automated assistant client, which can then provide the corresponding output to the user.

[0004] Many users may utilize automated assistants using multiple client devices. For example, some users own a coordinated “ecosystem” of client devices, such as smartphones, tablet computers, vehicle computing systems, wearable computing devices, smart TVs, standalone assistant-centric interactive speakers, and interactive displays with standalone assistant-centric speakers, among other client devices. A user can use any of these client devices to engage in human-computer interactions with an automated assistant (assuming an automated assistant client is installed). In some cases, these client devices may be scattered throughout a user's primary residence, secondary residence, workplace, and / or other structures. For example, mobile client devices, such as smartphones, tablets, and smartwatches, may be worn by the user and / or located where the user last left them. Other client devices, such as traditional desktop computers, smart TVs, and standalone assistant-centric devices, may be more fixed but may be located in various locations (e.g., rooms) within a user's home or workplace.

[0005] Techniques exist for enabling a user (e.g., a single user, multiple users in a household, a coworker, a roommate, etc.) to utilize any one automated assistant client of a coordinated ecosystem of client devices to control any one of multiple smart devices associated with the automated assistant client. For example, a user can issue a voice command to the automated assistant client on a client device to "turn off the living room light" to cause a corresponding smart device (i.e., a light linked to the automated assistant client and labeled "living room" light) to turn off the light.

[0006] When controlling a smart device in response to a user input received at a client device, many existing techniques transmit data corresponding to the user input to a remote automated assistant server via the Internet. The remote automated assistant server processes the data to determine a smart device to be controlled based on the request, generates a control request for the smart device based on the request, and transmits the control request via the Internet to a server of a party associated with the smart device (e.g., the manufacturer of the smart device). When generating the control request, the remote automated assistant server may process the data (e.g., using a natural language understanding component) to generate a semantic representation of the data and then generate the control request based on the semantic representation. The server of the separate party receives the control request and then transmits the corresponding control command to the smart device via the Internet through a hub coexisting with the smart device (e.g., in the case of BLE, Z-Wave, ZigBee, etc.) or directly to the smart device via an IP connection (e.g., in the case of Wi-Fi and other smart devices that do not require a hub). Summary of the Invention [Problem to be solved by the invention]

[0007] However, such techniques present drawbacks, such as high latency, excessive use of client and / or server resources, and / or excessive use of network resources. For example, high latency and / or excessive use of server resources may be the result of the remote assistant server needing to process received audio data and / or text to generate a semantic representation of the corresponding utterance and then generate a control request based on the generated semantic representation. Additionally or alternatively, high latency may be the result of transmitting requests from the remote assistant server to the separate party server, often exacerbated by the remote assistant server and the separate party server not being geographically close to each other. Also, for example, excessive use of network resources may be the result of transmitting high-bandwidth audio data and / or text from the client device to the remote assistant server and / or transmitting high-bandwidth response text, voice, and / or graphics (e.g., confirming requested changes to the smart device) from the remote assistant server to the client device. [Means for solving the problem]

[0008] Recently, efforts have been made to shift the performance of various automated assistant functions from cloud automated assistant components to the automated assistant client device (i.e., the client device that provides the automated assistant interface). For example, recent efforts have been made to generate speech recognition models that can be utilized within the constraints of the hardware resources (e.g., memory resources, processor resources, etc.) of various client devices while still achieving good results. Similarly, efforts have been made to shift natural language understanding (NLU) and / or fulfillment (e.g., performance of actions in response to user input) on the device. Executing such functions on the device provides various technical advantages, such as reduced latency (e.g., due to the absence of client-server round trips), reduced network usage (e.g., due to the absence of client-server round trips), and / or the ability to operate in low or no connectivity conditions.

[0009] However, on-device NLU and / or on-device fulfillment may fail for many requests. As a non-limiting example, consider an utterance provided to an assistant client device located in a user's living room that includes "Turn these lights a little brighter and make them warmer." Generating an appropriate semantic representation of the utterance requires resolving what is meant by "these lights," "a little brighter," and "warmer." Limited hardware resources on the assistant client device may preclude the use of local storage and / or the machine learning models, rules, and / or device topology necessary to resolve the meaning. However, a remote assistant system with more robust resources than the assistant client device may be able to resolve the meaning. For example, the remote assistant system may leverage a remotely stored device topology to determine "these lights" for an utterance received at the living room client device (e.g., loudest and / or clearest heard) and refer to lights A, B, and C in the living room. Additionally, the remote assistant system may utilize remote machine learning models and / or rules to determine that "a little brighter" means "30% (or other percent or value) brighter than the current brightness intensity levels of living room lights A, B, and C." Additionally, the remote assistant system may utilize remote machine learning models and / or rules to determine that "warmer" means "20% (or other percent or value) cooler color temperature than the current color temperature."

[0010] Therefore, due to hardware and / or other constraints of the assistant client device, the remote assistant system still needs to be utilized in resolving various user requests, such as a specific request to control a smart device. However, constantly providing unresolvable audio data and / or spoken text to the remote assistant system for complete processing may result in excessive resource usage and / or excessive latency in the remote assistant system when performing a corresponding action (e.g., controlling the smart device).

[0011] Accordingly, implementations described herein relate to particular techniques for controlling smart devices that are low latency and / or provide computational (client and / or server) and / or network efficiency. These implementations relate to generating and / or utilizing cache entries in a cache stored locally on an assistant client device in controlling various smart devices (e.g., smart lighting, smart thermostats, smart plugs, smart appliances, smart routers, etc.). Each cache entry includes a mapping of text to one or more corresponding semantic representations.

[0012] The semantic representation of the cache entry may be a semantic representation of text and / or corresponding audio data previously generated by one or more remote automated assistant servers (also referred to herein as "remote assistant systems") from the assistant client device to the remote assistant system in response to a previous transmission. The cache entry may be stored in the cache by the assistant client device in response to a cache request sent by the remote assistant system to the client device in response to a previous transmission. The text of the cache entry, mapped to the semantic representation of the cache entry, may be represented in the cache entry as the text itself and / or as a representation thereof, such as its Word2Vec embedding and / or a variant thereof that does not include stop words.

[0013] As described in detail herein, the semantic representation of the cache entry can include a cloud semantic representation and / or a locally interpretable semantic representation. The locally interpretable semantic representation is interpretable by the client device to generate a corresponding control command that can be transmitted by the client device to the local smart device and / or smart device controller via one or more local channels to cause corresponding local control of the smart device. The control command can be generated to conform to the protocol suite of the local smart device and / or smart device controller. Meanwhile, the cloud semantic representation is interpretable by the remote assistant system to generate a corresponding control request that is transmitted by the client device to the remote assistant system and transmitted to the corresponding smart device control system to cause corresponding control of the smart device.

[0014] A non-limiting example of generating and utilizing a cloud semantic representation is provided here. Assume that the utterance "Turn the lights a little" is captured in audio data detected via one or more microphones of an assistant client device. The assistant client device can process the audio data using a local speech-to-text model to generate the recognized text "Turn the lights a little." The assistant client device can check its local cache to determine whether there is a cache entry that matches the recognized text. The assistant client device can determine that a matching cache entry does not exist, which may be the first instance of the utterance on the assistant client device or the result of a previously matching cache entry being recently purged from the cache (e.g., to determine changes to device topology and / or other conditions). The assistant client device can optionally further determine that the recognized text is not locally interpretable utilizing an NLU component on the device. For example, the assistant client device may not have a robust local device topology that allows resolution of "light" to a particular light and / or may not have robust models and / or rules that allow resolution of "a little" to a particular degree.

[0015] The client device can further transmit the recognized text and / or audio data to the remote assistant system. Optionally, the transmission of the text and / or audio data to the remote assistant system is responsive to a determination that there is no matching cache entry and / or that the recognized text is not locally interpretable.

[0016] In response to receiving the recognized text and / or audio data (from which the remote assistant system can generate the recognized text), the remote assistant system can process the recognized text to generate a cloud semantic representation of the recognized text. For example, the remote assistant system can utilize the account identifier received in the text transmission to identify the account identifier's corresponding remotely stored device topology. Based on lights A and B being mapped in the device topology, the corresponding device topology can be utilized to resolve "light" to lights A and B as the default lights for the assistant client device (e.g., through a user-directed mapping or as a result of being assigned to the same room). Also, for example, the remote assistant system can utilize one or more remote models and / or rules to determine that "dim a little..." means "dim 20% (or other percentage or value) below the current brightness intensity level." A semantic representation can be generated based on these solutions, for example, [device=A and B, dimmer, relative intensity 20%], where "A and B" are unique identifiers of the corresponding lights, and "dimmer, relative intensity 20%" indicates that the lights are dimmed to an extent that they are reduced by 20% relative to their current intensity.

[0017] Once the cloud semantic representation is generated, the remote assistant system can generate a control request to activate the semantic representation. For example, the remote assistant system can identify the “current brightness intensity level” of “Lights A and B” based on their most recently reported status and / or by requesting (and receiving) the current status of those lights from a smart device remote system that controls “Lights A and B” (e.g., a smart device remote system controlled by the manufacturer of “Lights A and B”). Furthermore, the remote assistant system can generate a “target brightness intensity level” by multiplying the “current brightness intensity level” by 0.80 (based on determining that “a little dimmer” means “20% dimmer than the current brightness intensity level”). A control request can then be generated that identifies “Lights A and B” and includes a request to set the lights to the generated “target brightness intensity level.” The control request can then be sent to the smart device remote system. In response to receiving the control request, the smart device remote system sends corresponding control commands to "Lights A and B" over the Internet through a hub coexisting with "Lights A and B" (e.g., in the case of BLE, Z-Wave, ZigBee, etc.), or directly to the smart devices via an IP connection (e.g., in the case of Wi-Fi and other smart devices that do not require a hub). The control command may match the control request or may be different from the control request, but is generated based on the control request. Optionally, the remote assistant system also sends text, audio data, and / or graphics to be rendered to the client device to confirm the change in the state of "Lights A and B." For example, the text "OK, dim lights A and B by 20%" can be sent to the client device, causing the client device to render the text graphically and / or render corresponding synthesized speech using a local text-to-speech model.

[0018] Additionally, the remote assistant system can send a cache request to the client device that includes the cloud semantic representation and, optionally, text (or a representation thereof), and a mapping between the text (or representation) and the cloud semantic representation. In response to receiving the cache request, the client device can generate a cache entry in its local cache that defines the mapping of the text to the cloud semantic representation. For example, the cache entry can include a mapping of the text itself to the cloud semantic representation and / or a mapping of a representation of the text (e.g., a Word2Vec representation or other representation) to the cloud semantic representation. In situations where the cache request itself does not include the text (or a representation thereof), the client device can include the mapping based on a cache request received in response to sending the text (or corresponding audio data). In some implementations, the remote assistant system can also include in the cache request text, audio data, and / or a graphic confirming the change in the state of “Lights A and B.”

[0019] Further, assume that after generation of the cache entry at the client device, another instance of the utterance "Turn the lights down a little" is captured in audio data detected via one or more microphones of the assistant client device. The assistant client device can process the audio data using a local speech-to-text model to generate the recognized text "Turn the lights down a little." The assistant client device can check its local cache and determine that the generated cache entry contains text (or a representation thereof) that matches the recognized text. As used herein, "match" can include an exact match and / or a soft match. For example, using a soft match, it can be determined that the text "Turn the lights down a little" matches "Turn the lights down a little" based on edit distance analysis, the presence or absence of stop words only, and / or the use of other techniques. In response to determining a match, the assistant client device can send to the remote assistant system a cloud semantic representation that is mapped to the text in the cache entry. Optionally, the assistant client device can send the cloud semantic representation without sending the audio data and / or the recognized text to the remote assistant system.

[0020] In response to receiving the cloud semantic representation, the remote assistant system can then generate a control request to activate the cloud semantic representation. In the "dim the lights a little" example, note that the control request may vary depending on the "current brightness levels" of "lights A and B" at the time of the user input that matches "dim the lights a little." For example, if the control request specifies "target brightness levels" for "lights A and B," those target brightness levels depend on the "current brightness levels." The control request can then be sent to the smart device remote system to trigger the corresponding control of "lights A and B." Thus, by sending the semantic representation to the remote assistant system, the remote assistant system can bypass text processing and instead utilize the received cloud semantic representation to regenerate the semantic representation.

[0021] These and other schemes reduce power, memory, and / or processor resource usage in the remote assistant system because less processing occurs in the remote assistant system. This reduction is particularly significant given the large number (e.g., thousands or hundreds of thousands) of smart device control requests that can be received and processed by the remote assistant system daily. Furthermore, these and other schemes allow control requests to be generated and sent more quickly, thereby allowing smart device states to be changed more quickly. For example, bypassing the generation of semantic representations from text can save at least 200 milliseconds, 500 milliseconds, 700 milliseconds, or other periods when generating control requests, which can be particularly important in controlling various smart devices (e.g., turning on lights in a dark room).

[0022] In an implementation in which the cache entry also includes text, audio data, and / or graphics confirming the change in the state of "Lights A and B," the assistant client device may also render such content in response to determining that the cache entry matches the input. The content may be rendered immediately or, optionally, after a delay that can be optionally specified in the cache entry (e.g., a delay based on the latency between receiving the input and enabling the corresponding state change).

[0023] Note that in the foregoing example, the smart device control request is a state-related state change request. That is, it requests a change in the state of the smart device in a manner that depends on the current state of the smart device. More specifically, it requests a change to the brightness state of lights A and B in a manner that is related to the current brightness state of lights A and B. In some implementations, the remote assistant system may determine to provide a cache request including the semantic representation in lieu of a control request in response to determining that the semantic representation is a relative state change representation. Such a determination may be made based on the semantic representation referencing the current state of the smart device.

[0024] In some of these implementations, the remote assistant system can provide a cache request including a control request in place of the semantic representation in response to determining that the semantic representation is not a relative state change representation. For example, assume instead an utterance of "set the lights to 500 lumens." In such an example, a cloud semantic representation can be generated that identifies "lights A and B" and identifies a "target brightness intensity level" of "500 lumens" for both lights. A corresponding control request can also be generated. Because the cloud semantic representation is not relative, the remote assistant system can optionally send a cache request including the control request. In response, the client device can generate a cache entry that maps the text "set the lights to 500 lumens" (or a representation thereof) to the control request. Thereafter, in response to determining that the received input matches the text of the cache entry, the client device can send the control request to the remote assistant system. The remote assistant system then sends the control request to the smart device remote system, thereby enabling the remote assistant system to directly utilize the control request and thereby bypass both generating the semantic representation and the control request. This in turn may have a positive impact on resource utilization and / or latency of the remote assistant system.

[0025] Also note that among assistant client devices in a user / account ecosystem, a particular cache entry can include a mapping of the same text to different semantic representations (or control requests). For example, assume that "Turn the lights down a bit" is received at different assistant client devices, and that "lights C and D" are mapped in the device topology as preferred lights for the different assistant client devices. In such an example, a semantic representation identifying "lights C and D" is generated instead of "lights A and B." Thus, cache entries can be generated for the different client devices that include a mapping of "Turn the lights down a bit" to a semantic representation identifying "lights C and D."

[0026] It should also be noted that if an utterance is detected at multiple client devices, one or more device arbitration techniques may be utilized to determine that it was received at only one of those client devices. Such techniques may consider various features in determining which client device should be the receiving device, based on determining that the client device is closest to the user and / or is otherwise intended to be interacted with by the user (despite not being closest). Such features may include the volume of the audio data spoken at the client device, the time of detection of the audio data at the client device, the recency of the interaction with the client device, the proximity of the user to the client device, and / or other features.

[0027] A non-limiting example of generating and utilizing a locally interpretable semantic representation is provided below. Assume again that the utterance "Turn the lights down a bit" is captured in audio data detected via one or more microphones of the assistant client device. The assistant client device can process the audio data using a local speech-to-text model to generate the recognized text "Turn the lights down a bit." The assistant client device can check its local cache to determine whether there is a cache entry that matches the recognized text. The assistant client device can determine that no matching cache entry exists, which may be the result of this being the first instance of the utterance on the assistant client device or that a previously matching cache entry has recently been purged from the cache. The assistant client device can optionally further determine that the recognized text is not locally interpretable utilizing an NLU component on the device.

[0028] The client device can further transmit the recognized text and / or audio data to the remote assistant system. Optionally, the transmission of the text and / or audio data to the remote assistant system is responsive to a determination that there is no matching cache entry and / or that the recognized text is not locally interpretable.

[0029] In response to receiving the recognized text and / or audio data (from which the remote assistant system can generate the recognized text), the remote assistant system can process the recognized text to generate a locally interpretable semantic representation of the recognized text. For example, the remote assistant system can use the account identifier received in the text transmission to identify the account identifier's corresponding remotely stored device topology. Based on the light C mapped in the device topology, the corresponding device topology can be used to resolve "light" to "light C" as the default light for the assistant client device.

[0030] Also, for example, the remote assistant system may utilize one or more remote models and / or rules to determine that “dim… a little” means “dim 20% (or other percentage or value) below the current brightness intensity level.” The remote assistant system may generate a locally interpretable semantic representation by including an identifier for “Light C” and a state change representation that is a relative state change representation that indicates how to locally resolve the target intensity level for “dim” a little. For example, the identifier for “Light C” may be a local address for “Light C.” Also, for example, the state change representation may be “set intensity = (current intensity * 0.8),” indicating that the intensity of “Light C” should be set to a value determined by identifying 80% of the current intensity value of “Light C” (e.g., the most recently reported intensity or a locally requested and locally received intensity value). Thus, in some implementations, the locally interpretable semantic representation may define the state change representation with more granularity than its cloud-based counterpart to enable local resolution of relative values ​​that can be utilized to generate local control commands (e.g., utilizing a local adapter described herein). The state change representation may optionally further include an indication of the adapter (e.g., an adapter specific to the manufacturer of the smart device) to be utilized in generating the corresponding local control command, an indication of the channel (e.g., Bluetooth or Wi-Fi) to be utilized to transmit the local control command, and / or an indication of the protocol suite to be utilized in generating a control command based on the locally interpretable semantic representation. A locally interpretable semantic representation may be, for example, [device=C; set strength=(current strength*0.8)].

[0031] In some implementations, the system may determine to generate a locally interpretable semantic representation in response to determining that the client device (or an additional client device locally connected to the client device) can locally control a smart device referenced in the recognized text, determining that the smart device referenced in the recognized text can be locally controlled, and / or determining that a state change requested in the recognized text can be enabled for local control (e.g., if some states can be changed through local control, but other states can only be changed through cloud control).

[0032] Once the locally interpretable semantic representation is generated, the remote assistant system can optionally transmit the locally interpretable semantic representation to the client device for immediate implementation. In other words, the remote assistant system can transmit the locally interpretable semantic representation to cause the client device to interpret the locally interpretable semantic representation in response to an utterance, generate a corresponding local control command, and locally transmit the local control command to effectuate a state change to “Light C.” For immediate implementation, transmitting the locally interpretable semantic representation to the client device can be the transmission of the cache request itself (described below) or a separate transmission. Alternatively, the remote assistant system can instead generate and transmit a corresponding cloud control request to the smart device control system of “Light C” to cause the smart device control system to generate and transmit a corresponding control command to “Light C” to effectuate the state change.

[0033] Regardless of the manner in which the state change is implemented in response to an immediate request, the remote assistant system can send a cache request to the client device that includes the locally interpretable semantic representation and, optionally, the text (or a representation thereof) and a mapping between the text (or representation) and the locally interpretable semantic representation. In response to receiving the cache request, the client device can generate a cache entry in the local cache that defines the mapping of the text to the locally interpretable semantic representation. For example, the cache entry can include a mapping of the text itself to the locally interpretable semantic representation and / or a mapping of a representation of the text to the locally interpretable semantic representation. In situations in which the cache request itself does not include the text (or a representation thereof), the client device can include the mapping based on a cache request received in response to transmitting the text (or corresponding audio data). In some implementations, the remote assistant system can also include in the cache request text, audio data, and / or a graphic confirming the change in the state of “Light C.”

[0034] Further, assume that after generation of the cache entry at the client device, another instance of the utterance "Turn the lights down a bit" is captured in audio data detected via one or more microphones of the assistant client device. The assistant client device can process the audio data using a local speech-to-text model to generate recognized text of "Turn the lights down a bit." The assistant client device can check its local cache and determine that the generated cache entry contains text (or a representation thereof) that matches the recognized text. Exact matches and / or soft matches can be utilized. In response, the assistant client device can process the locally interpretable semantic representation of the cache entry to generate a corresponding local control request to send locally to "Light C." For example, based on the locally interpretable semantic representation including "set intensity = (current intensity * 0.8)," the assistant client device can locally identify the "current intensity" of "Light C" and then determine the value of "set intensity" by multiplying the current intensity by 0.8. Thus, the "setting strength" to the determined value may be a particular state change that is generated based on the relative state change of the locally interpretable semantic representation.

[0035] Additionally, the assistant client device may generate control commands based on the "set intensity" value and, optionally, utilizing one or more local adapters. The control commands may be generated to conform to a protocol suite conforming to "Lighting C," such as BLE, Z-Wave, Zigbee, Wi-Fi, or other protocol suite. The protocol suite utilized in generating the control commands may be selected based on an identifier of the smart device included in the locally interpretable semantic representation and / or based on an adapter indication that may be included in the locally interpretable semantic representation.

[0036] Furthermore, the generated control command can be locally transmitted to “Light C” (or a corresponding local device that controls “Light C”). The control command can be transmitted to “Light C” (or a corresponding local device) based on a locally interpretable semantic representation that identifies “Light C” (e.g., identifies the address of “Light C”) and can be transmitted via a local channel optionally identified via the locally interpretable semantic representation. The local channel can be, for example, a Wi-Fi wireless channel, a BLUETOOTH® wireless channel, or another channel. Thus, a state change can be enabled in “Light C” without the need to retransmit any data to the remote assistant system and / or without the remote assistant system having to regenerate the semantic representation. This can conserve network resources by eliminating the need to transmit audio data and / or recognized text to the remote assistant system, and can conserve processing resources in the remote assistant system because the semantic representation does not have to be regenerated in the remote system. Furthermore, by bypassing transmission to and processing in the remote assistant system, latency in the implementation of a requested state change in the smart device can be significantly reduced.

[0037] As described above, various implementations utilize an adapter to locally interpret a locally interpretable semantic representation, or specific state changes generated thereon, into corresponding control commands. In these implementations, the client device can include one or more adapters, each of which can be tailored to a specific smart device, a specific smart device manufacturer, and / or a specific protocol suite. When executed by the assistant client device, the adapter can process the locally interpretable semantic representation (and / or specific state changes generated based on the locally interpretable semantic representation) to generate tailored specific control commands that, when sent locally to at least one corresponding smart device (or a hub / bridge or other component connected to the smart device), can be directly interpreted by the corresponding smart device (or other component connected to the smart device) to activate the state change in the corresponding smart device. For example, the adapter can be implemented in JavaScript (or other interpreted programming language) and can convert generic smart device control commands generated based on the locally interpretable semantic representation into specific control commands that conform to the protocol suite of the smart device. The adapter can optionally run in a container within the automated assistant client. In implementations in which multiple adapters are available on the assistant client device, a particular adapter can be selected for a locally interpretable semantic representation based on an identifier of the smart device included in the semantic representation and / or based on an indication of the adapter that can be included in the semantic representation.

[0038] In an implementation in which the cache entry also includes text, audio data, and / or graphics confirming the change in the state of "Lights A and B," the assistant client device can also render such content in response to determining that the cache entry matches the input. The content can be rendered immediately or, optionally, after a delay.

[0039] Note that in the foregoing example, the smart device control request is a state-related state change request. That is, the smart device control request requests a change in the state of the smart device in a manner that depends on the current state of the smart device. More specifically, the smart device control request requests a change to the brightness state of light C in a manner that is related to the current brightness state of light C. In some implementations, the remote assistant system may determine to provide a cached request including a locally interpretable semantic representation in place of a local control command in response to determining that the semantic representation is a relative state change representation. Such a determination may be made based on the semantic representation that references the current state of the smart device.

[0040] In some of these implementations, the remote assistant system can provide a cache request including a local control command in place of the semantic representation in response to determining that the semantic representation is not a relative state change representation. For example, assume instead that the utterance is, "Set the light to 500 lumens." In such an example, a locally interpretable semantic representation can be generated that identifies "Light C" and identifies a "target brightness intensity level" of "500 lumens" for that light. A corresponding control request can also be generated. Because the semantic representation is not relative, the remote assistant system can optionally send a cache request including a local control command generated based on the locally interpretable semantic representation. In response, the client device can generate a cache entry that maps the text "Set the light to 500 lumens" (or a representation thereof) to the control command. Thereafter, in response to determining that the received input matches the text in the cache entry, the client device can locally transmit the control command in the cache entry, thereby bypassing the generation of the control command based on the locally interpretable semantic representation.

[0041] It should also be noted that, similar to cloud semantic representations, within an assistant client device in a user / account ecosystem, a particular cache entry may contain mappings of the same text to different locally interpretable semantic representations (or local control commands).

[0042] Other examples of cloud semantic representations and locally interpretable semantic representations are provided above. However, in some implementations, a single cache entry can include a mapping of text to both a locally interpretable semantic representation and a cloud semantic representation. For example, lights "A" and "B" may both be identified for a request, and light "A" may be locally controllable, while light "B" may be controllable only via cloud control. As a result, the remote assistant system can generate and provide a cache request that maps the text of the request to both (a) a locally interpretable semantic representation that can be processed locally to effect a state change in "light A" and (b) a cloud semantic representation that is sent to the remote assistant system to effect a state change in "light B."

[0043] In various implementations, the remote assistant system and / or client device can determine a change in the account's device topology and / or the occurrence of other conditions, resulting in the purging (e.g., permanently deleting or flagging as stale) of one or more cache entries in the cache of the ecosystem's client devices. A change in device topology can include, for example, the addition or removal of a smart device from the device topology, the renaming of a smart device and / or assistant client device from the device topology, the assignment of a device to a new room and / or group in the device topology, and / or other changes to the device topology. Such changes can affect some or all semantic representations in the cache, and as a result, in response to detecting the change, at least the affected cache entries (and optionally all cache entries) can be purged from the cache. As an example, assume that additional lights are added to the device topology and assigned to a room with a “kitchen” label. Upon that addition, any semantic cache entries associated with those lights previously assigned to “kitchen lights” become stale because they do not reflect the newly added additional lights. Thus, in response to detecting the change, at least those cache entries associated with “kitchen lights” can be purged from the cache. For example, the remote assistant system can determine such a change and send a cache clearing request to all assistant devices in the ecosystem to clear the local caches of all assistant client devices as a result. Thereafter, when spoken, typed, or other input is received, a new cache entry is generated at the assistant client device. For example, if the cache entry for the text "Dim the kitchen lights" is cleared, a new cache entry is created when the user subsequently provides the utterance "Dim the kitchen lights."

[0044] In some implementations, the assistant client device can at least selectively process audio data using an on-device speech-to-text model without detecting an explicit automated assistant invocation at the assistant client device. An explicit invocation cue is a cue that, when detected alone, always activates at least on-device speech recognition. Some non-limiting examples of explicit invocation cues include detecting a spoken hotword with at least a threshold confidence, an explicit assistant interface element activation (e.g., a hardware button or a graphic button on a touchscreen display), a "phone squeeze" (e.g., detected by a sensor in a mobile phone bezel) with at least a threshold intensity, and / or other explicit invocation cues.

[0045] As an example, the assistant client device can process audio data without detecting an explicit automated assistant invocation when a human presence is detected near the device (e.g., based on a passive infrared sensor and / or other sensor), in response to detecting any voice activity (e.g., using a voice activity detector), in response to detecting at least a threshold level of noise near the device during a particular day and / or time, and / or in response to other conditions. In an implementation in which the assistant client device processes audio data, the assistant client device can utilize an on-device speech-to-text model to determine whether any recognized text matches any "hot phrases" so indicated locally on the assistant client device. Each "hot phrase" comprises a sequence of one or more words. If it is determined that the recognized text from on-device speech recognition of the audio data does not match a hot phrase, the assistant client device can discard such recognized text and audio data. If it is determined that the recognized text from on-device speech recognition of the audio data matches a hot phrase, the assistant client device can automatically initiate one or more corresponding actions based on the text. The user can control which phrases are shown as hot phrases, and optionally, explicit user approval may be required before a phrase is shown as a hot phrase.

[0046] In some implementations, a hot phrase can be a phrase that matches the text of a corresponding cache entry described herein. In those implementations, an action automatically initiated by the assistant client device based on a hot phrase can include transmitting a corresponding cloud semantic representation or locally interpreting a corresponding locally interpretable semantic representation. In some of those implementations, the text of a cache entry can be designated as a hot phrase for the assistant client device based on a determination that one or more criteria are met at the assistant client device. The criteria can include, for example, that the text and / or matching text has been determined to be present at least a threshold number of times (overall or within a recent temporal period) and / or at least a threshold frequency (overall or within a recent temporal period) in user input (e.g., typed and / or spoken) received at the assistant client device. In some versions of those implementations, the text of the cache entry can be automatically designated as a hot phrase, optionally with confirmation of such designation provided via a user interface input (e.g., "By the way, I designated X as a hot phrase. This can be easily changed in settings"). In some other versions, text may be designated as a hot phrase only after a confirmatory user input in response to a prompt. For example, a confirmatory "yes" in response to the prompt, "If I request that you "turn on the lights frequently," would you like to designate that as a hot phrase so that I don't have to say "OK Assistant" first?" In still other implementations, the text of any local cache entry in the assistant client device's cache can be automatically designated as a hot phrase for the assistant client device.Implementations that designate text in a cache entry as a hot phrase can shorten the duration of user interaction with the assistant client device by eliminating the need for the user to first provide an explicit invocation, such as an invocation hotword (e.g., "OK Assistant"). Reducing the duration of interaction with the assistant client device improves human-assistant client device interaction, allowing more smart device state changes to take effect in a shorter period of time.

[0047] The above description is provided as a summary of only some implementations of the present disclosure. Further descriptions of those and other implementations are set forth in more detail herein.

[0048] Additionally, some implementations include one or more processors of one or more computing devices, the one or more processors operable to execute instructions stored in associated memory, the instructions configured to cause performance of any of the methods described herein. Some implementations include a client device having one or more processors that execute locally stored instructions and interface with locally stored data to perform one or more of the methods described herein. Some implementations also include one or more computer-readable storage media (transitory or non-transitory) that store computer instructions executable by the one or more processors to perform any of the methods described herein.

[0049] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as part of the subject matter disclosed herein, for example, all combinations of subject matter recited in the claims appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. [Brief explanation of the drawings]

[0050] [Figure 1] FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented. [Figure 2] FIG. 1 illustrates an example environment including multiple smart devices from multiple different parties and including multiple assistant client devices. [Figure 3A] FIG. 3 illustrates an example of how an utterance received at the assistant client device of FIG. 2 can be handled if the assistant client device does not have a cache entry that matches the text of the utterance. [Figure 3B] FIG. 3B illustrates an example of how the same utterance of FIG. 3A, received at the same client device of FIG. 3A, can be processed differently after a cache entry with a cloud semantic representation is stored in the assistant client device of FIG. 3A. [Figure 4A] FIG. 3B illustrates an example of how the same utterance of FIGS. 3A and 3B received at an alternative assistant client device of FIG. 2 can be processed if the alternative assistant client device does not have a cache entry that matches the text of the utterance. [Figure 4B] FIG. 4B illustrates an example of how the same utterance of FIG. 4A received at an alternative client device of FIG. 4A may be processed differently after a cache entry having a cloud semantic representation is stored in the alternative client device of FIG. 4A. [Figure 5A] FIG. 3B illustrates an example of how another utterance received at the same assistant client device of FIG. 3A and FIG. 3B can be handled if the assistant client device does not have a cache entry that matches the text of the utterance. [Figure 5B]FIG. 5B illustrates an example of how the same utterance of FIG. 5A received at the client device of FIG. 5A may be processed after a cache entry having a locally interpretable semantic representation is stored in the client device of FIG. 5A. [Figure 6] 1 is a flowchart illustrating an example method that may be implemented by an assistant client device according to various implementations disclosed herein. [Figure 7] 10 is a flowchart illustrating another example method that may be implemented by an assistant client device according to various implementations disclosed herein. [Figure 8] 1 is a flowchart illustrating an example method that may be implemented by a remote assistant system according to various implementations disclosed herein. [Figure 9] FIG. 1 illustrates an exemplary architecture of a computing device. DETAILED DESCRIPTION OF THE INVENTION

[0051] There has been a proliferation of smart network-connected devices (also referred to herein as smart devices or Internet of Things (IoT) devices), such as smart home alarms, smart door locks, smart cameras, smart lighting, smart thermostats, smart scales, smart beds, smart irrigation systems, smart garage door openers, smart plugs, smart appliances, smart baby monitors, smart televisions (TVs), smart fire alarms, smart humidity detectors, smart routers, etc. Often, multiple smart devices are located within a structure such as a home, or within multiple related structures, such as a user's primary residence and the user's secondary residences and / or workplaces.

[0052] Additionally, there is a proliferation of assistant client devices, each of which can include an assistant client that can optionally interact with one or more remote automated assistant components to form a logical instance of an automated assistant. Assistant client devices can be dedicated solely to assistant functionality (e.g., a standalone speaker and / or a standalone audio / visual device that includes only an interface associated with the assistant client and is dedicated solely to assistant functionality), or can perform assistant functionality in addition to other functions (e.g., a mobile phone or tablet that includes an assistant client as one of multiple applications). Furthermore, some smart devices may be assistant client devices. For example, some smart devices can include an assistant client and at least a speaker and / or microphone that (at least in part) function as user interface output and / or input devices for the assistant interface of the assistant client.

[0053] Various techniques have been proposed for associating smart devices with corresponding logical instances of an automated assistant (and optionally with individual assistant client devices). For example, a user, a group of users, an assistant client device, and / or a group of assistant client devices (e.g., all within a structure) can be linked (e.g., in one or more databases) with multiple different smart devices to enable interaction with (e.g., control of) the smart devices via the automated assistant. For example, each of multiple assistant client devices in a home can be linked to each of multiple different smart devices in the home to enable any user (or a restricted group of users) to interface with any of the assistant client devices to interact with any of the multiple different smart devices.

[0054] Such linking of smart devices and assistant client devices is referred to herein as a device topology and can be represented by various data structures. The device topology can be created by a user and / or automatically and can define various assistant client devices, various smart devices, their respective identifiers, and / or their respective attributes. For example, a device identifier can specify the room (and / or other area) in the structure (e.g., living room, kitchen) in which the device is located and / or specify a nickname and / or alias for the device (e.g., sofa lamp, front door lock, bedroom speaker, kitchen assistant, etc.). In this manner, a device identifier can be the name, alias, and / or location of each device that a user is likely to associate with each device.

[0055] The device topology representation can further specify one or more device attributes associated with each device. The device attributes of an assistant client device can include, for example, one or more input and / or output modalities supported by the assistant client device and / or a preferred smart device to be controlled by the assistant client device (e.g., ambiguous smart TV commands received at assistant client device 1 should be assumed to be directed to smart TV 1, while assistant client device 2 can assume they are directed to smart TV 2). For example, the device attributes of a standalone speaker-only assistant client device can indicate that it can provide audible output but not visual output. Also, for example, the device attributes of the same standalone speaker-only assistant client device can indicate that lighting control requests that do not specify specific lights should be interpreted as relating to lights assigned to the same room as the assistant client device. The device attributes of the smart device can, for example, identify one or more states of the controllable smart device, identify a party (e.g., 3P) that manufactures, distributes, and / or creates firmware for the smart device, and / or identify a unique identifier for the smart device, such as the smart device's address and / or a 1P- or 3P-provided fixed identifier. According to various implementations disclosed herein, the device topology representation can optionally further specify which smart devices can be locally controlled by which assistant client devices, the local addresses of the locally controllable smart devices (or the local addresses of hubs that can directly locally control those smart devices), local signal strength and / or other priority indicators between the assistant client devices and the smart devices.Additionally, according to various implementations disclosed herein, the device topology representation (or variations thereof) can be stored remotely at the remote assistant system and / or locally at each of multiple assistant client devices for use in locally controlling the smart devices. Optionally, each locally stored device topology may be less robust (e.g., define fewer devices, attributes, and identifiers) than the remotely stored device topology, for example, due to hardware constraints of the assistant client device.

[0056] 1, an example environment in which the techniques disclosed herein may be implemented is shown. The example environment includes multiple assistant client devices 110. 1-N (also referred to herein simply as a “client device”), a cloud automated assistant component 120, and a smart device system 140 A-N , Smart Devices 145 A-N , and the client device 110 1-N The device topology 152 for the client device 110 of FIG. 1-N and smart devices 145 1-N represents client devices and smart devices that are at least selectively associated with each other (e.g., via device topology 152). For example, smart device 145 1-N may all be in the home (e.g., inside and / or outside the home) and may be connected to the client device 110 1-N may be in the same house at least occasionally, and smart devices 145 1-N and client device 110 1-N may be linked to one another using one or more techniques as described herein. Through such links, client devices 110 1-N may be connected to the smart device 145 according to the implementation described herein. 1-N can be used to control

[0057] Client Device 110 1-N One or more (e.g., all) of the client devices 110 may run a respective instance of the automated assistant client. 1-N One or more of the smart devices 110 may optionally include an engine and hardware components for controlling one or more smart devices even without an instance of the automated assistant client. 1-N It may be an application separate from the operating system of the corresponding client device 110 (e.g., installed "on top" of the operating system), or 1-N Each instance of the automated assistant client can be implemented directly by the operating system of the client device 110. 1-N The client device 110 may optionally interact with the cloud automated assistant component 120 in response to various requests provided by a user via any one of the I / O components 111. 1-N Other engines can optionally interact with the cloud automated assistant component 120.

[0058] Client Device 110 1-N One or more (e.g., all) of the STT models 152 may be used to generate corresponding recognized text for an utterance. 1-N and a respective speech-to-text (STT) engine 112 that processes the audio data capturing the speech. 1-N Each instance of audio data can be transmitted to the I / O component 111 of the respective client device. 1-N In some implementations, the STT engine 112 1-Ncorresponds to the client device 110 1-N In some implementations, the STT engine 112 processes only audio data following an explicit automated assistant invocation detected in 1-N may at least selectively process audio data even in the absence of an explicit automated assistant invocation, but discard (without further processing) any audio data and / or recognized text that is determined not to contain any "hot phrases" as described herein. In situations where a "hot phrase" is detected, one or more responsive actions may be taken based on the "hot phrase." For example, the "hot phrase" may be stored in a local cache 158 1-N If the semantic representation of the cache entry matches the text of the cache entry, the semantic representation of the cache entry may be further processed as described herein.

[0059] Client Device 110 1-N One or more (e.g., all) of the may also use a respective locally stored NLU model 153 to process the text to attempt to locally generate a local semantic representation of the text. 1-N Each natural language understanding (NLU) engine uses 113 1-N The text may be recognized text from the STT engine 112, or may be typed text (e.g., input via a virtual keyboard on a touchscreen), or may be text associated with a selected virtual and / or hardware button (e.g., text that maps to the selection of a particular virtual button displayed on a touchscreen). 1-N may be able to generate valid semantic representations of various text phrases. However, as described herein, the NLU engine 113 1-Nmay also be unable to generate valid semantic representations of various other text phrases, such as various text phrases related to control of a smart device (e.g., relative control of a smart device).

[0060] Client Device 110 1-N One or more (e.g., all) of the fulfillment engines 114 may also 1-N Each fulfillment engine 114 may include 1-N Each cloud module 115 1-N , each local module 116 1-N , and / or their respective text-to-speech (TTS) engines 117 1-N may include:

[0061] Cloud Module 115 1-N can fulfill various requests through interactions with the cloud automated assistant component 120. For example, the cloud module 1151 may process the text of the received input (e.g., the recognized text STT engine 112 1-N ) matches the text of a cache entry in local cache 1581, as described herein, can send a cloud semantic representation mapped to the text in the cache entry to cloud automated assistant component 120. Cloud module 1151 can also optionally send a cloud semantic representation generated by NLU engine 1131 to cloud automated assistant component 120. Cloud fulfillment engine 127 of cloud automated assistant component 120 can then utilize the received cloud semantic representation in fulfilling the received input. For example, in the case of a smart device control input, cloud fulfillment engine 127 can generate a corresponding control request to send to corresponding smart device system 140 A-NThe cloud semantic representation can be utilized to generate and send corresponding control commands to the corresponding smart device 145. A-N can be sent to.

[0062] Cloud Module 115 1-N Also, each NLU engine 113 1-N and / or their respective local caches 158 1-N For received input that does not match any cache entry in the cloud-based automated assistant component 120, data representing the received input can be sent to the cloud-based automated assistant component 120. For example, if the received input is a speech captured in audio data, the audio data and / or its locally recognized text (e.g., stored in the respective STT engine 112) can be used to generate a local recognition text. 1-N (from) to the cloud-based automated assistant component 120. For at least some of those transmissions, the cloud-based component 120 may, in response, forward cache requests to the respective cache engines 118 to enable future occurrences of the input to be satisfied more efficiently and / or with reduced latency. 1-N Further, cloud-based component 120 may additionally or alternatively respond to the received inputs by performing corresponding cloud fulfillments and / or providing locally interpretable semantic representations and / or local commands to respective client devices 110 in performing local fulfillments. 1-N For use by each client 110 1-N can be provided to.

[0063] Local Module 116 1-N can fulfill various requests locally and without interaction with the cloud automated assistant component 120. For example, the local module 1161 may process the text of the received input (e.g., the STT engine 112 1-NIn response to determining that the recognized text from the local cache 1581 matches the text of a cache entry in the local cache 1581, the local module 1161 may utilize the locally interpretable semantic representation mapped to the text in the cache entry in fulfilling the request locally. For example, the local module 1161 may communicate with one or more corresponding smart devices 145 via each of the radios 1191. A-N Optionally, as described herein, if the locally interpretable semantic representation is for smart device control, the local module 116 may utilize the locally interpretable semantic representation in generating the local control command for local transmission to the smart device. 1-N may be configured to generate local control commands for one or more respective adapters 156. 1-N The local module 1161 can also optionally locally interpret any locally interpretable semantic representations generated by the NLU engine 1131.

[0064] Text-to-Speech (TTS) Engine 117 1-N can optionally be utilized to generate synthesized speech for rendering on demand. For example, the TTS engine 1171 can process the text of a received input (e.g., the TTS engine 112 1-NIn response to determining that the recognized text from the local cache 1581 matches the text of a cache entry in the local cache 1581, the response text mapped to the text in the cache entry can be utilized to generate corresponding synthetic speech, such that the synthetic speech is rendered in response to the received input. The synthetic speech can be rendered in addition to other actions (interpretable in the cloud and / or locally) performed based on the semantic representation, which is also mapped to the text in the cache entry. Furthermore, the response text can additionally or alternatively be visually rendered, and / or alternative content can additionally or alternatively be rendered by the fulfillment engine in response to being mapped to the text in the cache entry.

[0065] Client Device 110 1-N One or more (e.g., all) of the cache engines 118 may also optionally be connected to the respective cache engines 118 described above. 1-N and their respective local caches 158 1-N Cache engine 118 1-N receive respective cache requests from the cache generation engine 125 of the cloud automated assistant component 120 and store respective cache entries in respective local caches 158 1-N The local cache 158 1-N For example, each client device 110 1-N As described herein, in some implementations, received cache requests from the cache generation engine 125 may be stored in the RAM and / or ROM of the respective local cache 158. 1-N In other implementations, the cache request may not have text to include in the cache entry, but may map to a semantic expression within the cache entry. In such implementations, each cache engine 118 1-NThe text can be determined based on the text most recently provided to the cloud automated assistant component 120 (or the recognized text of the audio data most recently provided to the cloud automated assistant component 120).

[0066] The cloud automated assistant component 120 communicates with the client device 110 via one or more wide area networks (e.g., the Internet), generally indicated by 1051 in FIG. 1-N The cloud automated assistant component 120 may be implemented on one or more computing systems (collectively referred to as a "cloud" or "remote" assistant system) communicatively coupled to the client device 110. For example, the cloud automated assistant component 120 may be implemented by one or more clusters of high performance servers. 1-N It should be noted that the client devices 110 may utilize one or more local area networks in accessing the wide area network 1051 and / or in communicating locally with each other. Such local area networks may be utilized by the client devices 110. 1-N This may include a Wi-Fi network and / or a mesh network between the devices.

[0067] The cloud automated assistant component 120 also communicates with the smart device system 140 via one or more wide area networks. A-N The communication coupling between the cloud automated assistant component 120 and the smart device system 140 is generally indicated by 1052 in FIG. 1. Additionally, each smart device system 140 can communicate with one or more smart devices 145 via one or more wide area networks generally indicated by 1104 in FIG. 1. A-N The smart device 145 may be communicatively coupled to a corresponding group of A-NIt should be noted that the may utilize one or more local area networks to access the wide area network 1053 .

[0068] Smart Device System 140 A-N Each of the smart devices 145 may be either a first party (1P—i.e., manufactured and / or distributed by the same party that controls the automated assistant) or a third party (3P—i.e., manufactured and / or distributed by a different party) system, and each may be connected to one or more corresponding smart devices 145 A-N For example, the first smart device system 140 A-N is controlled by the first 3P and the first smart device 145 A1 The second smart device system 140 can be communicatively coupled to the second smart device 145, and the second smart device system 140 can be controlled by the second 3P and can communicate with the second smart device 145. B1 and third smart devices 145 B2 and the like.

[0069] Smart Device System 140 A-N 145 of those respective smart devices A-N Controlling firmware updates for each of those smart devices 145 A-N and deliver it to their respective smart devices. A-N The device 145 may be connected to the wide area network 1053 to check the status of the device 145 or the like. A-N For example, a given one of the smart device systems 140 may communicate with a smart device 145 to control the smart device in response to user input received via a mobile application on the smart device system that enables control of the smart device. A-N It can communicate with one of

[0070] Also, for example, a given one of the smart device systems 140 may be configured to control the smart device 145 in response to requests from the cloud automated assistant component 120. A-N For example, according to some techniques, a user can provide a request to control a smart device via one or more I / O components 1111 of client device 1101, such as a voice input of "turn off the couch lights" provided via a microphone of I / O component 1111. The request (e.g., audio data capturing the voice input or locally generated text) can be transmitted by client device 1101 to cloud automated assistant component 120 via wide area network 1051. Cloud automated assistant component 120 processes the request to determine the smart device to be controlled based on the request and transmits the control request to smart device system 140 via wide area network 1052. A-N , which then transmits a corresponding command to the smart device over the wide area network 1053. However, as described herein, such techniques present drawbacks such as high latency and / or excessive consumption of network resources.

[0071] In some implementations, multiple client computing devices 110 1-N and smart devices 145 A-N may be associated with one another in various ways to facilitate performance of the techniques described herein. For example, in some implementations, multiple client devices 110 1-N and smart devices 145 A-N may be associated with one another because they are communicatively coupled via one or more LANs and / or via one or more peer-to-peer networks. 1-NThis may be the case when multiple client devices 110 are deployed across a particular area or environment, such as a home, building, etc. Additionally or alternatively, in some implementations, multiple client devices 110 1-N and smart devices 145 A-N are client devices 110 that are at least selectively accessible by one or more users (e.g., individuals, family members, employees of an organization, other predefined groups, etc.). 1-N and smart devices 145 A-N In some of these implementations, client devices 110 may be associated with each other by being members of a collaborative ecosystem. 1-N and smart devices 145 1-N The ecosystems can be manually and / or automatically associated with each other in the device topology 152.

[0072] Client Device 110 1-N An instance of an automated assistant client in the cloud, interacting with one or more cloud automated assistant components 120, can form what appears from a user's perspective to be a logical instance of an automated assistant with which the user can conduct a human-computer interaction. For example, a user may use either client device 1101 and automated assistant client 1171, or client device 110 N and automated assistant client 117 N The same logical instance of the automated assistant can be utilized by using the automated assistant clients 1171 and 1172. N The specific instances of may vary (e.g., provide different smart device controls for the same command) and / or may be different I / O components 1111 and 111 N and / or provide user interface output via different I / O components 1111 and 111 N(e.g., I / O component 1111 may include a touch screen, while I / O component 1111 may accept different user interface inputs via N 1101 and 1102), but the user can still effectively utilize the same logical instance of the automated assistant. For simplicity and clarity, the term "automated assistant" as used herein refers to an automated assistant client running on client device 110 and, optionally, one or more cloud automated assistant components 120 (which may be shared among multiple automated assistant clients). Two client devices 1101 and 1102 in a coordinated ecosystem N 1, it is understood that many additional client devices can be included in the ecosystem. Furthermore, it is understood that separate, coordinated ecosystems of client devices can also be provided, each associated with a different user (e.g., account) and / or environment, and that such separate, coordinated ecosystems can also interact with the cloud automated assistant component 120 (provided that the interactions are tailored to those separate ecosystem accounts).

[0073] Client Device 110 1-Nmay include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in the user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone assistant-centric interactive speaker, a standalone assistant-centric interactive display with a speaker, a smart appliance such as a smart television, and / or a user wearable device that includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided.

[0074] As mentioned above, the client device 110 1-N One or more of may at least selectively interface with cloud automated assistant component 120 in processing the input and / or generating output based on the input and / or generating smart device control commands based on the input. Cloud automated assistant component 120 may include STT engine 121, NLU engine 122, TTS engine 123, cache generation engine 125, and / or cloud fulfillment engine 127.

[0075] As mentioned above, the client device 110 1-N Each NLU engine in 113 1-N and / or their respective local caches 158 1-NFor received requests for which there is no matching cache entry in the cloud automated assistant component 110, text and / or audio data corresponding to the request can be sent to the cloud automated assistant component 120. The cloud automated assistant component 120 can utilize its more robust NLU engine 122 to generate a semantic representation based on processing the text of the request. The processed text can be sent by the client device or can be recognized text generated by the STT engine 121 utilizing audio data sent by the client device. The generated semantic representation can include a cloud semantic representation and / or a locally interpretable semantic representation, as described herein. As also described herein, in generating the semantic representation, the NLU engine 122 may use the client device 110 to generate a semantic representation. 1-N and smart devices 145 A-N The device topology 152 can be utilized.

[0076] The cache generation engine 125 is connected to the client device 110. 1-N In response to a transmission from one of the local caches 158, the cache generation engine 125 may generate a cache request including the generated semantic representation and, optionally, a cache entry including text (or a representation thereof) and a mapping of the text to the semantic representation. The cache entry of the generated cache request may include additional and / or alternative content, such as rendered response content, protocol suites, adapters, and / or radios, and / or other content, utilized with the locally interpretable semantic representation. The rendered response content may include text, audio data (e.g., synthesized speech for the response text generated using the TTS engine 123), and / or graphics. The cache generation engine 125 may generate a cache entry for each local cache 158. 1-N, and sends a corresponding cache request to each client device 110 to have the client device store the cache entry. 1-N Each cache engine 118 1-N Send to.

[0077] If the generated semantic representation includes a cloud semantic representation, the cloud fulfillment engine 127 can process the cloud semantic representation to generate a corresponding control request, which can then be transmitted to the corresponding smart device system 140. A-N The smart device system 140 A-N In response to the control request, the smart device 145 generates a corresponding control command and transmits the control command to the corresponding smart device 145. A-N can be sent to.

[0078] If the generated semantic representation includes a locally interpretable semantic representation, each client device 110 A-N generates corresponding local control commands and sends those local control commands (e.g., to the respective local modules 116 A-N Alternatively, the cloud automated assistant component 120 can provide a locally interpretable semantic representation in a separate transmission, which can be sent to each client device 110. A-N generates corresponding local control commands and sends those local control commands (e.g., to the respective local modules 116 A-N As yet another alternative, the cloud automated assistant component 120 itself can generate local control commands and transmit the local control commands to each client device 110 in order to transmit the control commands locally. A-N can be sent to.

[0079] Also as explained above, for received requests, each local cache 158 containing the cloud semantic representation 1-N , the client device 110 that matches the cache entry. 1-N In the cloud fulfillment engine 127, the cloud semantic representation can be sent to the cloud automated assistant component 120 (e.g., instead of the corresponding text and / or audio data). The cloud fulfillment engine 127 can process the received cloud semantic representation to generate a corresponding control request, which can then be sent to the corresponding smart device system 140. A-N The smart device system 140 A-N In response to the control request, the smart device 145 generates a corresponding control command and transmits the control command to the corresponding smart device 145. A-N can be sent to.

[0080] Further description of the various components of Figure 1 will now be provided with reference to additional figures. Figure 2 shows a home floor plan including multiple rooms 250, 252, 254, 256, 258, 260, and 262. Multiple client devices 110 1-3 The client devices 110 are deployed throughout at least some of the rooms. 1-3Each of the client devices 1101 may optionally implement an instance of an automated assistant client configured in accordance with selected aspects of the present disclosure and may optionally include one or more input devices, such as a microphone, a touchscreen, etc., and / or one or more output devices, such as a speaker, a display, etc. For example, a first client device 1101 in the form of an interactive standalone speaker is deployed in room 254, which in this example is a bedroom. A second client device 1102 in the form of a standalone interactive speaker and display device (e.g., a display screen, a projector, etc.) is deployed in room 252, which in this example is a living room. A third client device 1103, also in the form of an interactive standalone speaker, is deployed in room 256.

[0081] multiple client devices 110 1-3 may be communicatively coupled with each other and / or with other resources (e.g., smart devices and the Internet) via wireless router 101 shown in room 252 and / or a local mesh network. Additionally, other client devices (particularly mobile devices such as smartphones, tablets, laptops, wearable devices, etc.) may also be present, e.g., carried by one or more people in the home (e.g., users 103), and may or may not be connected to the same LAN. It should be understood that the configuration of client devices shown in FIG. 2 is merely an example, and that more, fewer, and / or different client devices may be deployed throughout any number of other rooms and / or areas outside the home.

[0082] A number of smart devices are further illustrated in Figure 2. The smart devices include smart lighting 145 A1 Including: Smart Lighting 145 A1 a first remote smart device system 140 AThe smart device can be controlled by a second remote smart device system 140. B Smart lighting controllable by 145 B1 and 145 B2 The second remote smart device system 140 B a first remote smart device system 140 A The smart device may be controlled by a party other than the party controlling the smart thermostat 145. The smart device may be controlled by at least a second client device 1102. C1 For example, smart thermostat 145 C1 Smart Thermostat 145 C1 Compliant with the protocol suite, Smart Thermostat 145 C1 and a second client device 1103. It should be understood that the configuration of smart devices 145 shown in FIG. 2 is only an example, and that more or fewer and / or different smart devices may be deployed throughout any number of other rooms and / or areas outside of a home.

[0083] 2 and the above description of FIG. 2 will now be utilized in describing various aspects of FIGS. 3A, 3B, 4A, 4B, 5A, and 5B.

[0084] 3A shows an example of how an utterance 352, "Turn the lights a little brighter," received at the second assistant client device 1102 of FIG. 2 can be processed if the second assistant client device 1102 does not have a cache entry that matches the recognized text 352A of the utterance 352 (the recognized text 352A is generated using an STT engine on the device), and optionally, if the local NLU engine of the second assistant client device 1102 cannot effectively process the recognized text 352A. In FIG. 3A, the second assistant client device 1102 sends the recognized text 352A to the cloud automated assistant component 120. The sending of the recognized text 352A can be responsive to the second assistant client device 1102 determining that there is no locally stored cache entry that matches the recognized text 352A of the utterance 352, and can optionally be responsive to the local NLU engine of the second assistant client device 1102 that cannot effectively process the recognized text 352A.

[0085] In response to receiving the recognized text 352A, the cloud automated assistant component 120 generates a cloud semantic representation 354 of the recognized text 352A. The cloud semantic representation 354 can be generated based on the recognized text being received from the second assistant client device 1102 and based on a reference to a remotely stored device topology. In generating the cloud semantic representation, the cloud automated assistant component 120 can resolve the “light” in the recognized text 352A to a specific light. For example, the cloud automated assistant component 120 can utilize the account identifier received in the transmission of the text 352A to identify the account identifier's corresponding remotely stored device topology. Additionally, the cloud automated assistant component 120 can utilize the identifier of the second assistant client device 1102 received with the text 352A to identify the second assistant client device 1102 within the device topology. Additionally, the cloud automated assistant component 120 maps the “lighting” in the text 352A to lighting 145 based on the lights that are mapped as default lights for the second assistant client device 1102 in the device topology. B1 and 145 B2 This can be solved as follows: Lighting 145 B1 and 145 B2 may be mapped as the default for the second assistant client device 1102 based on previous user interface input and / or based on all of them being assigned to the room 252 identifier in the device topology.

[0086] In generating the cloud semantic representation, cloud automated assistant component 120 may also resolve "make ... a little brighter" to mean "make it 2 / 5 (e.g., 40%) brighter than the current brightness intensity level." This may be based on references to one or more remotely stored models and / or rules accessible to cloud automated assistant component 120. Cloud semantic representation 354 may be generated based on these resolutions, e.g., [device=145], as shown in cache request 356. B1 and 145 B2 , brighter, relative intensity 2 / 5], where "145 B1 and 145 B2 " is a unique identifier for the corresponding light, and "brighter, relative intensity 2 / 5" indicates that the lights will be about 2 / 5 brighter compared to their current intensity.

[0087] The cloud automated assistant component 120 utilizes the generated cloud semantic representation 354 to generate a corresponding control request. B1 and 145 B2 Compatible with smart device systems 140 B In response, the smart device system 140 B generates corresponding control commands to make the lights "a little brighter" (i.e., increase the relative luminance by 40%, as indicated by the cloud semantic representation) and sends those control commands to the lighting 145 B1 and 145 B2 can be provided to.

[0088] Additionally, the cloud automated assistant component 120 generates and sends to the client device 1102 a cache request 356 that includes the cloud semantic representation 354 and, optionally, the recognized text 352A and / or a mapping of the recognized text to the cloud semantic representation 354. In response to receiving the cache request 356, the client device 1102 stores a corresponding cache entry that includes the mapping of the recognized text 352A to the cloud semantic representation 354 included in the cache request 356.

[0089] FIG. 3B shows an example of how another instance of the same utterance 352 of FIG. 3A, also received at the second assistant client device 1102, can be processed differently after a cache entry having a cloud semantic representation 354 (FIG. 3A) is stored in the second assistant client device 1102 of FIG. 3A.

[0090] 3B, second assistant client device 1102 processes audio data capturing speech 352 to generate recognized text. Furthermore, second assistant client device 1102 determines that the recognized text matches the text of a cache entry stored in the local cache of second assistant client device 1102 in FIG. 3A. In response, second assistant client device 1102 transmits a cloud semantic representation of the matching cache entry to cloud automated assistant component 120. Cloud automated assistant component 120 then generates a corresponding control request and transmits the control request to smart device system 140. B The cloud semantic representation 354 can be directly processed for transmission to the smart device system 140. B generates corresponding control commands to make the lights "a little brighter" and sends those control commands to the lighting 145 B1 and 145B2 In these and other ways, resources of the cloud automated assistant component can be conserved by preventing the generation of cloud semantic representations from occurring again. Furthermore, control requests can be generated and sent more quickly, reducing the time it takes to control lighting 145. B1 and 145 B2 3B may differ from those in FIG. 3A, even though they are generated based on the same semantic representation. This is due to the relative state changes in the cloud semantic representation, and the lighting 145 when generating the control request. B1 and 145 B2 3A and 3B.

[0091] 4A is a diagram illustrating an example of how the same utterance of FIGS. 3A and 3B ("Turn the lights a little brighter") received at first assistant client device 1102 of FIG. 2 can be processed if first assistant client device 1102 does not have a cache entry that matches the text of the utterance. In FIG. 4A, user 103 provides utterance 452, "Turn the lights a little brighter." Utterance 452 is the same utterance as utterance 352 of FIGS. 3A and 3B, but is labeled as 452 in FIGS. 4A and 4B for consistency in numbering.

[0092] 4A, first assistant client device 1101 uses a local STT engine to send recognized text 452A generated by processing audio data capturing utterance 452 to cloud automated assistant component 120. The sending of recognized text 452A can be responsive to first assistant client device 1101 determining that there is no locally stored cache entry that matches recognized text 452A of utterance 452, and can optionally be responsive to the local NLU engine of first assistant client device 1101 being unable to effectively process recognized text 452A.

[0093] In response to receiving the recognized text 452A, the cloud automated assistant component 120 generates a cloud semantic representation 454 of the recognized text 452A. The cloud semantic representation 454 may be, for example, a representation of the recognized text 452A, as shown in cache request 456, where [device=145 A1 , brighter, relative intensity 2 / 5], where "145 A1 " is a unique identifier for the corresponding light, and "brighter, relative intensity 2 / 5" indicates that the lights will be about 2 / 5 brighter compared to their current intensity. B1 and 145 B2 Identifiers instead of lighting 145 A1 3A in that it utilizes the identifier of first assistant client device 1101 received with text 452A to identify first assistant client device 1101, and maps "light" in text 452A to light 145 as the default light for first assistant client device 1101 based on that light being mapped in the device topology. A13A and 4A, different cloud semantic representations can be generated based at least in part on the utterances received from different assistant client devices.

[0094] The cloud automated assistant component 120 utilizes the generated cloud semantic representation 454 to generate a corresponding control request. The cloud automated assistant component 120 performs illumination 145 of the semantic representation 454. A1 Compatible with smart device systems 140 A In response, the smart device system 140 A generates corresponding control commands to make the lights "a little brighter" and sends those control commands to the lighting 145 A1 can be provided to.

[0095] Additionally, the cloud automated assistant component 120 generates and sends to the first client device 1101 a cache request 456 that includes the cloud semantic representation 454 and, optionally, the recognized text 452A and / or a mapping of the recognized text to the cloud semantic representation 454. In response to receiving the cache request 456, the first client device 1101 stores a corresponding cache entry that includes the mapping of the recognized text 452A to the cloud semantic representation 454 included in the cache request 456.

[0096] FIG. 4B illustrates an example of how another instance of the same utterance 452 of FIG. 4A received at the first assistant client device 1101 can be processed differently after a cache entry having a cloud semantic representation 454 (FIG. 4A) is stored in the first assistant client device 1101 of FIG. 4A.

[0097] 4B, first assistant client device 1101 processes audio data capturing speech 452 to generate recognized text. Furthermore, first assistant client device 1101 determines that the recognized text matches the text of a cache entry stored in the local cache of first assistant client device 1101 in FIG. 4A. In response, first assistant client device 1101 sends cloud semantic representation 454 of the matching cache entry to cloud automated assistant component 120. Cloud automated assistant component 120 then generates a corresponding control request and transmits the control request to smart device system 140. A In response, the smart device system 140 can process the cloud semantic representation 454 directly for transmission to the cloud. A generates corresponding control commands to make the lights "a little brighter" and sends those control commands to the lighting 145 A1 can be provided to.

[0098] FIG. 5A shows an example of how another utterance 552, "Turn the temperature down a little," received at the second assistant client device 1102 of FIG. 2 can be processed if the second assistant client device 1102 does not have a cache entry that matches the recognized text 552A of the utterance 552 (generated using an STT engine on the device), and optionally if the local NLU engine of the second assistant client device 1102 cannot effectively process the recognized text 552A.

[0099] 5A, second assistant client device 1102 sends recognized text 552A to cloud automated assistant component 120. The sending of recognized text 552A can be responsive to second assistant client device 1102 determining that there is no locally stored cache entry that matches recognized text 552A of utterance 552, and optionally, can be responsive to the local NLU engine of second assistant client device 1102 being unable to effectively process recognized text 552A.

[0100] In response to receiving the recognized text 552A, the cloud automated assistant component 120 generates a locally interpretable semantic representation 554 of the recognized text 552A. In some implementations, the cloud automated assistant component 120 generates a locally interpretable semantic representation 554 of the recognized text 552A (the thermostat 145 C1 ) is locally controllable by the second assistant client device 1102 and / or another client device in network communication with the second assistant client device 1102, generates a locally interpretable semantic representation 554 (e.g., instead of or in addition to a cloud-based one).

[0101] The request to "lower the temperature" is thermostat 145 C1 To determine that the text 552A is directed to the thermostat 145, a locally interpretable semantic representation 554 can be generated based on a reference to the remotely stored device topology. (For example, it may be the only smart device in the device topology that has "turn down the temperature" as a supported state change.) Thus, the text 552A may be directed to the thermostat 145. C1may be resolved to inferentially reference the target smart device. In generating the cloud semantic representation, cloud automated assistant component 120 may also resolve "turn ... down a little" to mean "two degrees below the current setpoint." This may be based on reference to one or more remotely stored models and / or rules accessible to cloud automated assistant component 120. Locally interpretable semantic representations 554 may be generated based on these resolutions and may further be generated to allow the client device to include a relative state change representation that instructs how to locally resolve the target setpoint of "two degrees below the current setpoint." For example, a relative state change representation may be "temperature setting = (current temperature - 2 degrees)."

[0102] The locally interpretable semantic representation 556 may optionally further include an indication of an adapter (e.g., an adapter specific to the manufacturer of the smart device) that should be utilized in generating the corresponding local control command, an indication of a channel (e.g., Bluetooth or Wi-Fi) that should be utilized to transmit the local control command, and / or an indication of a protocol suite that should be utilized in generating the control command based on the locally interpretable semantic representation 556. For example, as shown in cache request 556, the locally interpretable semantic representation may include an indication of a protocol suite that should be utilized in generating the control command based on the locally interpretable semantic representation 556. C1 , temperature setting = (current temperature - 2 degrees), adapter / channel = C], and in the above formula, "145 C1 " is thermostat 145 C1 A unique identifier for the thermostat (for example, thermostat145 C1's local address), "Temperature Setting = (Current Temperature - 2 degrees)" is a representation of a state change that depends on the current state that can be locally interpreted to resolve the current state change, and "Adapter / Channel = C" indicates the adapter and / or protocol suite that should be utilized in generating the local control command (e.g., after determining the current state change) and / or the channel that should be utilized to transmit the local control command.

[0103] The cloud automated assistant component 120 generates and sends to the second client device 1102 a cache request 556 that includes the cloud semantic representation 554 and, optionally, the recognized text 552A and / or a mapping of the recognized text to the cloud semantic representation 554. In response to receiving the cache request 556, the second client device 1102 stores a corresponding cache entry that includes the mapping of the recognized text 552A to the cloud semantic representation 554 included in the cache request 556.

[0104] In some implementations, the second client device 1102 and the thermostat 145 C1 The second client device 1102 is connected to the thermostat 145, as indicated by the dashed line between C1 and locally generate local control commands for the thermostat 145 in response to the utterance of FIG. 5A. C1 A locally interpretable semantic representation of the cache request (or from a separate transmission from the cloud automated assistant component 120) can be utilized to send it locally to the cloud automated assistant component 120.

[0105] In some alternative implementations, the cloud automated assistant component 120 generates a corresponding cloud semantic representation based on the recognized text 552A. Additionally, the cloud automated assistant component 120 can utilize a separate cloud semantic representation to generate a corresponding control request. The cloud automated assistant component 120 can generate a corresponding control request based on the recognized text 552A. C1 Compatible with smart device systems 140 C In response, the smart device system 140 C generates corresponding control commands to make its setpoint "a little lower" (i.e., 2 degrees lower as indicated by the cloud semantic representation) and sends those control commands to the thermostat 145 C1 can be provided to.

[0106] FIG. 5B shows an example of how another instance of the same utterance 552 of FIG. 5A received at the second assistant client device 1102 can be processed differently after a cache entry having a locally interpretable semantic representation 554 (FIG. 5A) is stored in the second assistant client device 1102 in FIG. 5A.

[0107] 5B, second assistant client device 1102 processes audio data capturing utterance 552 to generate recognized text. Furthermore, second assistant client device 1102 determines that the recognized text matches the text of a cache entry stored in the local cache of second assistant client device 1102 in FIG. 4A. In response, second assistant client device 1102 identifies a locally interpretable semantic representation 554 of the cache entry and uses the locally interpretable semantic representation 554 to generate a corresponding control command. The state representation of locally interpretable semantic representation 554 is a representation of a state that depends on the current state of thermostat 145. C1 5A and 5B, the generated control commands in FIG. 5B may differ from those in FIG. 5A. For example, in FIG. 5B, second assistant client device 1102 may resolve the target set point by subtracting 2 degrees from the current set point, as dictated by the relative state change in locally interpretable semantic representation 554. Second assistant client device 1102 then generates a control command and transmits the control command to thermostat 145, which is optionally included in locally interpretable semantic representation 554. C1 In response to receiving the control command, the thermostat 145 may utilize a corresponding adapter to process the target set point and transmit it to the address of the C1 can lower its set point by two degrees.

[0108] FIG. 6 is a flowchart illustrating an example method 600 that may be implemented by one or more processors of an assistant client device according to various implementations disclosed herein.

[0109] In block 602, the assistant client device processes the audio data capturing the speech to generate recognized text of the speech. In other implementations, instead of processing the audio data to generate recognized text, the assistant client device can identify provided text based on directly typed text (e.g., using a virtual keyboard) or based on corresponding hardware or software button presses (e.g., the text "Dim the lights" mapped to a graphical software button labeled "Dim").

[0110] In block 604, the assistant client device determines whether the text matches a local cache entry. If so, the client device proceeds to block 606 and selects a semantic representation from the matching local cache entry. Then, in block 608, the system uses the semantic representation. Block 608 can include sub-blocks 608A and / or 608B. In sub-block 608A, the client device locally processes any of the selected semantic representations, which are locally interpretable semantic representations, to generate local control commands and transmit the control commands via a local channel. In sub-block 608B, the client device transmits any of the selected semantic representations, which are cloud semantic representations, to a remote system. For example, they can be transmitted to a remote system that can perform blocks 812 and 814 of method 800 (FIG. 8, described below). In some implementations, block 606 can include additional sub-blocks, such as a sub-block in which the assistant client device renders any response content included in the local cache entry. For example, the local cache entry can include response text, and the assistant client device can perform local TTS processing on the response text and audibly render the resulting synthesized speech. As another example, the local cache entry can additionally or alternatively include a response graphic, and the graphic can be visually rendered on the display of the assistant client device.

[0111] If the determination in block 604 is no, the assistant client device proceeds to optional block 610 to determine whether the text is resolvable locally (e.g., using a local NLU engine and a local model). If so, the system proceeds to block 612 to generate a semantic representation (locally interpretable and / or cloud), and then proceeds to block 608 to use the generated semantic representation.

[0112] If the determination in block 610 is NO (or if the determination in 604 is NO and 610 is omitted), the client device proceeds to block 614 and transmits the audio data and / or text to the remote assistant system. Optionally, block 614 may occur before and / or during execution of blocks 604 and / or 610, with further processing at the remote system optionally stopped in response to a "YES" determination in either of blocks 604 and 610 (e.g., in response to a stop of transmission from the client device). For example, if block 614 occurs before and / or during execution of blocks 604 and / or 610, and the assistant client device makes a "YES" determination in blocks 604 and / or 610, a stop request can be sent to the remote assistant system to stop any further processing by the remote assistant system related to the transmitted audio data and / or text. These and other schemes may conserve remote assistant system resources by reducing latency when the determination at blocks 604 and / or 610 is "no" and by allowing such processing to stop when the determination is "yes," while allowing processing at the remote assistant system to begin more quickly.

[0113] In block 616, the client device receives a cache request from the remote assistant system, including a semantic representation of the text and optionally the text itself (and / or its representation). Block 616 optionally includes sub-block 616A, where the assistant client device optionally proceeds to block 608A to process one or more locally interpretable semantic representations (if any) of the cache request (or a separate transmission). In other words, in block 616, the assistant client device can immediately act on the locally interpretable semantic representations of the cache request to enable any local smart device actions in response to the current utterance or other current user input.

[0114] At block 618, the client device stores in a local cache and, in response to receiving the cache request, a cache entry including a mapping of the text to the semantic representation of the cache request.

[0115] Block 618 optionally includes sub-block 618A, which may occur near or after block 618. In sub-block 618A, the client device optionally assigns the text of the cache entry for an invocation-free action. For example, the client device can assign the text as a "hot phrase" as described herein. The assignment of the text as a "hot phrase" can optionally be based on a determination of whether one or more criteria are met. If the recognized text from on-device speech recognition of audio data occurring independent of an explicit automated assistant invocation is determined to match a hot phrase, the assistant client device can automatically initiate one or more corresponding actions based on the text (e.g., actions based on the semantic representation of the corresponding cache entry).

[0116] 7 is a flowchart illustrating another exemplary method 700 that may be implemented by one or more processors of an assistant client device according to various implementations disclosed herein. Method 700 illustrates a specific implementation of method 600 of FIG. 6, where the text is determined to match the text of a cache entry (i.e., "Yes" in block 604 of FIG. 6), and the semantic representation of the cache entry includes a locally interpretable semantic representation.

[0117] In block 702, the assistant client device processes the audio data capturing the speech to generate recognized text of the speech. In other implementations, instead of processing the audio data to generate recognized text, the assistant client device can identify provided text based on directly typed text (e.g., using a virtual keyboard) or based on corresponding hardware or software button presses (e.g., the text "Dim the lights" mapped to a graphical software button labeled "Dim").

[0118] In block 704, the assistant client device determines that the text matches the text of the cache entry. Block 704 optionally includes sub-block 704A, in which the client device optionally determines that the text is assigned to an uninvoked action (e.g., at the assistant client device). For example, block 704A can be optionally executed if the processing of block 702 occurs without an explicit invocation of an automated assistant. Furthermore, in such an example, one or more subsequent blocks of method 700 can be executed only if it is determined that the text is assigned to an uninvoked action (e.g., locally at the client device). In other words, block 704A can be executed when the processing of block 702 occurs without an explicit invocation of an automated assistant, and if it is determined that the text is not assigned to an uninvoked action, the text can be discarded without executing further blocks of method 700 based on the text, but if it is determined that the text is assigned to an uninvoked action, further blocks of method 700 are executed based on the text.

[0119] In block 706, the client device selects a locally interpretable semantic representation based on which it is mapped in the cache entry to the text of the cache entry.

[0120] In block 708, the client device determines whether the state change in the locally interpretable semantic representation is a state change that is dependent on the current state. If the state change is not dependent on the current state, the client device proceeds to block 712 and uses the state change in the locally interpretable semantic representation. For example, if the state change indicates a fixed value, the fixed value may be utilized as the state change. If the determination in block 708 is "yes," the client device proceeds to block 710 and resolves the state change based on the current state value. For example, if the state change is dependent on the current state, it may be resolved by identifying the current state of one or more smart devices and then determining the state change based on the identified current state. For example, if the state change is "set point = (current temperature + 3 degrees)," it may be resolved by identifying "current temperature."

[0121] At block 714, the client device generates a local control command using the state change and, optionally, using the adapter identified based on the cache entry.

[0122] At block 716, the client device sends local commands over a local channel optionally selected based on the cache entry (e.g., explicitly identified in the cache entry or selected based on the protocol suite, smart device, and / or radio / channel identified in the cache entry). The sent local commands may be addressed to the smart devices based on their addresses identified in or resolvable from locally interpretable semantic representations.

[0123] At block 718, the client device determines whether there are any additional unprocessed semantic representations in the cache entry. If there are no semantic representations, the client device proceeds to block 720 and method 700 ends. If there are semantic representations, the client device proceeds to block 722 and determines whether the next unprocessed semantic representation is locally interpretable. If it is locally interpretable, the client device returns to block 708. If it is not locally interpretable (i.e., it is a cloud semantic representation), the client device proceeds to block 608B of method 600 of FIG. 6 and then returns to block 718 after performing block 608B.

[0124] FIG. 8 is a flowchart illustrating an example method 800 that may be implemented by one or more processors of a remote assistant system according to various implementations disclosed herein.

[0125] In block 802, the system receives a request from an assistant client device. In block 804, the system determines whether the request is a cloud semantic representation. If the request is a cloud semantic representation, the system executes blocks 812 and 814 (described below) of method 800 without executing any other blocks of method 800.

[0126] If the request is not a cloud semantic representation, the system proceeds to block 808 and generates a semantic representation of the utterance based on the representation of the utterance in the request. In other implementations, the representation may be typed speech or text mapped to the pressing of a hardware or software graphical element.

[0127] In optional block 810, the system generates confirmatory content for rendering at the assistant client device in response to the request.

[0128] In optional block 812, the system generates a control request based on the cloud semantic representation generated based on the request. Optional block 812 may include optional block 812A, where if the state representation of the semantic representation is a relative / current state-dependent state representation, the system generates the control request based on the current state of the smart device.

[0129] In optional block 814, the system sends the generated control request to the smart device remote system.

[0130] In block 816, the system generates a cache request that includes the semantic representation (locally interpretable and / or cloud) and optionally includes a mapping of text to the semantic representation.

[0131] In block 818, the system sends a cache request to the assistant client device. Block 818 optionally includes sub-block 818A, where if the semantic expression of the cache request is a locally interpretable semantic expression, the system optionally generates another request to implement the locally interpretable semantic expression in response to the current request.

[0132] 9 is a block diagram of an exemplary computing device 910 that may optionally be utilized to perform one or more aspects of the techniques described herein. The computing device 910 typically includes at least one processor 914 that communicates with several peripheral devices via a bus subsystem 912. These peripheral devices may include, for example, a storage subsystem 924 including a memory subsystem 925 and a file storage subsystem 928, a user interface output device 920, a user interface input device 922, and a network interface subsystem 918. The input and output devices enable user interaction with the computing device 910. The network interface subsystem 918 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0133] The user interface input devices 922 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touchscreen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 910 or onto a communications network.

[0134] The user interface output devices 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and ways of outputting information from the computing device 910 to a user or to another machine or computing device.

[0135] Storage subsystem 924 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 924 may include logic for performing selected aspects of one or more methods described herein.

[0136] These software modules are generally executed by the processor 914 alone or in combination with other processors. The memory 925 used in the storage subsystem 924 can include several memories, including a main random access memory (RAM) 930 for storing instructions and data during program execution, and a read-only memory (ROM) 932 in which fixed instructions are stored. The file storage subsystem 928 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules that implement the functionality of a particular implementation may be stored by the file storage subsystem 928 in the storage subsystem 924 or on other machines accessible by the processor 914.

[0137] The bus subsystem 912 provides a mechanism that allows the various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0138] Computing device 910 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 910 shown in Figure 9 is intended only as a specific example to illustrate some implementations. Many other configurations of computing device 910 are possible, having more or fewer components than the computing device shown in Figure 9.

[0139] In situations where particular implementations described herein may collect or use personal information about users (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, the user's activity and demographic information, relationships between users, etc.), users are provided with one or more opportunities to control whether the information is collected, whether the personal information is stored, whether the personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods described herein collect, store, and / or use a user's personal information only upon receiving explicit permission to do so from the associated user.

[0140] In some implementations, a method is provided that includes receiving, in a remote assistant system and from an assistant client device, a representation of an utterance captured at the assistant client device. The utterance is a request related to one or more states of one or more smart devices linked to the assistant client device. As an example, the request may be a request to change a state of the smart device (i.e., transition the state from a current state value to a new state value). The representation of the utterance comprises audio data capturing the utterance and / or text of the utterance, generated at the client device using a speech-to-text model stored locally on the client device. The method further includes, at the remote system and in response to receiving a representation of the utterance, generating a semantic representation of the utterance based on the representation of the utterance, generating at least one control request based on the semantic representation of the utterance for transmission to at least one smart device remote system that controls one or more smart devices, where the at least one control request is different from the semantic representation, transmitting the at least one control request to the at least one smart device remote system to cause the at least one smart device remote system to provide one or more corresponding commands to the one or more smart devices, and transmitting a cache request including the semantic representation to the assistant client device. Sending the cache request to the assistant client device causes the assistant client device to store a cache entry including a mapping of text to the semantic representation in a cache of the assistant client device.

[0141] Implementations of these and other techniques may optionally include one or more of the following features.

[0142] In some implementations, after transmitting a cache request to the assistant client device, the method further includes capturing additional audio data at the assistant client device that captures the additional utterance, processing the additional audio data using a speech-to-text model stored locally on the assistant client device to generate current text corresponding to the additional utterance, determining that the current text matches the text of the cache entry, and sending the semantic representation to the remote assistant system in response to determining that the current text matches the text of the cache entry and in response to the cache entry including a mapping of the text to a semantic representation. In some of these implementations, processing the additional audio data to generate the current text is performed without detecting an explicit automated assistant invocation at the assistant client device. In some versions of these implementations, sending the semantic representation to the remote assistant system is further responsive to determining that the current text meets one or more criteria for enabling the matching text to be acted upon without an explicit automated assistant invocation at the assistant client device. In some additional or alternative versions, the method further includes, at the remote assistant system, receiving a semantic representation sent from the assistant client device; generating, based on the semantic representation, at least one alternate control request for transmission to at least one smart device remote system that controls one or more smart devices, wherein the at least one alternate control request is different from the at least one control request; and transmitting the at least one alternate control request to the at least one smart device remote system to cause the at least one smart device remote system to change one or more states of the one or more smart devices.Optionally, in those alternative or additional implementations, the step of generating the at least one control request is further based on at least one current state of the one or more smart devices at the time of generation of the at least one control request, and the step of generating the at least one alternative control request is further based on at least one alternative current state of the one or more smart devices at the different time of generation of the at least one alternative control request, and the at least one alternative control request differs from the at least one control request based on at least one current state that differs from the at least one alternative current state.

[0143] In some implementations, generating a semantic representation of the utterance is further based on a device topology of the account for the assistant client device, the device topology defining the assistant client device, one or more smart devices, the additional assistant client device, and the additional smart device. In some of these implementations, generating a semantic representation of the utterance further based on the device topology includes resolving, based on the device topology, that ambiguous terms in the text of the utterance refer to one or more smart devices.

[0144] In some implementations, the semantic representation includes a state change representation of one or more state changes and a corresponding unique identifier for each of the one or more smart devices. In some versions of these implementations, the state change representation is a relative state change representation that defines the one or more state changes in a relative manner rather than an absolute manner. In some additional or alternative versions of these implementations, the method further includes, at the remote assistant system or the assistant client device, determining a change to the device topology and, in response to determining the change to the device topology, clearing a cache entry from the cache of the assistant device. Determining the change to the device topology optionally includes determining that the change is to one or more properties defined in the device topology for the assistant client device, the one or more smart devices, and / or one or more rooms assigned to the assistant client device and / or the one or more smart devices; and optionally, clearing the cache entry from the cache can further be responsive to determining that the cache entry is affected by the one or more properties.

[0145] In some implementations, the method further includes receiving, at the remote assistant system and from an additional assistant client device, an additional representation of the utterance. The utterance is captured at the additional assistant client device, and both the additional assistant client device and the assistant client device are members of a common device topology. In some of these implementations, the method further includes, in response to receiving, at the remote system and the additional representation of the utterance, generating an alternative semantic representation of the utterance based on the representation of the utterance, where the alternative semantic representation of the utterance differs from the additional representation captured at the additional assistant client device and the semantic representation based on the representation captured at the assistant client device; and sending an additional cache request to the additional assistant client device that includes the alternative semantic representation. Upon sending the additional cache request to the additional assistant client device, the additional assistant client device stores an additional cache entry in an additional cache of the additional assistant client device that includes a mapping of text to the alternative semantic representation.

[0146] In some implementations, a method is provided that includes receiving, at a remote assistant system and from an assistant client device, a representation of an utterance captured at the assistant client device. The utterance is a request to change at least a given state of a given smart device linked to the assistant client device. The representation of the utterance includes audio data capturing the utterance and / or text of the utterance generated at the client device using a speech-to-text model stored locally on the client device. The method further includes, at the remote system and in response to receiving the representation of the utterance, determining that the given smart device is locally controllable by the assistant client device and / or by an additional client device having a local connection to the assistant client device; generating a semantic representation of the utterance including a locally interpretable semantic representation; and sending a cache request to the assistant client device that includes the semantic representation. The locally interpretable semantic representation is locally interpretable by the assistant client device and / or the additional client device to generate a corresponding control command that can be transmitted over a local channel to cause at least a given state change of the given smart device. The step of generating a locally interpretable semantic representation of the at least one smart device includes, in response to determining that the given smart device is locally controllable, including the locally interpretable semantic representation in the semantic representation. When the cache request is sent to the assistant client device, the assistant client device stores a cache entry in the cache of the assistant client device, the cache entry including a mapping of text to the semantic representation.

[0147] Implementations of these and other techniques may optionally include one or more of the following features.

[0148] In some implementations, the method further includes, after transmitting the cache request to the assistant client device, capturing additional audio data at the assistant client device capturing the additional utterance, processing the additional audio data using a speech-to-text model stored locally on the assistant client device to generate current text corresponding to the additional utterance, determining that the current text matches the text of the cache entry, and processing the locally interpretable semantic representation at the assistant client device to generate a given control command in response to determining that the current text matches the text of the cache entry and in response to the cache entry including a mapping of the text to a semantic representation, and transmitting the given control command via a local channel to cause a change in at least a given state of the given smart device. In some versions of these implementations, the locally interpretable semantic representation includes an identifier of the given smart device and a current-state-dependent state change representation that defines a change in at least a given state of the given smart device relative to a corresponding current value of the given state of the given smart device. In some of these versions, processing the locally interpretable semantic representation to generate the given control command includes, at the assistant client device, determining a current value of a given state of the given smart device, and generating the given control command based on applying the current value to a state change representation that depends on the current state. Optionally, the locally interpretable semantic representation further includes instructions from a manufacturer of the smart device, and generating the given control command optionally includes utilizing an adapter stored locally on the assistant client device in response to an adapter assigned to the instructions from the manufacturer of the smart device.Optionally, the method further includes selecting a local channel for transmitting the given control command based on the local channel being directly or indirectly indicated in the locally interpretable semantic representation, for example based on a local channel assigned in a manufacturer's instructions for the smart device included in the locally interpretable semantic representation and / or assigned to a protocol suite included in the locally interpretable semantic representation.

[0149] In some implementations, the step of processing the additional audio data to generate the current text is performed without detecting an explicit automated assistant call at the assistant client device. In some of these implementations, the step of sending the given control command is further responsive to a determination that the current text satisfies one or more criteria for enabling the matching text to act without an explicit automated assistant call at the assistant client device.

[0150] In some implementations, generating a semantic representation of the utterance is further based on a device topology of the account for the assistant client device. The device topology defines the assistant client device, the given smart device, the additional assistant client device, and the additional smart device. In some of these implementations, generating a semantic representation of the utterance includes resolving ambiguous terms in the text of the utterance to refer to the given smart device.

[0151] In some implementations, the method further includes, at the remote assistant system or the assistant client device, determining a change to the device topology and, in response to determining the change to the device topology, clearing a cache entry from the cache of the assistant client device. In some of these implementations, determining the change to the device topology includes determining that the change is to one or more properties defined in the device topology for the assistant client device, the given smart device, and / or one or more rooms assigned to the assistant client device and / or the given smart device, and clearing the cache entry from the cache is further responsive to determining that the cache entry is affected by the one or more properties.

[0152] In some implementations, a method is provided that includes storing a cache entry in a cache on a client device, the cache entry including a mapping of text to a semantic representation. The semantic representation includes a locally interpretable semantic representation that is interpretable locally by the client device. The method further includes capturing, at the client device, additional audio data capturing the utterance. The method further includes processing the additional audio data to generate current text corresponding to the utterance using a speech-to-text model stored locally on the client device. The method further includes determining that the current text matches text of the cache entry. The method further includes processing, at the client device, the locally interpretable semantic representation to generate a given control command in response to determining that the current text matches the text of the cache entry and in response to the cache entry including the mapping of text to a semantic representation, and transmitting the given control command via a local channel to cause a change in at least a given state of the given smart device. [Explanation of symbols]

[0153] 101 Wireless Router 103 users 1051 Wide Area Network 1052 Wide Area Network 1053 Wide Area Network 1101 client device, first client device 110 1-N Client Device 1102 Second Client Device 1103 Third Client Device 111 I / O Components 111 1-N I / O Components 112 STT engine 112 1-N Speech-to-Text (STT) Engine 1131NLU engine 113 1-N Natural Language Understanding (NLU) Engine 114 1-N Fulfillment Engine 1151 Cloud Module 115 1-N Cloud Module 1161 Local Module 116 1-N Local Modules 1171 Automated Assistant Client, TTS Engine 117 1-N Text-to-Speech (TTS) Engine 118 1-N Cache Engine 1191 Radio 120 Cloud Automated Assistant Components 120 Cloud-Based Components 121 STT engine 122 NLU engine 123 TTS engine 125 Cache Generation Engine 127 Cloud Fulfillment Engine 140 Smart Device System 140 A The first remote smart device system 140 B Second Remote Smart Device System 140 A-N Smart Device System 145 A-N Smart Devices 145 1-N Smart Devices 145 A1 Smart Lighting 145 A1 The first smart device 145 B1 Second Smart Device 145 B1Smart Lighting 145 B2 The third smart device 145 B2 Smart Lighting 145 C1 Smart Thermostat 152 Device Topology 152 1-N STT model 153 1-N NLU model 156 1-N adapter 1581 Local Cache 158 1-N Local Cache 250 rooms 252 rooms 254 rooms 256 rooms 258 rooms 260 rooms 262 rooms 352 utterances 352A Recognized Text 354 Cloud Semantic Representation 356 Cache Requests 452 utterances 452A Recognized Text 454 Cloud Semantic Representation 456 Cache Request 552 utterances 552A Recognized Text 554 Cloud Semantic Representation, Locally Interpretable Semantic Representation 556 Cache Requests 600 ways 700 methods 800 ways 910 Computing Devices 912 Bus Subsystem 914 processor 916 Network Interface Subsystem 920 User Interface Output Device 922 User Interface Input Devices 924 Storage Subsystem 925 Memory Subsystem 926 File Storage Subsystem 930 Main Random Access Memory (RAM) 932 Read-Only Memory (ROM)

Claims

1. storing a cache entry in a cache on a client device, the cache entry including a mapping of text to a semantic representation, the semantic representation being locally interpretable by the client device to generate control commands for at least a given smart device of one or more smart devices included in a device topology, and at least one control command differing from the semantic representation; identifying changes to the device topology after storing the cache entry; determining whether to update the cache entry based on the change to the device topology in response to identifying the change to the device topology; capturing and processing user utterances related to the changes to the device topology; In response to determining to update the cache entry, generating and storing an updated cache entry to replace the cache entry in the cache on the client device, the updated cache entry including an updated mapping, the updated mapping comprising: mapping the text to an updated semantic representation based on the changes to the device topology; an updated text mapping to the semantic representation based on the changes to the device topology; or mapping the updated text to the updated semantic representation; 11. A method implemented by one or more processors of a client device, comprising:

2. responsive to generating and storing the updated cache entry, flushing the cache entry on at least the client device. The method of claim 1 further comprising:

3. 10. The method of claim 1, wherein the device topology is created at least in part by a user and includes a corresponding identifier for each of the smart devices included in the device topology and includes one or more corresponding attributes for each of the smart devices included in the device topology.

4. The method of claim 1 , wherein the change to the device topology includes adding a new smart device to the device topology.

5. 2. The method of claim 1, wherein the change to the device topology includes a change in the name of the given smart device, and the updated mapping of the updated cache entry is a mapping of the updated text to the semantic representation or a mapping of the updated text to the updated semantic representation.

6. 2. The method of claim 1, wherein the change to the device topology includes assigning an additional smart device to a room or group to which at least the given smart device is already assigned, and the updated mapping of the updated cache entry is a mapping of the text to the updated semantic representation or a mapping of the updated text to the updated semantic representation.

7. 2. The method of claim 1, wherein determining whether to update the cache entry based on the change to the device topology comprises determining whether the change to the device topology affects the text and / or the semantic representation of the cache entry.

8. a memory for storing instructions; and one or more processors operable to execute the instructions to perform operations, the operations comprising: storing a cache entry in a cache on a client device, the cache entry including a mapping of text to a semantic representation, the semantic representation being locally interpretable by the client device to generate control commands for at least a given smart device of one or more smart devices included in a device topology, and at least one control command differing from the semantic representation; identifying changes to the device topology after storing the cache entry; determining whether to update the cache entry based on the change to the device topology in response to identifying the change to the device topology; capturing and processing user utterances related to the changes to the device topology; In response to determining to update the cache entry, generating and storing an updated cache entry to replace the cache entry in the cache on the client device, the updated cache entry including an updated mapping, the updated mapping comprising: mapping the text to an updated semantic representation based on the changes to the device topology; an updated text mapping to the semantic representation based on the changes to the device topology; or a mapping of the updated text to the updated semantic representation; and Including, the system.

9. one or more of said processors:

9. The system of claim 8, further operable to execute the instructions to flush the cache entry on at least the client device in response to generating and storing the updated cache entry.

10. 10. The system of claim 8, wherein the device topology is created at least in part by a user and includes a corresponding identifier for each of the smart devices included in the device topology and includes one or more corresponding attributes for each of the smart devices included in the device topology.

11. The system of claim 8 , wherein the change to the device topology includes adding a new smart device to the device topology.

12. 9. The system of claim 8, wherein the change to the device topology includes a change in the name of the given smart device, and the updated mapping of the updated cache entry is a mapping of the updated text to the semantic representation or a mapping of the updated text to the updated semantic representation.

13. 9. The system of claim 8, wherein the change to the device topology includes assigning an additional smart device to a room or group to which at least the given smart device is already assigned, and the updated mapping of the updated cache entry is a mapping of the text to the updated semantic representation or a mapping of the updated text to the updated semantic representation.

Images