Efficient and low latency automated assistant control of smart devices

By caching semantic representations on client devices and utilizing remote systems only when needed, the method addresses latency and resource issues in smart device control, enhancing efficiency and speed in managing smart devices.

JP2024160306AActive Publication Date: 2024-11-13GOOGLE LLC
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2024130807
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-10-15
Filing Date
2024-08-07
Publication Date
2024-11-13
Estimated Expiration
2039-12-11

AI Technical Summary

Technical Problem

Existing techniques for controlling smart devices through automated assistants suffer from high latency, excessive resource usage, and network resource consumption due to the need for remote processing of user inputs, particularly in scenarios where cloud-based systems are geographically distant and hardware limitations of client devices hinder local processing.

Method used

Implementing a caching mechanism on client devices to store semantic representations of user inputs, allowing for local control of smart devices when possible, and leveraging remote systems only when necessary, thereby reducing latency and resource usage.

Benefits of technology

This approach reduces latency and resource consumption by enabling local processing of smart device control commands, allowing for faster and more efficient management of smart devices with minimal reliance on remote systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024160306000001_ABST
    Figure 2024160306000001_ABST
Patent Text Reader

Abstract

To provide methods, devices and programs for efficiently and with low latency controlling automatic assistants on smart devices.SOLUTION: A method includes: processing audio data that captures speech so that an assistant client device generates a recognized text of the speech; determining that the text matches the text of the cache entry; selecting a locally interpretable semantic representation based on the client device being mapped to the text of the cache entry; determining whether a state change of the locally interpretable semantic representation is the state change that depends on the current state; using the state change and optionally using an adapter identified based on the cache entry, to generate a local control command, and sending a local command through an optionally selected local channel based on the cache entry.SELECTED DRAWING: Figure 7
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to efficient, low latency automated assistant control of smart devices. [Background technology]

[0002] Humans can engage in human-computer interaction using interactive software applications referred to herein as "automated assistants." For example, a human (which may be referred to as a "user" when interacting with an automated assistant) provides input to the automated assistant that can cause the automated assistant to generate and provide a response output to control one or more smart devices and / or perform one or more other functions. The input provided by the user may be, for example, touch input (e.g., via a touchscreen), gestures (e.g., detected via a camera), and / or spoken natural language input (i.e., speech detected via a microphone), which may be converted into text (or other semantic representation) in some cases and then further processed.

[0003] In many cases, an automated assistant includes an automated assistant client that runs locally on an assistant client device and is utilized directly by a user, and a cloud counterpart that leverages the more robust resources of the cloud to assist the automated assistant client in responding to user input. For example, the automated assistant client can provide the cloud counterpart with an audio recording of the user's speech (or a text transcription thereof) and, optionally, data indicating the user's identity (e.g., credentials). The cloud counterpart can perform various processing on the query to return results to the automated assistant client, which can then provide the corresponding output to the user.

[0004] Many users may utilize automated assistants using multiple client devices. For example, some users own a coordinated "ecosystem" of client devices, such as smartphones, tablet computers, vehicle computing systems, wearable computing devices, smart televisions, standalone assistant-centric interactive speakers, and interactive displays with standalone assistant-centric speakers, among other client devices. A user may use any of these client devices to conduct human-computer interactions with an automated assistant (assuming an automated assistant client is installed). In some cases, these client devices may be scattered throughout a user's primary residence, secondary residence, workplace, and / or other structures. For example, mobile client devices, such as smartphones, tablets, and smart watches, may be worn by the user and / or located where the user last left them. Other client devices, such as traditional desktop computers, smart televisions, and standalone assistant-centric devices, may be more fixed, but may be located in various locations (e.g., rooms) within a user's home or workplace.

[0005] Techniques exist for enabling a user (e.g., a single user, multiple users in a household, coworkers, housemates, etc.) to utilize any one automated assistant client of a coordinated ecosystem of client devices to control any one of multiple smart devices associated with the automated assistant client. For example, a user can issue a voice command to the automated assistant client of a client device to "turn off the living room light" to cause a corresponding smart device (i.e., a light linked to the automated assistant client and labeled "living room" light) to turn off the light.

[0006] In controlling a smart device in response to a user input received at a client device, many existing techniques transmit data corresponding to the user input to a remote automated assistant server via the Internet. The remote automated assistant server processes the data to determine a smart device to be controlled based on the request, generates a control request for the smart device based on the request, and transmits the control request via the Internet to a server of a party associated with the smart device (e.g., the manufacturer of the smart device). In generating the control request, the remote automated assistant server can process the data (e.g., using a natural language understanding component) to generate a semantic representation of the data and then generate a control request based on the semantic representation. The server of the separate party receives the control request and then transmits the corresponding control command to the smart device via the Internet through a hub coexisting with the smart device (e.g., in the case of BLE, Z-Wave, ZigBee, etc.) or directly to the smart device via an IP connection (e.g., in the case of Wi-Fi and other smart devices that do not require a hub). Summary of the Invention [Problem to be solved by the invention]

[0007] However, such techniques present drawbacks such as high latency, excessive use of client and / or server resources, and / or excessive use of network resources. For example, high latency and / or excessive use of server resources may be the result of the remote assistant server having to process the received audio data and / or text to generate a semantic representation of the corresponding utterance and then generate a control request based on the generated semantic representation. Additionally or alternatively, high latency may be the result of sending requests from the remote assistant server to the separate party server, often exacerbated by the remote assistant server and the separate party server not being geographically close to each other. Also, for example, excessive use of network resources may be the result of sending high bandwidth audio data and / or text from the client device to the remote assistant server, and / or sending high bandwidth response text, voice, and / or graphics (e.g., confirming the requested changes to the smart device) from the remote assistant server to the client device. [Means for solving the problem]

[0008] Recently, efforts have been made to shift the performance of various automated assistant functions from the cloud automated assistant component to the automated assistant client device (i.e., the client device that provides the automated assistant interface). For example, recent efforts have been made to generate speech recognition models that can be utilized within the constraints of the hardware resources (e.g., memory resources, processor resources, etc.) of various client devices while still achieving good results. Similarly, efforts have been made to shift natural language understanding (NLU) and / or fulfillment (e.g., performance of actions in response to user input) on the device. Running such functions on the device provides various technical advantages, such as reduced latency (e.g., due to no client-server round trips), reduced network usage (e.g., due to no client-server round trips), and / or operability in low or no connectivity situations.

[0009] However, on-device NLU and / or on-device fulfillment may fail for many requests. As a non-limiting example, consider an utterance provided to an assistant client device located in a user's living room that includes "make these lights a little brighter and warmer." In generating an appropriate semantic representation of the utterance, it is necessary to resolve what is meant by "these lights," "a little brighter," and "warmer." Limited hardware resources on the assistant client device may preclude the use of local storage and / or machine learning models, rules, and / or device topology required to resolve the meaning. However, a remote assistant system with more robust resources than the assistant client device may be able to resolve the meaning. For example, the remote assistant system may leverage a remotely stored device topology to determine "these lights" for an utterance received (e.g., loudest and / or clearest heard) at the living room client device, referencing lights A, B, and C in the living room. Additionally, the remote assistant system may utilize remote machine learning models and / or rules to determine that "a little brighter" means "30% (or other percentage or value) brighter than the current luminance intensity levels of lights A, B, and C in the living room." Additionally, the remote assistant system may utilize remote machine learning models and / or rules to determine that "warmer" means "20% (or other percentage or value) lower color temperature than the current color temperature."

[0010] Therefore, due to hardware and / or other constraints of the assistant client device, the remote assistant system still needs to be utilized in resolving various user requests, such as a specific request to control a smart device. However, constantly providing the remote assistant system with unresolvable audio data and / or spoken text for complete processing may result in excessive resource usage and / or excessive latency in the remote assistant system in performing the corresponding action (e.g., control of the smart device).

[0011] Accordingly, implementations described herein relate to specific techniques for controlling smart devices that are low latency and / or provide computational (client and / or server) and / or network efficiency. The implementations relate to generating and / or utilizing cache entries for a cache stored locally on an assistant client device in controlling various smart devices (e.g., smart lighting, smart thermostats, smart plugs, smart appliances, smart routers, etc.). Each of the cache entries includes a mapping of text to one or more corresponding semantic representations.

[0012] The semantic representation of the cache entry may be a semantic representation of text and / or corresponding audio data previously generated by one or more remote automated assistant servers (also referred to herein as "remote assistant systems") in response to a previous transmission from the assistant client device to the remote assistant system. The cache entry may be stored in the cache by the assistant client device in response to a cache request sent by the remote assistant system to the client device in response to a previous transmission. The text of the cache entry that is mapped to the semantic representation of the cache entry may be represented in the cache entry as the text itself and / or as a representation thereof, such as its embeddings in Word2Vec and / or its variants that do not include stop words.

[0013] As described in detail herein, the semantic representation of the cache entry may include a cloud semantic representation and / or a locally interpretable semantic representation. The locally interpretable semantic representation is interpretable by the client device to generate a corresponding control command that can be sent by the client device to the local smart device and / or smart device controller via one or more local channels to cause a corresponding local control of the smart device. The control command can be generated to conform to the protocol suite of the local smart device and / or smart device controller. Meanwhile, the cloud semantic representation is interpretable by the remote assistant system to generate a corresponding control request to be sent by the client device to the remote assistant system and sent to the corresponding smart device control system to cause a corresponding control of the smart device.

[0014] A non-limiting example of generating and utilizing a cloud semantic representation is provided herein. Assume that an utterance of "dim the lights a little" is captured in audio data detected via one or more microphones of an assistant client device. The assistant client device can process the audio data using a local speech-to-text model to generate the recognized text of "dim the lights a little." The assistant client device can check its local cache to determine whether there is a cache entry that matches the recognized text. The assistant client device can determine that no matching cache entry exists, which may be the first instance of the utterance on the assistant client device or may be the result of a previously matching cache entry having recently been purged from the cache (e.g., to determine changes to device topology and / or other conditions). The assistant client device can optionally further determine that the recognized text is not locally interpretable utilizing an NLU component on the device. For example, the assistant client device may not have a robust local device topology that allows for the resolution of "light" to a particular light and / or may not have robust models and / or rules that allow for the resolution of "a little" to a particular degree.

[0015] The client device may further transmit the recognized text and / or audio data to the remote assistant system. Optionally, the transmission of the text and / or audio data to the remote assistant system is in response to a determination that there is no matching cache entry and / or that the recognized text is not locally interpretable.

[0016] In response to receiving the recognized text and / or audio data (from which the remote assistant system can generate the recognized text), the remote assistant system can process the recognized text to generate a cloud semantic representation of the recognized text. For example, the remote assistant system can utilize the account identifier received in the text transmission to identify the account identifier's corresponding remotely stored device topology. Based on the lights A and B being mapped in the device topology, the corresponding device topology can be utilized to resolve "light" to lights A and B as default lights for the assistant client device (e.g., through a user-instructed mapping or as a result of being assigned to the same room). Also, for example, the remote assistant system can utilize one or more remote models and / or rules to determine that "dim...a little" means "dim 20% (or other percentage or value) below the current brightness intensity level." A semantic representation can be generated based on these solutions, for example, [device=A and B, dimmer, relative intensity 20%], where "A and B" are unique identifiers of the corresponding lights, and "dimmer, relative intensity 20%" indicates that the lights are dimmed to an extent that they are reduced by 20% relative to their current intensity.

[0017] Once the cloud semantic representation is generated, the remote assistant system can generate a control request to activate the semantic representation. For example, the remote assistant system can identify the "current brightness intensity level" of "lights A and B" based on the most recently reported state of those lights and / or by requesting (and receiving) the current state of those lights from the smart device remote system that controls "lights A and B" (e.g., the smart device remote system controlled by the manufacturer of "lights A and B"). Additionally, the remote assistant system can generate a "target brightness intensity level" by multiplying the "current brightness intensity level" by 0.80 (based on a determination that "a little dimmer" means "20% dimmer than the current brightness intensity level"). A control request can then be generated that identifies "lights A and B" and includes a request to set the lights to the generated "target brightness intensity level". The control request can then be sent to the smart device remote system. In response to receiving the control request, the smart device remote system sends a corresponding control command to "Lights A and B" over the Internet through a hub coexisting with "Lights A and B" (e.g., in the case of BLE, Z-Wave, ZigBee, etc.) or directly to the smart device over an IP connection (e.g., in the case of Wi-Fi and other smart devices that do not require a hub). The control command is generated based on the control request, which may match or may differ from the control request. Optionally, the remote assistant system also sends text, audio data, and / or graphics to be rendered to the client device to confirm the change in the state of "Lights A and B". For example, the text "OK, dim lights A and B by 20%" can be sent to the client device, causing the client device to render the text graphically and / or render the corresponding synthetic voice using a local text-to-speech model.

[0018] Further, the remote assistant system can send to the client device a cache request that includes the cloud semantic representation and, optionally, the text (or a representation thereof), and a mapping between the text (or the representation) and the cloud semantic representation. In response to receiving the cache request, the client device can generate a cache entry in the local cache that defines the mapping of the text to the cloud semantic representation. For example, the cache entry can include a mapping of the text itself to the cloud semantic representation, and / or a mapping of a representation of the text (e.g., a Word2Vec representation or other representation) to the cloud semantic representation. In situations where the cache request itself does not include the text (or a representation thereof), the client device can include a mapping based on the cache request received in response to sending the text (or corresponding audio data). In some implementations, the remote assistant system can also include in the cache request text, audio data, and / or a graphic confirming the change in the state of “Lights A and B.”

[0019] Further assume that after generation of the cache entry at the client device, another instance of the utterance "dim the lights a little" is captured in the audio data detected via one or more microphones of the assistant client device. The assistant client device can process the audio data using a local speech-to-text model to generate the recognized text of "dim the lights a little." The assistant client device can check its local cache to determine that the generated cache entry includes text (or a representation thereof) that matches the recognized text. As used herein, "match" can include an exact match and / or a soft match. For example, using a soft match, the text "dim the lights a little" can be determined to match "dim the lights a little" based on an analysis of edit distance, the presence or absence of only stop words, and / or the use of other techniques. In response to determining a match, the assistant client device can transmit to the remote assistant system a cloud semantic representation that is mapped to the text in the cache entry. Optionally, the assistant client device can transmit the cloud semantic representation without transmitting the audio data and / or the recognized text to the remote assistant system.

[0020] In response to receiving the cloud semantic representation, the remote assistant system can then generate a control request to activate the cloud semantic representation. In the example of "dim the lights a little", note that the control request may vary depending on the "current brightness level" of "lights A and B" at the time of the user input that matches "dim the lights a little". For example, if the control request specifies the "target brightness level" of "lights A and B", their target brightness level depends on the "current brightness level". The control request can then be sent to the smart device remote system to cause the corresponding control of "lights A and B". Thus, by sending the semantic representation to the remote assistant system, the remote assistant system can bypass processing of the text to generate the semantic representation again and instead utilize the received cloud semantic representation.

[0021] These and other schemes reduce power, memory, and / or processor resource usage in the remote assistant system because less processing occurs in the remote assistant system. This reduction is particularly important given the large number (e.g., thousands, hundreds of thousands) of smart device control requests that may be received and processed by the remote assistant system daily. Furthermore, these and other schemes allow control requests to be generated and sent more quickly, thereby allowing smart device states to be changed more quickly. For example, bypassing the generation of semantic representations from text can save at least 200 ms, 500 ms, 700 ms, or other periods in generating control requests, which may be particularly important in controlling various smart devices (e.g., turning on lights in a dark room).

[0022] In implementations in which the cache entry also includes text, audio data, and / or graphics confirming the change in state of "Lights A and B," the assistant client device may also render such content in response to determining that the cache entry matches the input. The content may be rendered immediately or, optionally, after a delay that may be optionally specified in the cache entry (e.g., a delay based on the latency between receiving the input and enabling the corresponding state change).

[0023] It should be noted that in the above example, the smart device control request is a state change request related to a state. That is, it requests a change in the state of the smart device in a manner that depends on the current state of the smart device. More specifically, it requests a change to the brightness state of lights A and B in a manner that is related to the current brightness state of lights A and B. In some implementations, the remote assistant system may determine to provide a cache request including the semantic representation in response to determining that the semantic representation is a relative state change representation in lieu of a control request. Such a determination may be based on the semantic representation that references the current state of the smart device.

[0024] In some of those implementations, the remote assistant system can provide a cache request including a control request in place of the semantic representation in response to a determination that the semantic representation is not a relative state change representation. For example, assume an utterance of "set the lights to 500 lumens" instead. In such an example, a cloud semantic representation can be generated that identifies "lights A and B" and identifies a "target brightness intensity level" of "500 lumens" for both lights. Additionally, a corresponding control request can also be generated. Because the cloud semantic representation is not relative, the remote assistant system can optionally send a cache request including the control request. In response, the client device can generate a cache entry that maps the text "set the lights to 500 lumens" (or a representation thereof) to the control request. Thereafter, in response to a determination that the received input matches the text of the cache entry, the client device can send the control request to the remote assistant system. The remote assistant system then sends the control request to the smart device remote system, thereby enabling the remote assistant system to directly utilize the control request, thereby bypassing both the generation of the semantic representation and the generation of the control request. This in turn may have a positive impact on resource utilization and / or latency of the remote assistant system.

[0025] It should also be noted that among assistant client devices in a user / account ecosystem, a particular cache entry can include a mapping of the same text to different semantic representations (or control requests). For example, assume that "dim the lights a bit" is received at different assistant client devices, and "lights C and D" are mapped in the device topology as preferred lights for the different assistant client devices. In such an example, a semantic representation identifying "lights C and D" is generated instead of "lights A and B." Thus, a cache entry can be generated for the different client devices that includes a mapping of "dim the lights a bit" to a semantic representation identifying "lights C and D."

[0026] It should also be noted that if speech is detected at multiple client devices, one or more device arbitration techniques may be utilized to determine that it was received at only one of the client devices. Such techniques may take into account various features in determining which client device should be the receiving device, based on determining that the client device is closest to the user and / or is otherwise intended to be interacted with by the user (despite not being closest). Such features may include the volume of the audio data spoken at the client device, the time of detection of the audio data at the client device, the recency of the interaction with the client device, the proximity of the user to the client device, and / or other features.

[0027] A non-limiting example of generating and utilizing a locally interpretable semantic representation is now provided. Assume again that the utterance "Turn the lights down a bit" is captured in audio data detected via one or more microphones of the assistant client device. The assistant client device can process the audio data using a local speech-to-text model to generate the recognized text "Turn the lights down a bit." The assistant client device can check a local cache to determine whether there is a cache entry that matches the recognized text. The assistant client device can determine that no matching cache entry exists, which may be a result of this being the first instance of the utterance at the assistant client device or a previously matching cache entry having recently been purged from the cache. The assistant client device can optionally further determine that the recognized text is not locally interpretable utilizing an NLU component on the device.

[0028] The client device may further transmit the recognized text and / or audio data to the remote assistant system. Optionally, the transmission of the text and / or audio data to the remote assistant system is in response to a determination that there is no matching cache entry and / or that the recognized text is not locally interpretable.

[0029] In response to receiving the recognized text and / or audio data (from which the remote assistant system can generate the recognized text), the remote assistant system can process the recognized text to generate a locally interpretable semantic representation of the recognized text. For example, the remote assistant system can utilize the account identifier received in the text transmission to identify the account identifier's corresponding remotely stored device topology. Based on the light C mapped in the device topology, the corresponding device topology can be utilized to resolve "light" to "light C" as the default light for the assistant client device.

[0030] Also, for example, the remote assistant system may utilize one or more remote models and / or rules to determine that "dim...a little" means "dim 20% (or other percentage or value) below the current luminance intensity level." The remote assistant system may generate a locally interpretable semantic representation by including an identifier for "Light C" and a state change representation that is a relative state change representation that indicates how to locally resolve the target intensity level for "dim a little." For example, the identifier for "Light C" may be a local address for "Light C." Also, for example, the state change representation may be "set intensity=(current intensity*0.8)," which indicates that the intensity of "Light C" should be set to a value determined by identifying 80% of the current intensity value of "Light C" (e.g., the most recently reported intensity, or the locally requested and locally received intensity value). Thus, in some implementations, the locally interpretable semantic representation may define the state change representation with more granularity than the cloud-based counterpart to enable local resolution of relative values ​​that may be utilized to generate a local control command (e.g., utilizing a local adapter described herein). The state change representation may optionally further include an indication of an adapter (e.g., an adapter specific to the manufacturer of the smart device) to be utilized in generating the corresponding local control command, an indication of a channel (e.g., Bluetooth or Wi-Fi) to be utilized to transmit the local control command, and / or an indication of a protocol suite to be utilized in generating a control command based on the locally interpretable semantic representation. The locally interpretable semantic representation may be, for example, [device=C; set strength=(current strength*0.8)].

[0031] In some implementations, the system can determine to generate a locally interpretable semantic representation in response to determining that the client device (or an additional client device locally connected to the client device) can locally control a smart device referenced in the recognized text, determining that the smart device referenced in the recognized text can be locally controlled, and / or determining that a state change requested in the recognized text can be enabled for local control (e.g., if some states can be changed through local control, but other states can only be changed through cloud control).

[0032] Once the locally interpretable semantic representation is generated, the remote assistant system can optionally transmit the locally interpretable semantic representation to the client device for immediate implementation. In other words, the remote assistant system can transmit the locally interpretable semantic representation to cause the client device to interpret the locally interpretable semantic representation in response to an utterance, generate a corresponding local control command, and locally transmit the local control command to effect a state change to “Light C”. For immediate implementation, the transmission of the locally interpretable semantic representation to the client device can be the transmission of the cache request itself (described below), or a separate transmission. Alternatively, the remote assistant system can instead generate and transmit a corresponding cloud control request to the smart device control system of “Light C” to cause the smart device control system to generate and transmit a corresponding control command to “Light C” to effect a state change.

[0033] Regardless of the manner in which the state change is implemented in response to an immediate request, the remote assistant system can send a cache request to the client device that includes the locally interpretable semantic representation and, optionally, the text (or a representation thereof) and a mapping between the text (or a representation) and the locally interpretable semantic representation. In response to receiving the cache request, the client device can generate a cache entry in the local cache that defines the mapping of the text to the locally interpretable semantic representation. For example, the cache entry can include a mapping of the text itself to the locally interpretable semantic representation and / or a mapping of the representation of the text to the locally interpretable semantic representation. In situations in which the cache request itself does not include the text (or a representation thereof), the client device can include a mapping based on the cache request received in response to sending the text (or corresponding audio data). In some implementations, the remote assistant system can also include in the cache request text, audio data, and / or a graphic confirming the change in the state of “light C.”

[0034] Further assume that after generation of the cache entry at the client device, another instance of the utterance "Turn the lights down a bit" is captured in the audio data detected via one or more microphones of the assistant client device. The assistant client device can process the audio data using a local speech-to-text model to generate the recognized text of "Turn the lights down a bit." The assistant client device can check its local cache to determine that the generated cache entry includes text (or a representation thereof) that matches the recognized text. Exact matches and / or soft matches can be utilized. In response, the assistant client device can process the locally interpretable semantic representation of the cache entry to generate a corresponding local control request for sending locally to "Light C." For example, based on the locally interpretable semantic representation including "Set Intensity = (Current Intensity * 0.8)," the assistant client device can locally identify the "Current Intensity" of "Light C" and then determine the value of "Set Intensity" by multiplying the current intensity by 0.8. Thus, the "set strength" to a determined value may be a particular state change that is generated based on the relative state change of the locally interpretable semantic representation.

[0035] Additionally, the assistant client device may generate a control command based on the value of the “set intensity” and, optionally, utilizing one or more local adapters. The control command may be generated to conform to a protocol suite conforming to “Lighting C”, such as BLE, Z-Wave, Zigbee, Wi-Fi, or other protocol suite. The protocol suite utilized in generating the control command may be selected based on an identifier of the smart device included in the locally interpretable semantic representation and / or based on an indication of the adapter that may be included in the locally interpretable semantic representation.

[0036] Further, the generated control command can be sent locally to “Light C” (or a corresponding local device that controls “Light C”). The control command can be sent to “Light C” (or a corresponding local device) based on a locally interpretable semantic representation that identifies “Light C” (e.g., identifies an address of “Light C”) and can be sent via a local channel that is optionally identified via the locally interpretable semantic representation. The local channel can be, for example, a Wi-Fi wireless channel, a BLUETOOTH wireless channel, or other channel. Thus, a state change can be effected in “Light C” without the need to transmit any data again to the remote assistant system and / or without the remote assistant system having to generate the semantic representation again. This can conserve network resources by eliminating the need to transmit audio data and / or recognized text to the remote assistant system, and can conserve processing resources in the remote assistant system by eliminating the need to generate the semantic representation again in the remote system. Furthermore, by bypassing transmission to and processing in the remote assistant system, latency in the implementation of the requested state change in the smart device can be significantly reduced.

[0037] As described above, various implementations utilize an adapter to locally interpret the locally interpretable semantic representation or the specific state changes generated thereon into corresponding control commands. In those implementations, the client device can include one or more adapters, each of which can be tailored to a specific smart device, a specific smart device manufacturer, and / or a specific protocol suite. When executed by the assistant client device, the adapter can process the locally interpretable semantic representation (and / or the specific state changes generated based on the locally interpretable semantic representation) to generate specific control commands, each tailored to be directly interpretable by the corresponding smart device (or other components connected to the smart device) to effect the state changes in the corresponding smart device, when sent locally to at least one corresponding smart device (or a hub / bridge, or other components connected to the smart device). For example, the adapter can be implemented by JavaScript (or other interpreted programming language) and can convert generic smart device control commands generated based on the locally interpretable semantic representation into specific control commands that conform to the protocol suite of the smart device. The adapter can optionally run in a container in the automated assistant client. In an implementation in which multiple adapters are available at the assistant client device, a particular adapter may be selected for a locally interpretable semantic representation based on an identifier of the smart device included in the semantic representation and / or based on an indication of the adapter that may be included in the semantic representation.

[0038] In implementations where the cache entry also includes text, audio data, and / or graphics confirming the change in state of "Lights A and B," the assistant client device can also render such content in response to determining that the cache entry matches the input. The content can be rendered immediately or, optionally, after a delay.

[0039] It should be noted that in the above example, the smart device control request is a state change request related to a state. That is, the smart device control request requests a change in the state of the smart device in a manner dependent on the current state of the smart device. More specifically, the smart device control request requests a change to the brightness state of the light C in a manner related to the current brightness state of the light C. In some implementations, the remote assistant system may determine to provide a cached request including a locally interpretable semantic representation in place of a local control command in response to determining that the semantic representation is a relative state change representation. Such a determination may be determined based on the semantic representation that references the current state of the smart device.

[0040] In some of those implementations, the remote assistant system can provide a cache request including a local control command in place of the semantic representation in response to a determination that the semantic representation is not a relative state change representation. For example, assume an utterance of "set the light to 500 lumens" instead. In such an example, a locally interpretable semantic representation can be generated that identifies "light C" and identifies a "target brightness intensity level" of "500 lumens" for that light. Additionally, a corresponding control request can also be generated. Because the semantic representation is not relative, the remote assistant system can optionally send a cache request including a local control command generated based on the locally interpretable semantic representation. In response, the client device can generate a cache entry that maps the text "set the light to 500 lumens" (or a representation thereof) to the control command. Thereafter, in response to a determination that the received input matches the text of the cache entry, the client device can locally send the control command of the cache entry, thereby bypassing generation of the control command based on the locally interpretable semantic representation.

[0041] It should also be noted that, similar to cloud semantic representations, within an assistant client device of a user / account ecosystem, a particular cache entry may contain mappings of the same text to different locally interpretable semantic representations (or local control commands).

[0042] Another example of a cloud semantic representation and a locally interpretable semantic representation is provided above. However, in some implementations, a single cache entry can include a mapping of text to both a locally interpretable semantic representation and a cloud semantic representation. For example, lights "A" and "B" can both be identified for a request, and light "A" may be locally controllable, while light "B" may only be controllable via cloud control. As a result, the remote assistant system can generate and provide a cache request that maps the text of the request to both (a) a locally interpretable semantic representation that can be processed locally to effect a state change in "light A," and (b) a cloud semantic representation that is sent to the remote assistant system to effect a state change in "light B."

[0043] In various implementations, the remote assistant system and / or the client device can determine a change in the device topology of the account and / or the occurrence of other conditions, which results in one or more cache entries of the cache of the client device of the ecosystem being flushed (e.g., deleted entirely or flagged as stale). A change in the device topology can include, for example, the addition or removal of a smart device in the device topology, the renaming of a smart device and / or assistant client device in the device topology, the assignment of a device to a new room and / or group in the device topology, and / or other changes to the device topology. Such changes can affect some or all of the semantic representations of the cache, and as a result, in response to detecting the change, at least the affected cache entries (and, optionally, all cache entries) can be flushed from the cache. As an example, assume that additional lights are added to the device topology and assigned to a room with a “kitchen” label. When that addition occurs, any semantic cache entries associated with those lights that were previously assigned to “kitchen lights” become stale because they do not reflect the newly added additional lights. Thus, in response to detecting the change, at least those cache entries associated with “kitchen lights” can be flushed from the cache. For example, the remote assistant system can determine such a change and send a cache clearing request to all assistant client devices in the ecosystem to clear the local caches of all assistant client devices. A new cache entry is then generated at the assistant client device when speech, typing, or other input is received. For example, if a cache entry for the text "dim the kitchen lights" is cleared, a new cache entry is created when the user subsequently provides the utterance "dim the kitchen lights."

[0044] In some implementations, the assistant client device can at least selectively process audio data utilizing an on-device speech-to-text model without detecting an explicit automated assistant invocation at the assistant client device. An explicit invocation cue is a cue that, when detected alone, always activates at least on-device speech recognition. Some non-limiting examples of explicit invocation cues include detecting a spoken hot word with at least a threshold confidence, an explicit assistant interface element actuation (e.g., a hardware button or a graphic button on a touch screen display), a "phone squeeze" with at least a threshold intensity (e.g., detected by a sensor in the bezel of a mobile phone), and / or other explicit invocation cues.

[0045] As an example, the assistant client device can process audio data without detecting an explicit automated assistant invocation when a human presence is detected near the device (e.g., based on a passive infrared sensor and / or other sensor), in response to detection of any voice activity (e.g., using a voice activity detector), in response to detection of at least a threshold level of noise near the device during a particular day and / or time, and / or in response to other conditions. In an implementation in which the assistant client device processes audio data, utilizing an on-device speech-to-text model, without detecting an explicit automated assistant invocation, the assistant client device can determine whether any recognized text matches any "hot phrases" so indicated locally on the assistant client device. Each "hot phrase" comprises a sequence of one or more words. If it is determined that the recognized text from on-device voice recognition of the audio data does not match a hot phrase, the assistant client device can discard such recognized text and audio data. If it is determined that the recognized text from on-device voice recognition of the audio data matches a hot phrase, the assistant client device can automatically initiate one or more corresponding actions based on the text. Users can control which phrases are shown as hot phrases, and optionally, explicit user approval may be required before a phrase is shown as a hot phrase.

[0046] In some implementations, a hot phrase may be a phrase that matches the text of a corresponding cache entry described herein. In those implementations, an action automatically initiated by an assistant client device based on a hot phrase may include a transmission of a corresponding cloud semantic representation or a local interpretation of a corresponding locally interpretable semantic representation. In some of those implementations, the text of a cache entry may be designated as a hot phrase for the assistant client device based on a determination that one or more criteria are met at the assistant client device. The criteria may include, for example, that the text and / or the matching text is determined to be present at least a threshold number of times (overall or within a recent temporal period) and / or at least a threshold frequency (overall or within a recent temporal period) in user input (e.g., typed and / or spoken) received at the assistant client device. In some versions of those implementations, the text of the cache entry may be automatically designated as a hot phrase, optionally with confirmation of such designation provided via a user interface input (e.g., "By the way, I designated X as a hot phrase. This can be easily changed in settings"). In some other versions, text may be designated as a hot phrase only after a confirmatory user input in response to a prompt, such as a confirmatory input of "yes" in response to the prompt, "If I request that you turn on the lights often, would you like to designate that as a hot phrase so that I don't have to say "OK Assistant" first?" In still other implementations, the text of any local cache entry in the assistant client device's cache can be automatically designated as a hot phrase for the assistant client device.Implementations that designate text of a cache entry as a hot phrase can shorten the duration of user interaction with the assistant client device by eliminating the need for the user to first provide an explicit invocation, such as an invocation hotword (e.g., "OK Assistant"). Through shortening the duration of interaction with the assistant client device, human interaction with the assistant client device is improved, allowing more smart device state changes to take effect in a shorter period of time.

[0047] The above description is provided as a summary of only some implementations of the present disclosure. Further descriptions of those and other implementations are described in more detail herein.

[0048] Further, some implementations include one or more processors of one or more computing devices, the one or more processors operable to execute instructions stored in associated memory, the instructions configured to cause the execution of any of the methods described herein. Some implementations include a client device having one or more processors that execute locally stored instructions and interface with locally stored data to perform one or more of the methods described herein. Some implementations also include one or more computer-readable storage media (transient or non-transient) that store computer instructions executable by the one or more processors to perform any of the methods described herein.

[0049] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as part of the subject matter disclosed herein, for example, all combinations of subject matter recited in the claims appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. [Brief description of the drawings]

[0050] [Figure 1] FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented. [Diagram 2] FIG. 1 illustrates an example environment including multiple smart devices from multiple different parties and including multiple assistant client devices. [Figure 3A] FIG. 4 illustrates an example of how an utterance received at the assistant client device of FIG. 2 can be handled if the assistant client device does not have a cache entry that matches the text of the utterance. [Figure 3B] A diagram showing an example of how the same utterance of FIG. 3A received at the same client device of FIG. 3A can be processed differently after a cache entry having a cloud semantic representation is stored in the assistant client device of FIG. 3A. [Figure 4A] FIG. 3B illustrates an example of how the same utterance of FIG. 3A and FIG. 3B received at an alternative assistant client device of FIG. 2 can be processed if the alternative assistant client device does not have a cache entry that matches the text of the utterance. [Figure 4B] FIG. 4B illustrates an example of how the same utterance of FIG. 4A received at an alternative client device of FIG. 4A may be processed differently after a cache entry having a cloud semantic representation is stored in the alternative client device of FIG. 4A. [Figure 5A] FIG. 3B shows an example of how another utterance received at the same assistant client device of FIG. 3A and FIG. 3B can be handled if the assistant client device does not have a cache entry that matches the text of the utterance. [Figure 5B]FIG. 5B illustrates an example of how the same utterance of FIG. 5A received at the client device of FIG. 5A may be processed after a cache entry having a locally interpretable semantic representation is stored in the client device of FIG. 5A. [Figure 6] 1 is a flowchart illustrating an example method that may be implemented by an assistant client device according to various implementations disclosed herein. [Figure 7] 1 is a flowchart illustrating another example method that may be implemented by an assistant client device according to various implementations disclosed herein. [Figure 8] 1 is a flowchart illustrating an example method that may be implemented by a remote assistant system according to various implementations disclosed herein. [Figure 9] FIG. 1 illustrates an example architecture of a computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0051] There has been a proliferation of smart network-connected devices (also referred to herein as smart devices or Internet of Things (IoT) devices), such as smart home alarms, smart door locks, smart cameras, smart lighting, smart thermostats, smart scales, smart beds, smart irrigation systems, smart garage door openers, smart plugs, smart appliances, smart baby monitors, smart televisions (TVs), smart fire alarms, smart humidity detectors, smart routers, etc. Often, multiple smart devices are located within a structure such as a home, or within multiple related structures, such as a user's primary residence and the user's secondary residences and / or workplaces.

[0052] Additionally, there is a proliferation of assistant client devices, each of which can include an assistant client that can optionally interact with one or more remote automated assistant components to form a logical instance of an automated assistant. Assistant client devices can be dedicated solely to assistant functions (e.g., a standalone speaker and / or standalone audio / visual device that includes only an interface associated with the assistant client and is dedicated solely to assistant functions) or can perform assistant functions in addition to other functions (e.g., a mobile phone or tablet that includes an assistant client as one of multiple applications). Additionally, some smart devices may be assistant client devices. For example, some smart devices can include an assistant client and at least a speaker and / or microphone that (at least in part) serves as a user interface output and / or input device for the assistant interface of the assistant client.

[0053] Various techniques have been proposed for associating smart devices with corresponding logical instances of an automated assistant (and optionally with individual assistant client devices). For example, a user, a group of users, an assistant client device, and / or a group of assistant client devices (e.g., all within a structure) may be linked (e.g., in one or more databases) with multiple different smart devices to enable interaction (e.g., control) with the smart devices via the automated assistant. For example, each of multiple assistant client devices in a home may be linked to each of multiple different smart devices in the home to enable any user (or a restricted group of users) to interface with any of the assistant client devices to interact with any of the multiple different smart devices.

[0054] Such linking of smart devices and assistant client devices is referred to herein as a device topology and can be represented by various data structures. The device topology can be created by a user and / or automatically, and can define various assistant client devices, various smart devices, their respective identifiers, and / or their respective attributes. For example, the device identifier can specify the room (and / or other area) in the structure (e.g., living room, kitchen) in which the device is located, and / or the device nickname and / or alias (e.g., sofa lamp, front door lock, bedroom speaker, kitchen assistant, etc.). In this manner, the device identifier can be the name, alias, and / or location of each device that a user is likely to associate with each device.

[0055] The device topology representation may further specify one or more device attributes associated with the respective device. The device attributes of the assistant client device may include, for example, one or more input and / or output modalities supported by the assistant client device and / or a preferred smart device to be controlled by the assistant client device (e.g., ambiguous smart TV commands received at assistant client device 1 should be assumed to be directed to smart TV 1, but they may be assumed to be directed to smart TV 2 at assistant client device 2). For example, the device attributes of a standalone speaker-only assistant client device may indicate that it can provide an audible output but cannot provide a visual output. Also, for example, the device attributes of the same standalone speaker-only assistant client device may indicate that a lighting control request that does not specify a particular light should be interpreted as relating to a light assigned to the same room as the assistant client device. The device attributes of the smart device can, for example, identify one or more states of the controllable smart device, identify a party (e.g., 3P) that manufactures, distributes, and / or creates firmware for the smart device, and / or identify a unique identifier for the smart device, such as an address of the smart device and / or a fixed identifier provided by 1P or 3P. According to various implementations disclosed herein, the device topology representation can optionally further specify which smart devices can be locally controlled by which assistant client device, local addresses of the locally controllable smart devices (or local addresses of hubs that can directly locally control those smart devices), local signal strength and / or other priority indicators between the assistant client device and the smart device.Further, according to various implementations disclosed herein, the device topology representation (or variations thereof) can be stored remotely in the remote assistant system and / or locally in each of a plurality of assistant client devices for use in controlling the smart device locally. Optionally, each locally stored device topology may be less robust (e.g., define fewer devices, attributes, and identifiers) than the remotely stored device topology, for example, due to hardware constraints of the assistant client device.

[0056] Referring now to FIG. 1, an example environment in which the techniques disclosed herein may be implemented is shown. The example environment includes a plurality of assistant client devices 110. 1-N (also referred to herein simply as a “client device”), a cloud automated assistant component 120, and a smart device system 140 A-N , Smart Devices 145 A-N , and the client device 110 1-N The client device 110 of FIG. 1-N and smart devices 145 1-N represents client devices and smart devices that are at least selectively associated with each other (e.g., via device topology 152). For example, smart device 145 1-N may all be in the home (e.g., inside and / or outside the home) and may be connected to the client device 110 1-N may be in the same house at least occasionally, and smart devices 145 1-N and client device 110 1-N may be linked together using one or more techniques as described herein. Through such links, the client devices 110 1-N may be implemented by a smart device 145 in accordance with the implementation described herein. 1-N can be utilized to control

[0057] Client Device 110 1-N One or more (e.g., all) of the client devices 110 may run a respective instance of the automated assistant client. However, in some implementations, 1-N One or more of the smart devices 110 may optionally include an engine and hardware components for still controlling one or more smart devices without an instance of the automated assistant client. An instance of the automated assistant client may be associated with a corresponding client device 110. 1-N The operating system of the client device 110 may be a separate application (e.g., installed “on top of” the operating system) or may be a separate application from the operating system of the client device 110. 1-N Each instance of the automated assistant client may be implemented directly by the operating system of the client device 110. 1-N The client device 110 may optionally interact with the cloud automated assistant component 120 in response to various requests provided by a user via any one of the I / O components 111. 1-N Other engines can optionally interact with the cloud automated assistant component 120.

[0058] Client Device 110 1-N One or more (e.g., all) of the STT models 152 may be used to generate corresponding recognized text for an utterance. 1-N 112, each of which processes the audio data capturing the speech. 1-N Each instance of audio data may be transmitted to the I / O component 111 of a respective client device. 1-N In some implementations, the STT engine 112 1-Ncorresponds to the client device 110 1-N In some implementations, the STT engine 112 processes only audio data following an explicit automated assistant invocation detected in 1-N may at least selectively process audio data even in the absence of an explicit automated assistant invocation, but discard (without further processing) any audio data and / or recognized text that is determined not to contain any "hot phrases" as described herein. In situations where a "hot phrase" is detected, one or more responsive actions may be taken based on the "hot phrase." For example, a "hot phrase" may be stored in a local cache 158 that corresponds to the "hot phrase." 1-N If it matches the text of a cache entry of, the semantic representation of the cache entry may be further processed as described herein.

[0059] Client Device 110 1-N One or more (e.g., all) of the may also use a respective locally stored NLU model 153 to process the text to attempt to locally generate a local semantic representation of the text. 1-N Each natural language understanding (NLU) engine uses 1-N The text may be recognized text from the STT engine 112, or may be typed text (e.g., input via a virtual keyboard on a touch screen), or may be text associated with a selected virtual and / or hardware button (e.g., text that maps to the selection of a particular virtual button displayed on a touch screen). 1-N may be able to generate valid semantic representations of various text phrases. However, as described herein, the NLU engine 113 1-Nmay also be unable to generate valid semantic representations of various other text phrases, such as various text phrases related to control of a smart device (e.g., relative control of a smart device).

[0060] Client Device 110 1-N One or more (e.g., all) of the fulfillment engines 114 may also 1-N Each fulfillment engine 114 may include 1-N Each cloud module 115 1-N , each local module 116 1-N , and / or their respective text-to-speech (TTS) engines 117 1-N may include.

[0061] Cloud Module 115 1-N can fulfill various requests through interactions with the cloud automated assistant component 120. For example, the cloud module 115 1 The STT engine 112 receives the text of the input (for example, the recognized text). 1-N ) in the local cache 158 1 In response to determining that the text matches the text in the cache entry, the cloud semantic representation mapped to the text in the cache entry can be sent to the cloud automated assistant component 120 as described herein. 1 Optionally, you can also use the NLU engine 1 The cloud semantic representation generated by the cloud automated assistant component 120 can then be sent to the cloud automated assistant component 120. The cloud fulfillment engine 127 of the cloud automated assistant component 120 can then utilize the received cloud semantic representation in fulfilling the received input. For example, in the case of a smart device control input, the cloud fulfillment engine 127 can generate a corresponding control request to send to the corresponding smart device system 140A-N The cloud semantic representation can then be utilized to generate and transmit corresponding control commands to the corresponding smart devices 145. A-N can be sent to.

[0062] Cloud Module 115 1-N Also, each NLU engine 113 1-N and / or their respective local caches 158 1-N For a received input that does not match any cache entry in the cloud-based automated assistant component 120, data representing the received input may be sent to the cloud-based automated assistant component 120. For example, if the received input is a speech captured in the audio data, the audio data and / or its locally recognized text (e.g., stored in the respective STT engine 112) may be stored in the cloud-based automated assistant component 120. 1-N For at least some of those transmissions, the cloud-based component 120 may transmit cache requests to the respective cache engines 118 in response to the input to enable future occurrences of the input to be satisfied more efficiently and / or with reduced latency. 1-N Further, the cloud-based component 120 may additionally or alternatively respond to the received inputs to perform corresponding cloud fulfillment and / or provide locally interpretable semantic representations and / or local commands to the respective client devices 110 in performing local fulfillment. 1-N For use by each client 110 1-N can be provided to.

[0063] Local Module 116 1-N can fulfill various requests locally and without interaction with the cloud automated assistant component 120. For example, the local module 116 1The text of the received input (for example, the STT engine 112 1-N The recognized text from the local cache 158 1 In response to determining that the text matches the text in the cache entry of the local module 116, the locally interpretable semantic representation that is mapped to the text in the cache entry may be utilized in fulfilling the request locally. 1 Radio 119 1 via each of one or more corresponding smart devices 145 A-N Optionally, as described herein, the locally interpretable semantic representation may be utilized in generating a local control command for locally transmitting to the smart device. 1-N 156 , which is a block diagram of a control system for a particular adapter 156 in generating a local control command. 1-N The local module 116 can be used. 1 Optionally, you can also use the NLU engine 1 Any locally interpretable semantic representation produced by

[0064] Text-to-Speech (TTS) Engine 117 1-N Optionally, the TTS engine 117 may be utilized to generate synthetic speech for rendering on demand. 1 The text of the received input (for example, the STT engine 112 1-N The recognized text from the local cache 158 1In response to a determination that the received input matches the text of a cache entry, the response text mapped to the text in the cache entry can be utilized to generate corresponding synthetic speech, such that the synthetic speech is rendered in response to the received input. The synthetic speech can be rendered in addition to other actions performed based on the semantic representation (which can be interpreted in the cloud and / or locally), which is also mapped to the text in the cache entry. Further, the response text can additionally or alternatively be visually rendered and / or alternative content can additionally or alternatively be rendered by the fulfillment engine in response to being mapped to the text in the cache entry.

[0065] Client Device 110 1-N One or more (e.g., all) of the cache engines 118 may also optionally be configured to cache the cache engines 118, respectively, as described above. 1-N and their respective local caches 158 1-N The cache engine 118 may include 1-N receive respective cache requests from the cache generation engine 125 of the cloud automated assistant component 120 and store respective cache entries in respective local caches 158 1-N The local cache 158 1-N For example, each client device 110 1-N As described herein, in some implementations, received cache requests from the cache generation engine 125 may be stored in the RAM and / or ROM of the respective local cache 158. 1-N In other implementations, the cache request may not have text to include in the cache entry, but may map to a semantic expression in the cache entry. In such implementations, each cache engine 118 1-Ncan determine the text based on the text most recently provided to the cloud automated assistant component 120 (or the recognized text of the audio data most recently provided to the cloud automated assistant component 120).

[0066] The cloud automated assistant component 120 is 105 in FIG. 1 1 through one or more wide area networks (e.g., the Internet), generally denoted by 1-N The cloud automated assistant component 120 may be implemented in one or more computing systems (collectively referred to as a "cloud" or "remote" assistant system) communicatively coupled to the client device 110. For example, the cloud automated assistant component 120 may be implemented by one or more clusters of high performance servers. 1-N Wide Area Network 105 1 It should be noted that one or more local area networks may be utilized in accessing and / or communicating locally with each other. Such local area networks may include client devices 110, 1-N This may include a Wi-Fi network and / or a mesh network between the

[0067] The cloud automated assistant component 120 also communicates with the smart device system 140 via one or more wide area networks. A-N The cloud automated assistant component 120 and the smart device system 140 can be communicatively coupled to each other as shown in FIG. 1 at 105. 2 In addition, each smart device system 140 is generally represented by 110 in FIG. 4 One or more smart devices 145 via one or more wide area networks, generally represented by A-N The smart device 145 may be communicatively coupled to a corresponding group ofA-N Wide Area Network 105 3 It should be noted that one or more local area networks may be utilized in accessing .

[0068] Smart Device System 140 A-N Each of the may be either a first party (1P—i.e., manufactured and / or distributed by the same party that controls the automated assistant) or a third party (3P—i.e., manufactured and / or distributed by a different party) system, and each may include one or more corresponding smart devices 145 A-N For example, the first smart device system 140 may be communicatively coupled to A-N is controlled by the first 3P and the first smart device 145 A1 and the second smart device system 140 is controlled by the second 3P and is communicatively coupled to the second smart device 145. B1 and 3rd smart device 145 B2 and the like.

[0069] Smart Device System 140 A-N Each of these smart devices 145 A-N Controlling firmware updates for each of those smart devices145 A-N to their respective smart devices. A-N The wide area network 105 3 via device 145 A-N For example, a given one of the smart device systems 140 may communicate with a smart device 145 to control the smart device in response to user input received via a mobile application of the smart device system that enables control of the smart device. A-N can communicate with one of

[0070] Also, for example, a given one of the smart device systems 140 may control a smart device 145 in response to a request from the cloud automated assistant component 120. A-N For example, according to some techniques, a user may communicate with one of the client devices 110. 1 One or more I / O components of 111 1 Through I / O component 111 1 A request to control a smart device, such as a voice input of "turn off the couch lights" provided via a microphone in the smart device 102, can be provided. The request (e.g., audio data capturing the voice input, or locally generated text) can be transmitted over the wide area network 105. 1 via the client device 110 1 The cloud automated assistant component 120 can process the request to determine which smart device is to be controlled based on the request and can then transmit the request to the cloud automated assistant component 120 via the wide area network 105. 2 A control request is sent to the smart device system 140 via the A-N and then transmits the signal to a wide area network 105 3 , and sends a corresponding command to the smart device via the smart device 100. However, as described herein, such techniques present drawbacks such as high latency and / or excessive consumption of network resources.

[0071] In some implementations, multiple client computing devices 110 1-N and smart devices 145 A-N may be associated with one another in various ways to facilitate performance of the techniques described herein. For example, in some implementations, multiple client devices 110 1-N and smart devices 145 A-Nmay be associated with one another because they are communicatively coupled via one or more LANs and / or via one or more peer-to-peer networks. 1-N This may be the case when multiple client devices 110 are deployed across a particular area or environment, such as a home, building, etc. Additionally or alternatively, in some implementations, multiple client devices 110 1-N and smart devices 145 A-N , which are client devices 110 that are at least selectively accessible by one or more users (e.g., individuals, family members, employees of an organization, other predefined groups, etc.). 1-N and smart devices 145 A-N In some of these implementations, the client devices 110 may be associated with each other by being members of a collaborative ecosystem of 1-N and smart devices 145 1-N The ecosystems can be associated with each other manually and / or automatically in the device topology 152.

[0072] Client Device 110 1-N An instance of an automated assistant client of the cloud may interact with one or more cloud automated assistant components 120 to form what appears from a user's perspective to be a logical instance of an automated assistant with which the user may conduct a human-computer interaction. For example, a user may use a client device 110 1 and Automated Assistant Client 117 1 or the client device 110 N and Automated Assistant Client 117 N The same logical instance of the automated assistant can be accessed using the automated assistant client 117. 1 and 117 NThe specific instances of may vary (e.g., provide different smart device controls for the same command) and / or may be different I / O components 111 1 and 111 N and / or different I / O components 111 1 and 111 N , and can accept different user interface inputs via I / O components 111 1 can include a touch screen, but the I / O components 111 N 110) but the user can still effectively utilize the same logical instance of the automated assistant. For brevity and clarity, the term "automated assistant" as used herein refers to an automated assistant client running on a client device 110 and, optionally, to one or more cloud automated assistant components 120 (which may be shared between multiple automated assistant clients). Two client devices 110 in a coordinated ecosystem 1 and 110 N 1, it is understood that many additional client devices can be included in the ecosystem. Additionally, separate, coordinated ecosystems of client devices can also be provided, each associated with a different user (e.g., account) and / or environment, and such separate, coordinated ecosystems can also interact with the cloud automated assistant component 120 (provided that there is interaction tailored to those separate ecosystem accounts).

[0073] Client Device 110 1-Nmay include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in the user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone assistant-centric interactive speaker, a standalone assistant-centric interactive display with a speaker, a smart appliance such as a smart television, and / or a user wearable device that includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided.

[0074] As mentioned above, the client device 110 1-N One or more of may at least selectively interface with the cloud automated assistant component 120 in processing the input and / or generating the output based on the input and / or generating the smart device control commands based on the input. The cloud automated assistant component 120 may include an STT engine 121, an NLU engine 122, a TTS engine 123, a cache generation engine 125, and / or a cloud fulfillment engine 127.

[0075] As mentioned above, the client device 110 1-N Each NLU engine in 113 1-N and / or their respective local caches 158 1-NFor received requests for which there is no matching cache entry in the cloud automated assistant component 120, text and / or audio data corresponding to the request can be sent to the cloud automated assistant component 120. The cloud automated assistant component 120 can utilize its more robust NLU engine 122 to generate a semantic representation based on processing the text of the request. The processed text can be sent by the client device or can be recognized text generated by the STT engine 121 utilizing audio data sent by the client device. The generated semantic representation can include a cloud semantic representation and / or a locally interpretable semantic representation, as described herein. As also described herein, in generating the semantic representation, the NLU engine 122 can generate a semantic representation based on the text of the request. 1-N and smart devices 145 A-N The above device topology 152 can be utilized.

[0076] The cache generation engine 125 is connected to the client device 110. 1-N In response to a transmission from one of the local caches 158, the cache generation engine 125 may generate a cache request including the generated semantic representation and optionally including a cache entry including text (or a representation thereof) and a mapping of the text to the semantic representation. The cache entry of the generated cache request may include additional and / or alternative content, such as rendered response content, protocol suites, adapters, and / or radios, and / or other content, utilized with the locally interpretable semantic representation. The rendered response content may include text, audio data (e.g., synthetic speech for the response text generated using the TTS engine 123), and / or graphics. The cache generation engine 125 may generate a cache request for each local cache 158. 1-NEach client device 110 may then send a corresponding cache request to the client device 110 to have the client device store the cache entry. 1-N Each cache engine 118 1-N Send to.

[0077] If the generated semantic representation includes a cloud semantic representation, the cloud fulfillment engine 127 can process the cloud semantic representation to generate a corresponding control request, which is then sent to the corresponding smart device system 140. A-N The smart device system 140 A-N In response to the control request, the smart device 145 generates a corresponding control command and transmits the corresponding control command to the smart device 145. A-N can be sent to.

[0078] If the generated semantic representation includes a locally interpretable semantic representation, each client device 110 A-N generate corresponding local control commands and transmit those local control commands (e.g., to respective local modules 116 A-N Alternatively, the cloud automated assistant component 120 can provide a locally interpretable semantic representation in a separate transmission to each client device 110. A-N generate corresponding local control commands and transmit those local control commands (e.g., to respective local modules 116 A-N As yet another alternative, the cloud automated assistant component 120 itself can generate local control commands and transmit the local control commands to each client device 110 in order to transmit the control commands locally. A-N can be sent to.

[0079] As also described above, for each received request, the cloud semantic representation is stored in a respective local cache 158 1-N , the client device 110 that matches the cache entry 1-N In the cloud fulfillment engine 127, the cloud semantic representation can be sent to the cloud automated assistant component 120 (e.g., instead of the corresponding text and / or audio data). The cloud fulfillment engine 127 can process the received cloud semantic representation to generate a corresponding control request, which can then be sent to the corresponding smart device system 140. A-N The smart device system 140 A-N In response to the control request, the smart device 145 generates a corresponding control command and transmits the corresponding control command to the smart device 145. A-N can be sent to.

[0080] A further description of the various components of Figure 1 will now be provided with reference to additional figures. Figure 2 shows a home floor plan including multiple rooms 250, 252, 254, 256, 258, 260, and 262. Multiple client devices 110 1-3 The client devices 110 are distributed throughout at least some of the rooms. 1-3 Each of the first client device 110 may optionally implement an instance of an automated assistant client configured in accordance with selected aspects of the present disclosure and may optionally include one or more input devices, such as a microphone, a touch screen, etc., and / or one or more output devices, such as a speaker, a display, etc. For example, the first client device 110 may take the form of an interactive standalone speaker. 1 is deployed in a room 254, which in this example is a bedroom. A second client device 110, which takes the form of a standalone interactive speaker and display device (e.g., a display screen, projector, etc.) 2is deployed in a room 252, which in this example is a living room. A third client device 110, also in the form of an interactive standalone speaker 3 will be stationed in room 256.

[0081] Multiple Client Devices 110 1-3 may be communicatively coupled to each other and / or other resources (e.g., smart devices and the Internet) via a wireless router 101 shown in room 252, and / or a local mesh network. Additionally, other client devices (e.g., mobile devices such as smartphones, tablets, laptops, wearable devices, etc.) may also be present, e.g., carried by one or more people (e.g., users 103) in the home, and may or may not be connected to the same LAN. It should be understood that the configuration of client devices shown in FIG. 2 is merely an example, and that more or fewer and / or different client devices may be deployed throughout any number of other rooms and / or areas outside the home.

[0082] A number of smart devices are further illustrated in FIG. 2. The smart devices include smart lighting 145 A1 Including: Smart Lighting 145 A1 a first remote smart device system 140 A The smart device can be controlled by a second remote smart device system 140. B 145 Smart Lighting Controllable by B1 and 145 B2 The second remote smart device system 140 B a first remote smart device system 140 A The smart device may be controlled by a different party than the party that controls the smart device. 2 Smart thermostat 145 locally controllable by C1For example, smart thermostat 145 C1 Smart Thermostat 145 C1 Compliant with the protocol suite of Smart Thermostat 145 C1 and a second client device 110 3 via a Bluetooth connection between the second client device 110 2 2 is merely an example, and it should be understood that more or fewer and / or different smart devices may be deployed throughout any number of other rooms and / or areas outside of the home.

[0083] FIG. 2 and the above description of FIG. 2 will now be utilized in describing various embodiments of FIGS. 3A, 3B, 4A, 4B, 5A, and 5B.

[0084] FIG. 3A illustrates the second assistant client device 110 of FIG. 2 The utterance 352 of “Turn on the lights a little brighter” received at the second assistant client device 110 2 If there is no cache entry that matches the recognized text 352A of the utterance 352 (the recognized text 352A is generated using an STT engine on the device), and optionally, the second assistant client device 110 2 3A illustrates an example of how a local NLU engine of the second assistant client device 110 can effectively process the recognized text 352A. 2 The second assistant client device 110 transmits the recognized text 352A to the cloud automated assistant component 120. The transmission of the recognized text 352A is performed when the second assistant client device 110 determines that there is no locally stored cache entry that matches the recognized text 352A of the utterance 352. 2and optionally a second assistant client device 110 that is not capable of effectively processing the recognized text 352A. 2 It can respond to the local NLU engine.

[0085] In response to receiving the recognized text 352A, the cloud automated assistant component 120 generates a cloud semantic representation 354 of the recognized text 352A. The cloud semantic representation 354 is provided to the second assistant client device 110. 2 The cloud semantic representation may be generated based on the recognized text being received from the second assistant client device 110 and based on a reference to the remotely stored device topology. In generating the cloud semantic representation, the cloud automated assistant component 120 may resolve the "light" in the recognized text 352A to a specific light. For example, the cloud automated assistant component 120 may utilize the account identifier received in the transmission of the text 352A to identify the account identifier's corresponding remotely stored device topology. Additionally, the cloud automated assistant component 120 may reference the second assistant client device 110 in the device topology. 2 The second assistant client device 110 2 In addition, the cloud automated assistant component 120 can use the identifier of the second assistant client device 110 in the device topology to find the “lighting” in the text 352A. 2 Based on those lights that are mapped as the default lights of Lighting 145 B1 and 145 B2 This can be solved as follows. Lighting 145 B1 and 145 B2 based on previous user interface inputs and / or based on all of those assigned to the room 252 identifier in the device topology, the second assistant client device 1102 can be mapped as the default.

[0086] In generating the cloud semantic representation, the cloud automated assistant component 120 can also resolve "make ... a little brighter" to mean "make it 2 / 5 (e.g., 40%) brighter than the current brightness intensity level." This can be based on a reference to one or more remotely stored models and / or rules accessible to the cloud automated assistant component 120. Cloud semantic representations 354 can be generated based on these resolutions, e.g., as shown in cache request 356, for example, [device=145 B1 and 145 B2 , brighter, relative intensity 2 / 5], where "145 B1 and 145 B2 " is a unique identifier for the corresponding light, and "brighter, relative intensity 2 / 5" indicates that the lights will be about 2 / 5 brighter compared to their current intensity.

[0087] The cloud automated assistant component 120 utilizes the generated cloud semantic representation 354 to generate a corresponding control request. B1 and 145 B2 140 compatible smart device systems B In response, the smart device system 140 B generates corresponding control commands to make the lights “a little brighter” (i.e., increase the relative luminance by 40%, as indicated by the cloud semantic representation) and assigns those control commands to the lighting 145 B1 and 145 B2 can be provided to.

[0088] Additionally, the cloud automated assistant component 120 generates a cache request 356 including the cloud semantic representation 354 and, optionally, including the recognized text 352A and / or a mapping of the recognized text to the cloud semantic representation 354, and transmits the cache request 356 to the client device 110. 2 In response to receiving the cache request 356, the client device 110 2 stores a corresponding cache entry that includes a mapping of the recognized text 352A to the cloud semantic representation 354 included in the cache request 356.

[0089] FIG. 3B also illustrates a second assistant client device 110 2 3A is received at the second assistant client device 110 in FIG. 3A. 2 1 shows an example of how the data can be processed differently after being stored in

[0090] In FIG. 3B, the second assistant client device 110 2 processes the audio data capturing the speech 352 to generate recognized text. 2 The recognized text is sent to the second assistant client device 110 in FIG. 2 In response, the second assistant client device 110 determines that the text matches the text of a cache entry stored in the local cache of the second assistant client device 110. 2 The cloud automated assistant component 120 then generates a corresponding control request and sends the control request to the smart device system 140. BIn response, the smart device system 140 can process the cloud semantic representation 354 directly to transmit it to the cloud. B generates corresponding control commands to make the light "a little brighter" and applies those control commands to the light 145 B1 and 145 B2 In these and other ways, resources of the cloud automated assistant component can be conserved by preventing the generation of the cloud semantic representation from occurring again. Furthermore, control requests can be generated and sent more quickly, and lighting 145 B1 and 145 B2 3B may differ from those in FIG. 3A, even though they are generated based on the same semantic representation. This is due to the relative state changes in the cloud semantic representation, the cloud state changes, and the lighting 145 when generating the control request. B1 and 145 B2 3A and 3B.

[0091] FIG. 4A illustrates the first assistant client device 110 of FIG. 2 3A and 3B ("Turn the lights a little brighter") received by the first assistant client device 110 2 4A and 4B show examples of how a user 103 may process an utterance 452 that matches the text of the utterance when there is no cache entry in the utterance that matches the text of the utterance. In FIG. 4A, the user 103 provides the utterance 452, "Turn the lights up a little." Utterance 452 is the same utterance as utterance 352 in FIG. 3A and 3B, but is labeled as 452 in FIG. 4A and 4B for consistency in numbering.

[0092] In FIG. 4A, the first assistant client device 110 1The cloud automated assistant component 120 uses a local STT engine to send the recognized text 452A, which is generated by processing the audio data capturing the utterance 452, to the cloud automated assistant component 120. The sending of the recognized text 452A is performed when the first assistant client device 110 determines that there is no locally stored cache entry that matches the recognized text 452A of the utterance 452. 1 and optionally a first assistant client device 110 that is unable to effectively process the recognized text 452A. 1 It can respond to the local NLU engine.

[0093] In response to receiving the recognized text 452A, the cloud automated assistant component 120 generates a cloud semantic representation 454 of the recognized text 452A. The cloud semantic representation 454 may be, for example, a representation of the recognized text 452A, as shown in the cache request 456, for example, [device=145 A1 , brighter, relative intensity 2 / 5], where "145 A1 " is a unique identifier for the corresponding light, and "brighter, relative intensity 2 / 5" indicates that the lights will be about 2 / 5 brighter compared to their current intensity. B1 and 145 B2 Identifier instead of lighting 145 A1 3A in that it includes an identifier of the first assistant client device 110. 1 The first assistant client device 110 1 4, and the first assistant client device 110 uses the identifier of the “light” in the text 452A based on the light that is mapped in the device topology. 1 Lighting 145 as default lighting A13A and 4A, different cloud semantic representations can be generated based at least in part on the utterances received from different assistant client devices.

[0094] The cloud automated assistant component 120 utilizes the generated cloud semantic representation 454 to generate a corresponding control request. The cloud automated assistant component 120 performs illumination 145 of the semantic representation 454. A1 140 compatible smart device systems A In response, the smart device system 140 A generates corresponding control commands to make the light "a little brighter" and applies those control commands to the light 145 A1 can be provided to.

[0095] Additionally, the cloud automated assistant component 120 generates a cache request 456 including the cloud semantic representation 454 and, optionally, including the recognized text 452A and / or a mapping of the recognized text to the cloud semantic representation 454, and transmits the cache request 456 to the first client device 110. 1 In response to receiving the cache request 456, the first client device 110 1 stores a corresponding cache entry that includes a mapping of the recognized text 452A to the cloud semantic representation 454 included in the cache request 456.

[0096] FIG. 4B illustrates a first assistant client device 110 1 4A is received at the first assistant client device 110 in FIG. 4A. 1FIG. 13 shows an example of how the image can be processed differently after being stored in the image.

[0097] In FIG. 4B, the first assistant client device 110 1 processes the audio data capturing the speech 452 to generate recognized text. 1 The recognized text is displayed on the first assistant client device 110 in FIG. 1 In response, the first assistant client device 110 determines that the text matches the text of a cache entry stored in the local cache of the first assistant client device 110. 1 The cloud automated assistant component 120 then generates a corresponding control request and transmits the control request to the smart device system 140. A In response, the smart device system 140 can process the cloud semantic representation 454 directly to transmit it to the cloud. A generates corresponding control commands to make the light "a little brighter" and applies those control commands to the light 145 A1 can be provided to.

[0098] FIG. 5A illustrates the second assistant client device 110 of FIG. 2 Another utterance 552, “Turn down the temperature a little,” received at 2 If there is no cache entry that matches the recognized text 552A of the utterance 552 (generated using an on-device STT engine), and optionally, the second assistant client device 110 2 4 shows an example of how a local NLU engine in a language other than the one shown in FIG. 4 cannot process recognized text 452A effectively.

[0099] In FIG. 5A, the second assistant client device 110 2 The second assistant client device 110 transmits the recognized text 552A to the cloud automated assistant component 120. The transmission of the recognized text 552A is performed when the second assistant client device 110 determines that there is no locally stored cache entry that matches the recognized text 552A of the utterance 552. 2 and optionally a second assistant client device 110 that is not capable of effectively processing the recognized text 552A. 2 It can respond to the local NLU engine.

[0100] In response to receiving the recognized text 552A, the cloud automated assistant component 120 generates a locally interpretable semantic representation 554 of the recognized text 552A. In some implementations, the cloud automated assistant component 120 generates a locally interpretable semantic representation 554 of the recognized text 552A (the thermostat 145 C1 ) is a smart device inferentially referenced by the second assistant client device 110 2 and / or a second assistant client device 110 2 In response to determining that the semantic representation 554 is locally controllable by another client device in network communication with the cloud-based device, the cloud-based device generates a semantic representation 554 that is locally interpretable (e.g., instead of or in addition to a cloud-based one).

[0101] The request to "turn down the temperature" is Thermostat 145 C1 A locally interpretable semantic representation 554 can be generated based on a reference to the remotely stored device topology to determine that the text 552A is directed to the thermostat 145 (e.g., it may be the only smart device in the device topology that has “turn down the temperature” as a supported state change). C1may be resolved to inferentially reference the target smart device as the target set point. In generating the cloud semantic representation, the cloud automated assistant component 120 may also resolve "turn down ... a little" to mean "2 degrees lower than the current set point". This may be based on a reference to one or more remotely stored models and / or rules accessible to the cloud automated assistant component 120. A locally interpretable semantic representation 554 may be generated based on these resolutions and may further be generated to allow the client device to include a relative state change representation that directs how to locally resolve the target set point of "2 degrees lower than the current set point". For example, the relative state change representation may be "Temperature setting = (current temperature - 2 degrees)".

[0102] The locally interpretable semantic representation 556 may optionally further include an indication of an adapter (e.g., an adapter specific to the manufacturer of the smart device) to be utilized in generating the corresponding local control command, an indication of a channel (e.g., Bluetooth or Wi-Fi) to be utilized to transmit the local control command, and / or an indication of a protocol suite to be utilized in generating a control command based on the locally interpretable semantic representation 556. For example, as shown in the cache request 556, the locally interpretable semantic representation may include [device=145 C1 , temperature setting = (current temperature - 2 degrees), adapter / channel = C], and in the above formula, "145 C1 " is Thermostat 145 C1 A unique identifier for the C1', "temperature setting=(current temperature - 2 degrees)" is a representation of a state change that depends on a locally interpretable current state for resolving the current state change, and "adapter / channel=C" indicates an adapter and / or protocol suite that should be utilized in generating a local control command (e.g., after determining the current state change) and / or a channel that should be utilized to transmit the local control command.

[0103] The cloud automated assistant component 120 generates a cache request 556 including the cloud semantic representation 554 and, optionally, including the recognized text 552A and / or a mapping of the recognized text to the cloud semantic representation 554, and transmits the cache request 556 to the second client device 110. 2 In response to receiving the cache request 556, the second client device 110 2 stores a corresponding cache entry that includes a mapping of the recognized text 552A to the cloud semantic representation 554 included in the cache request 556.

[0104] In some implementations, the second client device 110 2 and thermostat 145 C1 As indicated by the dashed line between 2 Thermostat 145 C1 and locally generate local control commands for the thermostat 145 in response to the utterance of FIG. C1 A locally interpretable (or from a separate transmission from the cloud automated assistant component 120) semantic representation of the cache request can be utilized to send it locally to the cloud automated assistant component 120.

[0105] In some alternative implementations, the cloud automated assistant component 120 generates a corresponding cloud semantic representation based on the recognized text 552A. Additionally, the cloud automated assistant component 120 can utilize a separate cloud semantic representation to generate a corresponding control request. The cloud automated assistant component 120 can generate a corresponding control request based on the recognized text 552A. C1 140 compatible smart device systems C In response, the smart device system 140 C 145 generates corresponding control commands to make its setpoint “a little lower” (i.e., 2 degrees lower as indicated by the cloud semantic representation) and sends those control commands to the thermostat 145 C1 can be provided to.

[0106] FIG. 5B illustrates a second assistant client device 110 2 5A is received at the second assistant client device 110 in FIG. 5A. A cache entry having a locally interpretable semantic representation 554 (FIG. 5A) is then read by the second assistant client device 110 in FIG. 2 1 shows an example of how the data can be processed differently after being stored in

[0107] In FIG. 5B, the second assistant client device 110 2 processes the audio data capturing the speech 552 to generate recognized text. 2 The recognized text is sent to the second assistant client device 110 in FIG. 2 In response, the second assistant client device 110 determines that the text matches the text of a cache entry stored in the local cache of the second assistant client device 110. 2identifies a locally interpretable semantic representation 554 of the cache entry and uses the locally interpretable semantic representation 554 to generate a corresponding control command. The state representation of the locally interpretable semantic representation 554 is a representation of a state that depends on the current state of the thermostat 145. C1 5A and 5B, the generated control commands in FIG. 5B may differ from those in FIG. 5A. For example, in FIG. 5B, the “current” temperature setpoint of the second assistant client device 110 2 may solve for the target set point by subtracting 2 degrees from the current set point, as dictated by the relative state change in the locally interpretable semantic representation 554. The second assistant client device 110 then 2 generates a control command and communicates the control command to the thermostat 145, which may optionally include a locally interpretable semantic representation 554. C1 In response to receiving the control command, the thermostat 145 may process the target set point using a corresponding adapter to transmit the control command to the address of the thermostat 145. C1 can lower its set point by two degrees.

[0108] FIG. 6 is a flowchart illustrating an example method 600 that may be implemented by one or more processors of an assistant client device according to various implementations disclosed herein.

[0109] In block 602, the assistant client device processes the audio data capturing the speech to generate recognized text of the speech. In other implementations, instead of processing the audio data to generate recognized text, the assistant client device can identify the provided text based on directly typed text (e.g., using a virtual keyboard) or based on corresponding hardware or software button presses (e.g., the text "dim the lights" mapped to a graphical software button labeled "dim").

[0110] In block 604, the assistant client device determines whether the text matches a local cache entry. If so, the client device proceeds to block 606 and selects a semantic representation that is from the matching local cache entry. Then, in block 608, the system uses the semantic representation. Block 608 can include sub-blocks 608A and / or 608B. In sub-block 608A, the client device locally processes any of the selected semantic representations that are locally interpretable semantic representations to generate local control commands and transmits the control commands via a local channel. In sub-block 608B, the client device transmits any of the selected semantic representations that are cloud semantic representations to a remote system. For example, they can be transmitted to a remote system that can perform blocks 812 and 814 of method 800 (FIG. 8, described below). In some implementations, block 606 can include additional sub-blocks, such as a sub-block in which the assistant client device renders any response content included in the local cache entry. For example, the local cache entry can include a response text, and the assistant client device can perform local TTS processing on the response text and audibly render the resulting synthetic speech. As another example, the local cache entry can additionally or alternatively include a response graphic, and the graphic can be visually rendered on the display of the assistant client device.

[0111] If the determination in block 604 is no, the assistant client device proceeds to optional block 610 to determine whether the text is resolvable in the local NLU (e.g., using a local NLU engine and a local model). If so, the system proceeds to block 612 to generate a semantic representation (locally interpretable and / or cloud), and then proceeds to block 608 to use the generated semantic representation.

[0112] If the determination in block 610 is no (or if the determination in 604 is no and 610 is omitted), the client device proceeds to block 614 and transmits the audio data and / or text to the remote assistant system. Optionally, block 614 may occur before and / or during execution of blocks 604 and / or 610, and further processing in the remote system is optionally stopped in response to a "yes" determination in either of blocks 604 and 610 (e.g., in response to a stop of transmission from the client device). For example, if block 614 occurs before and / or during execution of blocks 604 and / or 610, and the assistant client device makes a "yes" determination in blocks 604 and / or 610, a stop request can be sent to the remote assistant system to stop any further processing by the remote assistant system related to the transmitted audio data and / or text. In these and other schemes, to conserve remote assistant system resources, processing in the remote assistant system can begin more quickly while reducing latency if the determination at blocks 604 and / or 610 is "no" and allowing such processing to stop if the determination is "yes."

[0113] In block 616, the client device receives a cache request from the remote assistant system, including a semantic representation of the text, and optionally the text itself (and / or its representation). Block 616 optionally includes sub-block 616A, where the assistant client device optionally proceeds to block 608A to process one or more locally interpretable semantic representations (if any) of the cache request (or a separate transmission). In other words, in block 616, the assistant client device can immediately act on the locally interpretable semantic representations of the cache request to enable any local smart device actions in response to the current utterance or other current user input.

[0114] At block 618, the client device stores in a local cache and, in response to receiving the cache request, a cache entry including a mapping of the text to the semantic representation of the cache request.

[0115] Block 618 optionally includes sub-block 618A, which may occur near or after block 618. In sub-block 618A, the client device optionally assigns the text of the cache entry for an action without invocation. For example, the client device may assign the text as a "hot phrase" as described herein. The assignment of the text as a "hot phrase" may optionally be based on a determination of whether one or more criteria are met. If the recognized text from on-device voice recognition of audio data occurring independent of an explicit automated assistant invocation is determined to match a hot phrase, the assistant client device may automatically initiate one or more corresponding actions based on the text (e.g., an action based on the semantic representation of the corresponding cache entry).

[0116] 7 is a flowchart illustrating another example method 700 that may be implemented by one or more processors of an assistant client device according to various implementations disclosed herein. Method 700 illustrates a particular implementation of method 600 of FIG. 6, where the text is determined to match the text of a cache entry (i.e., "yes" in block 604 of FIG. 6), and the semantic representation of the cache entry includes a locally interpretable semantic representation.

[0117] In block 702, the assistant client device processes the audio data capturing the speech to generate recognized text of the speech. In other implementations, instead of processing the audio data to generate recognized text, the assistant client device can identify the provided text based on directly typed text (e.g., using a virtual keyboard) or based on corresponding hardware or software button presses (e.g., the text "dim the lights" mapped to a graphical software button labeled "dim").

[0118] In block 704, the assistant client device determines that the text matches the text of the cache entry. Block 704 optionally includes sub-block 704A, where the client device optionally determines that the text is assigned to an uninvoked action (e.g., at the assistant client device). For example, block 704A can be optionally executed if the processing of block 702 is occurring without an explicit invocation of an automated assistant. Furthermore, in such an example, one or more subsequent blocks of method 700 can be executed only if it is determined that the text is assigned to an uninvoked action (e.g., locally at the client device). In other words, block 704A can be executed when the processing of block 702 is occurring without an explicit invocation of an automated assistant, and if it is determined that the text is not assigned to an uninvoked action, the text can be discarded without executing further blocks of method 700 based on the text, but if it is determined that the text is assigned to an uninvoked action, further blocks of method 700 are executed based on the text.

[0119] In block 706, the client device selects a locally interpretable semantic representation based on which it is mapped in the cache entry to the text of the cache entry.

[0120] In block 708, the client device determines whether the state change of the locally interpretable semantic representation is a state change that is dependent on the current state. If the state change is not dependent on the current state, the client device proceeds to block 712 and uses the state change of the locally interpretable semantic representation. For example, if the state change indicates a fixed value, the fixed value may be utilized as the state change. If the determination in block 708 is "yes," the client device proceeds to block 710 and resolves the state change based on the current state value. For example, if the state change is dependent on the current state, it may be resolved by identifying a current state of one or more smart devices and then determining a state change based on the identified current state. For example, if the state change is "set point = (current temperature + 3 degrees)," it may be resolved by identifying "current temperature."

[0121] At block 714, the client device generates a local control command using the state change, and optionally using the adapter identified based on the cache entry.

[0122] At block 716, the client device transmits local commands over a local channel that is optionally selected based on the cache entry (e.g., explicitly identified in the cache entry or selected based on the protocol suite, smart device, and / or radio / channel identified in the cache entry). The transmitted local commands can be addressed to the smart devices based on their addresses identified in or resolvable from the locally interpretable semantic representation.

[0123] In block 718, the client device determines whether there are any additional unprocessed semantic representations in the cache entry. If there are no semantic representations, the client device proceeds to block 720 and the method 700 ends. If there are semantic representations, the client device proceeds to block 722 and determines whether the next unprocessed semantic representation is locally interpretable. If it is locally interpretable, the client device returns to block 708. If it is not locally interpretable (i.e., it is a cloud semantic representation), the client device proceeds to block 608B of the method 600 of FIG. 6 and then returns to block 718 after performing block 608B.

[0124] FIG. 8 is a flowchart illustrating an example method 800 that may be implemented by one or more processors of a remote assistant system according to various implementations disclosed herein.

[0125] In block 802, the system receives a request from an assistant client device. In block 804, the system determines whether the request is a cloud semantic representation. If the request is a cloud semantic representation, the system executes blocks 812 and 814 (described below) of method 800 without executing any other blocks of method 800.

[0126] If the request is not a cloud semantic representation, the system proceeds to block 808 and generates a semantic representation of the utterance based on the representation of the utterance in the request. In other implementations, the representation may be typed speech or text mapped to the press of a hardware or software graphical element.

[0127] In optional block 810, the system generates confirmatory content for rendering at the assistant client device in response to the request.

[0128] In optional block 812, the system generates a control request based on the cloud semantic representation generated based on the request. Optional block 812 may include optional block 812A, where if the state representation of the semantic representation is a relative / current state dependent state representation, the system generates the control request based on the current state of the smart device.

[0129] In optional block 814, the system sends the generated control request to the smart device remote system.

[0130] In block 816, the system generates a cache request that includes the semantic representation (locally interpretable and / or cloud) and optionally includes a mapping of text to the semantic representation.

[0131] In block 818, the system sends a cache request to the assistant client device. Block 818 optionally includes sub-block 818A, in which, if the semantic expression of the cache request is a locally interpretable semantic expression, the system optionally generates another request to implement the locally interpretable semantic expression in response to the current request.

[0132] 9 is a block diagram of an exemplary computing device 910 that may optionally be utilized to perform one or more aspects of the techniques described herein. The computing device 910 typically includes at least one processor 914 that communicates with a number of peripheral devices via a bus subsystem 912. These peripheral devices may include, for example, a storage subsystem 924 including a memory subsystem 925 and a file storage subsystem 928, a user interface output device 920, a user interface input device 922, and a network interface subsystem 918. The input and output devices enable user interaction with the computing device 910. The network interface subsystem 918 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0133] The user interface input devices 922 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 910 or onto a communications network.

[0134] The user interface output devices 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and ways of outputting information from the computing device 910 to a user or to another machine or computing device.

[0135] Storage subsystem 924 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 924 may include logic for performing selected aspects of one or more methods described herein.

[0136] These software modules are generally executed by the processor 914 alone or in combination with other processors. The memory 925 used in the storage subsystem 924 can include several memories including a main random access memory (RAM) 930 for storing instructions and data during program execution, and a read-only memory (ROM) 932 in which fixed instructions are stored. The file storage subsystem 928 can provide persistent storage for program and data files and can include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing the functionality of a particular implementation can be stored in the storage subsystem 924 by the file storage subsystem 928 or in other machines accessible by the processor 914.

[0137] The bus subsystem 912 provides a mechanism that allows the various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is shown diagrammatically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0138] The computing device 910 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 910 shown in Figure 9 is intended only as a specific example to illustrate some implementations. Many other configurations of the computing device 910 are possible, having more or fewer components than the computing device shown in Figure 9.

[0139] In situations where certain implementations described herein may collect or use personal information about users (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, user activity and demographic information, relationships between users, etc.), users are provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods described herein collect, store, and / or use a user's personal information only upon receiving explicit permission to do so from the associated user.

[0140] In some implementations, a method is provided that includes receiving, in a remote assistant system and from an assistant client device, a representation of an utterance captured at the assistant client device. The utterance is a request related to one or more states of one or more smart devices linked to the assistant client device. As an example, the request may be a request to change a state of the smart device (i.e., transition the state from a current state value to a new state value). The representation of the utterance comprises audio data capturing the utterance and / or text of the utterance, generated at the client device utilizing a speech-to-text model stored locally on the client device. The method further includes, in the remote system and in response to receiving the representation of the utterance, generating a semantic representation of the utterance based on the representation of the utterance, generating at least one control request based on the semantic representation of the utterance for sending to at least one smart device remote system for controlling one or more smart devices, the at least one control request being different from the semantic representation, sending the at least one control request to the at least one smart device remote system to cause the at least one smart device remote system to provide one or more corresponding commands to the one or more smart devices, and sending a cache request including the semantic representation to the assistant client device. Sending the cache request to the assistant client device causes the assistant client device to store a cache entry including a mapping of text to the semantic representation in a cache of the assistant client device.

[0141] Implementations of these and other techniques may optionally include one or more of the following features.

[0142] In some implementations, the method further includes, after the step of sending the cache request to the assistant client device, capturing additional audio data at the assistant client device capturing the additional speech, processing the additional audio data to generate a current text corresponding to the additional speech using a speech-to-text model stored locally on the assistant client device, determining that the current text matches the text of the cache entry, and in response to the determination that the current text matches the text of the cache entry and in response to the cache entry including a mapping of the text to a semantic representation, sending the semantic representation to the remote assistant system. In some of these implementations, the step of processing the additional audio data to generate the current text is performed without detection of an explicit automated assistant invocation at the assistant client device. In some versions of these implementations, the step of sending the semantic representation to the remote assistant system is further responsive to a determination that the current text meets one or more criteria for allowing the matching text to act without an explicit automated assistant invocation at the assistant client device. In some additional or alternative versions, the method further includes, at the remote assistant system, receiving a semantic representation sent from the assistant client device; generating at least one alternate control request based on the semantic representation for sending to at least one smart device remote system that controls one or more smart devices, where the at least one alternate control request is different from the at least one control request; and sending the at least one alternate control request to the at least one smart device remote system to cause the at least one smart device remote system to change one or more states of the one or more smart devices.Optionally, in those alternative or additional implementations, the step of generating the at least one control request is further based on at least one current state of the one or more smart devices at a time of generation of the at least one control request, and the step of generating the at least one alternative control request is further based on at least one alternative current state of the one or more smart devices at a different time of generation of the at least one alternative control request, and the at least one alternative control request differs from the at least one control request based on at least one current state that differs from the at least one alternative current state.

[0143] In some implementations, generating the semantic representation of the utterance is further based on a device topology of the account of the assistant client device, the device topology defining the assistant client device, the one or more smart devices, the additional assistant client device, and the additional smart device. In some of these implementations, generating the semantic representation of the utterance further based on the device topology includes resolving ambiguous terms in the text of the utterance to refer to one or more smart devices based on the device topology.

[0144] In some implementations, the semantic representation includes a state change representation of one or more state changes and a corresponding unique identifier for each of the one or more smart devices. In some versions of those implementations, the state change representation is a relative state change representation that defines one or more state changes in a relative manner rather than an absolute manner. In some additional or alternative versions of those implementations, the method further includes, in the remote assistant system or the assistant client device, determining a change to the device topology, and, in response to determining the change to the device topology, causing a cache entry to be flushed from the cache of the assistant device. The step of determining the change to the device topology may optionally include determining that the change is to one or more properties defined in the device topology for the assistant client device, the one or more smart devices, and / or one or more rooms assigned to the assistant client device and / or the one or more smart devices, and optionally, causing the cache entry to be flushed from the cache may further be responsive to a determination that the cache entry is affected by the one or more properties.

[0145] In some implementations, the method further includes receiving an additional representation of the utterance at the remote assistant system and from the additional assistant client device. The utterance is captured at the additional assistant client device, and both the additional assistant client device and the assistant client device are members of a common device topology. In some of those implementations, the method further includes generating an alternative semantic representation of the utterance based on the representation of the utterance at the remote system and in response to receiving the additional representation of the utterance, the alternative semantic representation of the utterance being different from the additional representation captured at the additional assistant client device and the semantic representation based on the representation captured at the assistant client device, and sending an additional cache request including the alternative semantic representation to the additional assistant client device. Upon sending the additional cache request to the additional assistant client device, the additional assistant client device stores an additional cache entry including a mapping of text to the alternative semantic representation in an additional cache of the additional assistant client device.

[0146] In some implementations, a method is provided that includes receiving, at a remote assistant system and from an assistant client device, a representation of an utterance captured at the assistant client device. The utterance is a request to change at least a given state of a given smart device linked to the assistant client device. The representation of the utterance includes audio data capturing the utterance and / or text of the utterance generated at the client device utilizing a speech-to-text model stored locally on the client device. The method further includes, at the remote system and in response to receiving the representation of the utterance, determining that the given smart device is locally controllable by the assistant client device and / or by an additional client device having a local connection to the assistant client device, generating a semantic representation of the utterance including a locally interpretable semantic representation, and sending a cache request to the assistant client device comprising the semantic representation. The locally interpretable semantic representation is locally interpretable by the assistant client device and / or the additional client device to generate a corresponding control command transmittable over a local channel to cause a change in at least a given state of the given smart device. The step of generating a locally interpretable semantic representation of the at least one smart device includes, in response to determining that the given smart device is locally controllable, including the locally interpretable semantic representation in the semantic representation. When the cache request is sent to the assistant client device, the assistant client device stores a cache entry in the cache of the assistant client device, the cache entry including a mapping of the text to the semantic representation.

[0147] Implementations of these and other techniques may optionally include one or more of the following features.

[0148] In some implementations, the method further includes, after the step of sending the cache request to the assistant client device, capturing additional audio data at the assistant client device capturing the additional utterance, processing the additional audio data to generate a current text corresponding to the additional utterance using a speech-to-text model stored locally on the assistant client device, determining that the current text matches the text of the cache entry, and in response to the determination that the current text matches the text of the cache entry and in response to the cache entry including the mapping of the text to the semantic representation, processing the locally interpretable semantic representation at the assistant client device to generate a given control command, and transmitting the given control command via a local channel to cause a change in at least a given state of the given smart device. In some versions of these implementations, the locally interpretable semantic representation includes an identifier of the given smart device and a current state-dependent state change representation that defines a change in at least a given state of the given smart device relative to a corresponding current value of the given state of the given smart device. In some of these versions, processing the locally interpretable semantic representation to generate a given control command includes, at the assistant client device, determining a current value of a given state of the given smart device, and generating the given control command based on applying the current value to a state change representation that depends on the current state. Optionally, the locally interpretable semantic representation further includes instructions from a manufacturer of the smart device, and generating the given control command optionally includes utilizing an adapter stored locally on the assistant client device in response to an adapter assigned to the instructions from the manufacturer of the smart device.Optionally, the method further comprises selecting a local channel for transmitting the given control command based on the local channel being directly or indirectly indicated in the locally interpretable semantic representation, e.g., based on a local channel assigned in a manufacturer's instructions for the smart device included in the locally interpretable semantic representation and / or assigned to a protocol suite included in the locally interpretable semantic representation.

[0149] In some implementations, the step of processing the additional audio data to generate the current text is performed without detection of an explicit automated assistant invocation at the assistant client device. In some of these implementations, the step of sending the given control command is further responsive to a determination that the current text meets one or more criteria for allowing the matching text to act without an explicit automated assistant invocation at the assistant client device.

[0150] In some implementations, generating the semantic representation of the utterance is further based on a device topology of the account of the assistant client device. The device topology defines the assistant client device, the given smart device, the additional assistant client device, and the additional smart device. In some of these implementations, generating the semantic representation of the utterance includes resolving ambiguous terms in the text of the utterance to refer to the given smart device.

[0151] In some implementations, the method further includes, at the remote assistant system or the assistant client device, determining a change to the device topology, and flushing a cache entry from the cache of the assistant client device in response to determining the change to the device topology. In some of these implementations, determining the change to the device topology includes determining that the change is to one or more properties defined in the device topology for the assistant client device, the given smart device, and / or one or more rooms assigned to the assistant client device and / or the given smart device, and flushing the cache entry from the cache is further responsive to determining that the cache entry is affected by the one or more properties.

[0152] In some implementations, a method is provided that includes storing a cache entry in a cache on a client device, the cache entry including a mapping of the text to a semantic representation. The semantic representation includes a locally interpretable semantic representation that is locally interpretable by the client device. The method further includes capturing additional audio data at the client device capturing the utterance. The method further includes processing the additional audio data to generate a current text corresponding to the utterance using a speech-to-text model stored locally on the client device. The method further includes determining that the current text matches the text of the cache entry. The method further includes processing the locally interpretable semantic representation at the client device to generate a given control command in response to determining that the current text matches the text of the cache entry and in response to the cache entry including the mapping of the text to the semantic representation, and transmitting the given control command via a local channel to cause a change in at least a given state of the given smart device. [Explanation of symbols]

[0153] 101 Wireless Router 103 users 105 1 Wide Area Network 105 2 Wide Area Network 105 3 Wide Area Network 110 1 A client device, a first client device 110 1-N Client Device 110 2 Second client device 110 3 Third Client Device 111 I / O Components 111 1-N I / O Components 112 STT engine 112 1-N Speech to Text (STT) Engine 113 1 NLU Engine 113 1-N Natural Language Understanding (NLU) Engine 114 1-N Fulfillment Engine 115 1 Cloud Module 115 1-N Cloud Module 116 1 Local Modules 116 1-N Local Modules 117 1 Automated Assistant Client, TTS Engine 117 1-N Text-to-Speech (TTS) Engine 118 1-N Cache Engine 119 1 radio 120 Cloud Automated Assistant Components 120 Cloud-Based Components 121 STT engine 122 NLU Engine 123 TTS engine 125 Cache Generation Engine 127 Cloud Fulfillment Engine 140 Smart Device System 140 A The first remote smart device system 140 B Second Remote Smart Device System 140 A-N Smart Device System 145 A-N Smart Devices 145 1-N Smart Devices 145 A1 Smart Lighting 145 A1 The first smart device 145 B1 Second Smart Device 145 B1 Smart Lighting 145 B2 The third smart device 145 B2 Smart Lighting 145 C1 Smart Thermostat 152 Device Topology 152 1-N STT model 153 1-N NLU Model 156 1-N adapter 158 1 Local Cache 158 1-N Local Cache 250 rooms 252 rooms 254 rooms 256 rooms 258 rooms 260 rooms 262 rooms 352 utterances 352A Recognized text 354 Cloud Semantic Representation 356 Cache Requests 452 utterances 452A Recognized text 454 Cloud Semantic Representation 456 Cache Request 552 utterances 552A Recognized text 554 Cloud Semantic Representation, Locally Interpretable Semantic Representation 556 Cache Requests 600 ways 700 methods 800 ways 910 Computing Devices 912 Bus Subsystem 914 Processor 916 Network Interface Subsystem 920 User Interface Output Device 922 User Interface Input Devices 924 Storage Subsystem 925 Memory Subsystem 926 File Storage Subsystem 930 Main Random Access Memory (RAM) 932 Read-Only Memory (ROM)

Claims

1. Receiving, at the remote assistant system and from the assistant client device, a representation of the utterance captured at the assistant client device, The utterance is a request related to one or more states of one or more smart devices linked to the assistant client device; the representation of the utterance comprises audio data capturing the utterance and / or text of the utterance, the audio data being generated at the client device utilizing a speech-to-text model stored locally on the client device; at the remote system and in response to receiving the representation of the utterance; generating a semantic representation of the utterance based on the representation of the utterance; generating at least one control request for transmission to at least one smart device remote system that controls the one or more smart devices based on the semantic representation of the utterance, the at least one control request being different from the semantic representation; sending the at least one control request to the at least one smart device remote system to cause the at least one smart device remote system to provide one or more corresponding commands to the one or more smart devices; sending a cache request to the assistant client device comprising the semantic representation, the cache request causing the assistant client device to store a cache entry in a cache of the assistant client device comprising a mapping of the text to the semantic representation; A method comprising:

2. After the step of sending the cache request to the assistant client device, Capturing additional audio data at the assistant client device, the additional audio data capturing the additional speech; Processing the additional audio data to generate current text corresponding to the additional utterance using the speech-to-text model stored locally on the assistant client device; determining that the current text matches the text of the cache entry; in response to a determination that the current text matches the text of the cache entry, and in response to the cache entry including the mapping to the semantic representation of the text, transmitting the semantic representation to the remote assistant system; The method of claim 1 , further comprising:

3. The method of claim 2, wherein the step of processing the additional audio data to generate the current text is performed without detection of an explicit automated assistant invocation at the assistant client device.

4. 4. The method of claim 3, wherein the step of transmitting the semantic representation to the remote assistant system is further responsive to a determination that the current text meets one or more criteria for allowing matching text to be acted upon without an explicit automated assistant invocation at the assistant client device.

5. In the remote assistant system, receiving the semantic representation transmitted from the assistant client device; generating at least one alternative control request for sending to the at least one smart device remote system that controls the one or more smart devices based on the semantic representation, the at least one alternative control request being different from the at least one control request; sending the at least one alternate control request to the at least one smart device remote system to cause the at least one smart device remote system to change the one or more states of the one or more smart devices; 5. The method of claim 2, further comprising:

6. the step of generating the at least one control request is further based on at least one current state of the one or more smart devices at a time of generating the at least one control request; the step of generating the at least one alternative control request is further based on a current state of at least one alternative of the one or more smart devices at a different time of generating the at least one alternative control request; The method of claim 5 , wherein the at least one alternate control request differs from the at least one control request based on the at least one current state differing from a current state of the at least one alternate.

7. 7. The method of claim 1, wherein the step of generating the semantic representation of the utterance is further based on a device topology of an account of the assistant client device, the assistant client device, the one or more smart devices, an additional assistant client device, and an additional smart device.

8. generating the semantic representation of the utterance further based on the device topology, 8. The method of claim 7, comprising resolving ambiguous terms in the text of the utterance to refer to the one or more smart devices based on the device topology.

9. The semantic representation is a state change representation of the one or more state changes; and for each of the one or more smart devices, a corresponding unique identifier; 9. The method of any one of claims 1 to 8, comprising:

10. The method of claim 9 , wherein the state change representation is a relative state change representation that defines the one or more state changes in a relative manner rather than in an absolute manner.

11. determining, at the remote assistant system or at the assistant client device, a change to a device topology; In response to determining the change in the device topology, causing the cache entry to be flushed from the cache of the assistant device; 11. The method of claim 7, further comprising:

12. determining the changes to the device topology, determining that the change is to one or more properties defined in the device topology for the assistant client device, the one or more smart devices, and / or one or more rooms assigned to the assistant client device and / or the one or more smart devices; The method of claim 11 , wherein causing the cache entry to be purged from the cache is further responsive to a determination that the cache entry is affected by the one or more properties.

13. receiving an additional representation of the utterance at the remote assistant system and from an additional assistant client device, the utterance being captured at the additional assistant client device, both of the additional assistant client device and the assistant client device being members of a common device topology; at the remote system and in response to receiving the additional representation of the utterance; Generating an alternative semantic representation of the utterance based on the representation of the utterance, the alternative semantic representation of the utterance being different from the semantic representation based on the additional representation captured at the additional assistant client device and the representation captured at the assistant client device; sending an additional cache request to the additional assistant client device comprising the alternative semantic representation, wherein sending the additional cache request to the additional assistant client device causes the additional assistant client device to store an additional cache entry in an additional cache of the additional assistant client device comprising a mapping of the text to the alternative semantic representation; 13. The method of claim 1, further comprising:

14. Receiving, at the remote assistant system and from the assistant client device, a representation of the utterance captured at the assistant client device, The utterance is a request to change at least a given state of a given smart device linked to the assistant client device; the representation of the utterance comprises audio data capturing the utterance and / or text of the utterance, the audio data being generated at the client device utilizing a speech-to-text model stored locally on the client device; at the remote system and in response to receiving the representation of the utterance; Determining that the given smart device is locally controllable by the assistant client device and / or by an additional client device having a local connection to the assistant client device; generating a semantic representation of the utterance, the semantic representation including a locally interpretable semantic representation, The locally interpretable semantic representation is locally interpretable by the assistant client device and / or the additional client device to generate a corresponding control command transmittable over a local channel to cause the change of at least the given state of the given smart device; generating the locally interpretable semantic representation of the at least one smart device comprises, in response to determining that the given smart device is locally controllable, including the locally interpretable semantic representation in the semantic representation; sending a cache request to the assistant client device comprising the semantic representation, the cache request causing the assistant client device to store a cache entry in a cache of the assistant client device comprising a mapping of the text to the semantic representation; A method comprising:

15. After the step of sending the cache request to the assistant client device, Capturing additional audio data at the assistant client device, the additional audio data capturing the additional speech; Processing the additional audio data to generate current text corresponding to the additional utterance using a speech-to-text model stored locally on the assistant client device; determining that the current text matches the text of the cache entry; in response to a determination that the current text matches the text of the cache entry, and in response to the cache entry including the mapping to the semantic representation of the text, Processing the locally interpretable semantic representation at the assistant client device to generate a given control command; transmitting, via a local channel, the given control command to cause the change in at least the given state of the given smart device; The method of claim 14, further comprising:

16. The locally interpretable semantic representation: an identifier of the given smart device; and a current state dependent state change expression that defines the change of at least the given state of the given smart device relative to a corresponding current value of the given state of the given smart device; The method of claim 15 comprising:

17. processing the locally interpretable semantic representation to generate the given control command, Determining, at the assistant client device, a current value of the given state of the given smart device; generating the given control command based on applying the current value to a state change representation that is dependent on the current state; The method of claim 16 comprising:

18. The method of claim 17 , wherein the locally interpretable semantic representation further comprises an indication of a manufacturer of the smart device.

19. 20. The method of claim 18, wherein generating the given control command comprises utilizing an adapter stored locally on the assistant client device in response to the adapter being assigned to the instructions of the manufacturer of the smart device.

20. 20. The method of claim 18 or 19, further comprising selecting the local channel for transmitting the given control command based on the local channel being assigned in the instructions of the manufacturer of the smart device.

21. 21. The method of claim 15, wherein the step of processing the additional audio data to generate the current text is performed without detection of an explicit automated assistant invocation at the assistant client device.

22. 22. The method of claim 21, wherein the step of sending the given control command is further responsive to a determination that the current text meets one or more criteria for allowing matching text to act without an explicit automated assistant invocation at the assistant client device.

23. 23. The method of claim 14, wherein the step of generating the semantic representation of the utterance is further based on a device topology of an account of the assistant client device, the device topology defining the assistant client device, the given smart device, an additional assistant client device, and an additional smart device.

24. generating the semantic representation of the utterance, 24. The method of claim 23, comprising resolving ambiguous terms in the text of the utterance to refer to the given smart device.

25. determining, at the remote assistant system or at the assistant client device, a change to a device topology; In response to determining the change in the device topology, causing the cache entry to be flushed from the cache of the assistant client device; 25. The method of any one of claims 14 to 24, further comprising:

26. determining the changes to the device topology, determining that the change is to one or more properties defined in the device topology for the assistant client device, the given smart device, and / or one or more rooms assigned to the assistant client device and / or the given smart device; 26. The method of claim 25, wherein causing the cache entry to be purged from the cache is further responsive to a determination that the cache entry is affected by the one or more properties.

27. receiving an additional representation of the utterance at the remote assistant system and from an additional assistant client device, the utterance being captured at the additional assistant client device, both of the additional assistant client device and the assistant client device being members of a common device topology; at the remote system and in response to receiving the additional representation of the utterance; Generating an alternative semantic representation of the utterance based on the representation of the utterance, the alternative semantic representation of the utterance being different from the semantic representation based on the additional representation captured at the additional assistant client device and the representation captured at the assistant client device; sending an additional cache request to the additional assistant client device comprising the alternative semantic representation, wherein sending the additional cache request to the additional assistant client device causes the additional assistant client device to store an additional cache entry in an additional cache of the additional assistant client device comprising a mapping of the text to the alternative semantic representation; 27. The method of any one of claims 14 to 26, further comprising:

28. storing a cache entry in a cache on a client device, the cache entry including a mapping of text to a semantic representation, the semantic representation including a locally interpretable semantic representation that is locally interpretable by the client device; capturing additional audio data at the client device capturing speech; processing the additional audio data to generate current text corresponding to the utterance using a speech-to-text model stored locally on the client device; determining that the current text matches the text of the cache entry; in response to a determination that the current text matches the text of the cache entry, and in response to the cache entry including the mapping to the semantic representation of the text, processing the locally interpretable semantic representation at the client device to generate a given control command; transmitting, via a local channel, the given control command to cause a change in at least a given state of the given smart device; 23. A method implemented by one or more processors of the client device, comprising:

29. The locally interpretable semantic representation: an identifier of the given smart device; and a current state dependent state change expression that defines the change of at least the given state of the given smart device relative to a corresponding current value of the given state of the given smart device; 30. The method of claim 28, comprising:

30. processing the locally interpretable semantic representation to generate the given control command, determining, at the client device, a current value of the given state of the given smart device; generating the given control command based on applying the current value to a state change representation that is dependent on the current state; 30. The method of claim 29, comprising:

31. 31. The method of claim 30, wherein the locally interpretable semantic representation further comprises an indication of a manufacturer of the smart device.

32. 32. The method of claim 31, wherein generating the given control command comprises utilizing an adapter stored locally on the assistant client device in response to the adapter being assigned to the instructions of the manufacturer of the smart device.

33. 33. The method of claim 31 or 32, further comprising selecting the local channel for transmitting the given control command based on the local channel being assigned in the instructions of the manufacturer of the smart device.

34. 34. The method of claim 28, wherein the step of processing the audio data to generate the current text is performed without detection of an explicit automated assistant invocation at the client device.

35. 35. The method of claim 34, wherein the step of sending the given control command is further responsive to a determination that the current text meets one or more criteria for allowing matching text to act without an explicit automated assistant invocation at the assistant client device.

36. At least one computing device comprising one or more processors and at least one memory storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 35.

37. 36. A computer program comprising computer-executable instructions which, when executed by one or more processors of at least one computing device, cause the computer to perform the method of any one of claims 1 to 35.

Citation Information

Patent Citations

  • Server used for recognizing browser voice commands and browser voice command recognition system

    CN102629246A

  • Method for managing voice cache

    JP2002073084A

  • Method and system for voice based media search

    US20140122059A1

  • Processing Voice Commands Based on Device Topology

    US20180096683A1

  • Assistance during audio and video calls

    US20190036856A1