Efficient and low latency automated assistant control of smart devices

Abstract

The invention relates to efficient and low latency automated assistant control of smart devices. Various embodiments relate to techniques for controlling smart devices with low latency and / or providing computing efficiency (client and / or server) and / or network efficiency. These embodiments relate to generating and / or utilizing cache entries of a cache locally stored on an assistant client device when controlling various smart devices (e.g., smart lights, smart thermostats, smart plugs, smart devices, smart routers, etc. Each cache entry includes a mapping of text to one or more corresponding semantic representations.

Description

Description of the case This application is a divisional application of the patent application with application number 201980022670.4, which was submitted to the China Patent Office on September 27, 2020, and the invention name is "Efficient and Low-Latency Automated Assistant Control of Smart Devices". Background Art

[0001] Humans can interact with interactive software applications referred to herein as "automated assistants". For example, humans (who may be referred to as "users" when interacting with automated assistants) can provide input to the automated assistant, causing the automated assistant to generate and provide responsive output to control one or more smart devices, and / or perform one or more other functions. The input provided by the user can be, for example, touch input (e.g., via a touch screen), gestures (e.g., detected via a camera), and / or spoken natural language input (i.e., speech detected via a microphone), which in some cases can be converted to text (or other semantic representations) and then further processed.

[0002] In many cases, an automated assistant includes an automated assistant client that is executed locally by an assistant client device and directly engaged by a user, and a cloud counterpart that leverages the more powerful resources of the cloud to help the automated assistant client respond to user input. For example, the automated assistant client can provide the cloud with an audio recording of the user's spoken words (or a textual conversion thereof), and optionally, data indicating the user's identity (e.g., a certificate). The cloud can perform various processing on the query to return the results to the automated assistant client, which can then provide corresponding output to the user.

[0003] Many users may use multiple client devices to engage with an automated assistant. For example, some users have a coordinated “ecosystem” of client devices, such as smart phones, tablets, vehicle computing systems, wearable computing devices, smart televisions, standalone assistant hub interactive speakers, standalone assistant hub interactive displays with speakers, and other client devices. A user may use any of these client devices to engage in a human-machine conversation with an automated assistant (assuming an automated assistant client is installed). In some cases, these client devices may be dispersed throughout a user's primary residence, secondary residence, workplace, and / or other buildings. For example, mobile client devices such as smart computers, tablets, smart watches, and the like may be on a user's person and / or where the user last placed them. Other client devices, such as traditional desktop computers, smart televisions, and standalone assistant hub devices, may be more stationary, but may still be located in various locations (e.g., rooms) in a user's home or workplace.

[0004] Technology exists that enables a user (e.g., a single user, multiple users in a household, coworkers, cohabitants, etc.) to utilize an automated assistant client of any one of a coordinated ecosystem of client devices to control any one of a plurality of smart devices associated with the automated assistant client. For example, a user may issue a verbal command to the automated assistant client of a client device, "turn off the living room lights," to cause the corresponding smart device (i.e., the light linked to the automated assistant client and labeled as the "living room" light) to turn off.

[0005] When controlling a smart device in response to user input received at a client device, many existing technologies send data corresponding to the user input to a remote automation assistant server via the Internet. The remote automation assistant server: processes the data to determine the smart device to be controlled based on the request; generates a control request for the smart device based on the request; and sends the control request to the server of one or more parties associated with the smart device (e.g., the manufacturer of the smart device) via the Internet. When generating the control request, the remote automation assistant server can process the data (e.g., using a natural language understanding component) to generate a semantic representation of the data, and then generate a control request based on the semantics. The servers of each party receive the control request and then send the corresponding control command to the smart device via the Internet, whether through a hub coexisting with the smart device (e.g., in the case of BLE, Z-Wave, ZigBee, etc.) or directly to the smart device via an IP connection (e.g., in the case of Wi-Fi and other smart devices that do not require a hub).

[0006] However, such techniques present disadvantages such as high latency, overuse of client and / or server resources, and / or overuse of network resources. For example, high latency and / or overuse of server resources may be the result of the remote assistant server needing to process the received audio data and / or text to generate a semantic representation of the corresponding utterance, and then generate a control request based on the generated semantic representation. High latency may additionally or alternatively be the result of sending the request from the remote assistant server to the party servers, which is typically exacerbated by the remote assistant server and the party servers not being geographically close to each other. Similarly, for example, overuse of network resources may be the result of sending high-bandwidth audio data and / or text from a client device to a remote assistant server and / or sending high-bandwidth response text, voice, and / or graphics (e.g., to confirm a requested change to a smart device) from a remote assistant server to a client device. Summary of the invention

[0007] Recent efforts have been made to move the performance of various automated assistant functions from cloud automated assistant components to automated assistant client devices (i.e., client devices that provide automated assistant interfaces). For example, recent efforts have been made to generate speech recognition models that can be used within the constraints of the hardware resources (e.g., memory resources, processor resources, etc.) of various client devices while still achieving good results. Similarly, efforts have been made to move natural language understanding (NLU) and / or fulfillment (e.g., performing actions in response to user input) on the device. Performing these functions on the device can achieve various technical advantages, such as reduced latency (e.g., due to no client-server round trips), reduced network usage (e.g., due to no client-server round trips), and / or operability in low or no connectivity situations.

[0008] However, for many requests, the on-device NLU and / or on-device fulfillment may fail. As a non-limiting example, consider a spoken utterance that is provided to an assistant client device located in a user's living room and includes "make these lights a bit brighter and make them warmer." Generating an appropriate semantic representation of the spoken utterance requires parsing the meanings of "these lights," "brighter," and "warmer." The limited hardware resources of the assistant client device may prevent local storage and / or use of machine learning models, rules, and / or device topologies required to parse the meanings. However, a remote assistant system with more powerful resources than the assistant client device is able to parse these meanings. For example, the remote assistant system may utilize the remotely stored device topology to determine that "these lights" refers to living room lights A, B, and C for a spoken utterance received at the living room client device (e.g., heard loudest and / or most clearly at the living room client device). In addition, the remote assistant system may utilize remote machine learning models and / or rules to determine that "brighter" means "30% (or other percentage or value) brighter than the current brightness intensity level of living room lights A, B, and C." Additionally, the remote assistant system may utilize remote machine learning models and / or rules to determine that “warmer” means “a color temperature that is 20% (or other percentage or value) cooler than the current color temperature.”

[0009] Therefore, due to hardware and / or other constraints of the assistant client device, the remote assistant system may still need to be utilized when parsing various user requests, such as certain requests to control a smart device. However, always providing the unparsed audio data and / or text of the spoken utterance to the remote assistant system for full processing may result in excessive use of resources on the remote assistant system and / or excessive delays in performing corresponding operations (e.g., controlling a smart device).

[0010] Technical Solutions

[0011] Thus, embodiments described herein relate to specific techniques for controlling smart devices that have low latency and / or provide computational efficiency (client and / or server) and / or network efficiency. These embodiments relate to generating and / or utilizing cache entries of a cache stored locally on an assistant client device when controlling various smart devices (e.g., smart lights, smart thermostats, smart plugs, smart appliances, smart routers, etc.). Each cache entry includes a mapping of text to one or more corresponding semantic representations.

[0012] The semantic representation of the cache entry may be a semantic representation previously generated by one or more remote automated assistant servers (also referred to herein as the "remote assistant system") in response to a prior transmission of text and / or corresponding audio data from an assistant client device to a remote assistant system. In response to a cache request sent by the remote assistant system to the client device in response to the prior transmission, the assistant client device may store the cache entry in the cache. The text of the cache entry mapped to the semantic representation of the cache entry may be represented in the cache entry as the text itself and / or a representation thereof, such as its Word2Vec embedding and / or stopword-free variants.

[0013] As described in detail herein, the semantic representation of the cache entry may include a cloud semantic representation and / or a locally interpretable semantic representation. The client device may interpret the locally interpretable semantic representation to generate corresponding control commands, which the client device may send to the local smart device and / or smart device controller via one or more local channels to actuate the corresponding local control smart device. The control commands may be generated to comply with the protocol suite for the local smart device and / or smart device controller. On the other hand, the cloud semantic representation is sent by the client device to the remote assistant system and may be interpreted by the remote assistant system to generate a corresponding control request that is sent to the corresponding smart device control system to actuate the corresponding control smart device.

[0014] A non-limiting example of generating and utilizing cloud semantic representations will now be provided. Assume that a spoken utterance of "dim the lights at touch" is captured in audio data detected via one or more microphones of an assistant client device. The assistant client device may process the audio data using a local speech-to-text model to generate the recognized text "dim the lights at touch". The assistant client device may check its local cache to determine whether there are any cache entries that match the recognized text. The assistant client device may determine that there are no matching cache entries, which may be the result of the initial instance of the spoken utterance at the assistant client device, or due to the fact that a previously matching cache entry was recently cleared from the cache (e.g., due to a change in determining device topology and / or other conditions). The assistant client device may optionally further determine that the recognized text is not interpretable locally using a device-built-in NLU component. For example, the assistant client device may lack a robust local device topology that can parse "light" as a specific light, and / or may lack a robust model and / or rule that can interpret "a little" as a specific degree.

[0015] The client device may further send the recognized text and / or audio data to the remote assistant system. Optionally, sending the text and / or audio data to the remote assistant system is in response to determining that there is no matching cache entry and / or the recognized text is not locally interpretable.

[0016] In response to receiving the recognized text and / or audio data (which the remote assistant system can use to generate the recognized text), the remote assistant system can process the recognized text to generate a cloud semantic representation of the recognized text. For example, the remote assistant system can utilize the account identifier received with the text transmission to identify the corresponding remotely stored device topology for the account identifier. Based on lights A and B mapped as default lights for the assistant client device in the device topology (e.g., through a mapping spoken by the user, or due to being assigned to the same room), the corresponding device topology can be used to resolve "lights" to lights A and B. In addition, for example, the remote assistant system can utilize one or more remote models and / or rules to determine that "dim a bit" means "20% dimmer (or other percentage or value) than the current brightness intensity level." The semantic representation can be generated based on these resolutions and can be, for example, [device = A and B; dimmer, relative intensity is 20%], where "A and B" are unique identifiers of the corresponding lights, and "dimmer, relative intensity is 20%" means that the lights will be dimmed to a degree that is 20% lower than their current intensity.

[0017] Once the cloud semantic representation is generated, the remote assistant system can then generate a control request for implementing the semantic representation. For example, the remote assistant system can identify the "current brightness intensity level" of "lights A and B" based on the most recently reported status of those lights and / or by requesting (and receiving) the current status of those lights from a smart device remote system that controls "lights A and B" (e.g., a smart device remote system controlled by the manufacturer of "lights A and B"). In addition, the remote assistant system can generate a "target brightness intensity level" by multiplying the "current brightness intensity level" by 0.80 (based on determining that "dim a bit" means "20% dimmer than the current brightness intensity level"). A control request can then be generated that identifies "lights A and B" and includes a request to set the lights to the generated "target brightness intensity level". The control request can then be sent to the smart device remote system. In response to receiving the control request, the smart device remote system then sends the corresponding control command to "Lights A and B" via the Internet, whether through a hub that coexists with "Lights A and B" (e.g., in the case of BLE, Z-Wave, ZigBee, etc.) or directly to the smart device via an IP connection (e.g., in the case of Wi-Fi and other smart devices that do not require a hub). The control command may match the control request or may be different from the control request, but may be generated based on the control request. Optionally, the remote assistant system also sends text, audio data, and / or graphics to be rendered to the client device to confirm the change to the state of "Lights A and B". For example, the text "OK, lights A& B dimmed 20%" may be sent to the client device, causing the client device to graphically render the text and / or render the corresponding synthesized speech using a local text-to-speech model.

[0018] In addition, the remote assistant system can send a cache request including a cloud semantic representation, and optionally text (or its representation) and a mapping between the text (or representation) and the cloud semantic representation to the client device. In response to receiving the cache request, the client device can generate a cache entry in a local cache that defines a mapping of the text to the cloud semantic representation. For example, the cache entry can include a mapping of the text itself to the cloud semantic representation, and / or a mapping of a representation of the text (e.g., a Word2Vec representation or other representation) to the cloud semantic representation. In the event that the cache request itself does not include the text (or its representation), the client device can include a mapping based on a cache request received in response to the transmission of the text (or corresponding audio data). In some embodiments, the remote assistant system can also include text, audio data, and / or graphics in the cache request confirming the change in the state of "Lights A and B".

[0019] Assume further that, after the cache entry is generated at the client device, another instance of the spoken utterance "dim the lights a little" is captured in audio data detected via one or more microphones of the assistant client device. The assistant client device may process the audio data using a local speech-to-text model to generate the recognized text "dim the lights a little". The assistant client device may check its local cache and determine that the generated cache entry includes text (or a representation thereof) that matches the recognized text. "Match" as used herein may include an exact match and / or a soft match. For example, in the case of a soft match, the text "dim lights by a touch" may be determined to match "dim the lights at touch" based on an analysis of edit distance, the presence or absence of stop words only, and / or utilizing other techniques. In response to determining the match, the assistant client device may send a cloud semantic representation of the text mapped to the cache entry to the remote assistant system. Optionally, the assistant client device may send the cloud semantic representation without sending the audio data and / or the recognized text to the remote assistant system.

[0020] In response to receiving the cloud semantic representation, the remote assistant system can then generate a control request for implementing the cloud semantic representation. It should be noted that in the example of "dim the lights a little", the control request can vary depending on the "current brightness level" of "lights A and B" at the time of the user input that matches "dim the lights a little". For example, where the control request specifies "target brightness levels" for "lights A and B", those target brightness levels will depend on the "current brightness level". The control request can then be sent to the smart device remote system to result in corresponding control of "lights A and B". Therefore, by sending the semantic representation to the remote assistant system, the remote assistant system can bypass processing the text to generate the semantic representation again - and instead utilize the received cloud semantic representation.

[0021] In these and other ways, less processing occurs at the remote assistant system, which can result in reduced use of power, memory, and / or processor resources at the remote assistant system. This reduction is particularly important given that the remote assistant system receives and processes a large number (e.g., tens of thousands) of smart device control requests every day. In addition, in these and other ways, control requests can be generated and sent more quickly, resulting in faster changes to the state of the smart device. For example, bypassing the process of generating semantic representations from text can save at least 200 ms, 500 ms, 700 ms, or other durations when generating control requests, which is particularly important in controlling various smart devices (e.g., turning on a light in a dark room).

[0022] In embodiments where the cache entry also includes text, audio data, and / or graphics confirming the change to the state of "Lights A and B," the assistant client device may also render such content in response to determining that the cache entry matches the input. The content may be rendered immediately, or, optionally, after a delay that may be optionally specified in the cache entry (e.g., a delay based on the latency between receiving the input and implementing the corresponding state change).

[0023] Note that in the foregoing example, the smart device control request is a state change request relative to the state. That is, the state of the smart device is requested to be changed in a manner that depends on the current state of the smart device. More specifically, the brightness state of lights A and B is requested to be changed relative to the current brightness state of lights A and B. In some embodiments, the remote assistant system may determine to provide a cached request including the semantic representation in place of the control request in response to determining that the semantic representation is a relative state change representation. This determination may be determined based on the semantic representation with reference to the current state of the smart device.

[0024] In some of those embodiments, in response to determining that the semantic representation is not a relative state change representation, the remote assistant system may provide a cached request including a control request in place of the semantic representation. For example, assume that the spoken utterance is "set the lights to 500 lumens". In such an example, a cloud semantic representation identifying "lights A and B" and identifying the "target brightness intensity level" of the two lights as "500 lumens" may be generated. In addition, a corresponding control request may also be generated. Since the cloud semantic representation is not relative, the remote assistant system may optionally send a cached request including a control request. In response, the client device may generate a cache entry that maps the text "set the lights to 500 lumens" (or its representation) to the control request. Thereafter, in response to determining that the received input matches the text of the cache entry, the client device may send the control request to the remote assistant system. The remote assistant system may then send the control request to the smart device remote system, thereby enabling the remote assistant system to directly utilize the control request, thereby bypassing the generation of the semantic representation and the generation of the control request. This may also positively impact the resource usage and / or latency of the remote assistant system.

[0025] It should also be noted that in the assistant client devices of the ecosystem of the user / account, certain cache entries may include mappings of the same text to different semantic representations (or control requests). For example, assume that "dim the lights a little" is received on different assistant client devices, and "lights C and D" are mapped as preferred lights for different assistant client devices in the device topology. In such an example, a semantic representation identifying "lights C and D" would be generated in place of "lights A and B". Therefore, cache entries may be generated for different client devices that include a mapping of "dim the lights a little" to a semantic representation identifying "lights C and D".

[0026] It should also be noted that when spoken utterances are detected at multiple client devices, one or more device arbitration techniques may be utilized to determine that the spoken utterance is received at only one of those client devices. In determining that a client device should be a receiving device based on determining that the client device is closest to the user and / or otherwise intends to interact with the user (although not closest), such techniques may take into account various features. Such features may include the loudness of spoken audio data at the client device, the time at which the audio data was detected at the client device, the most recent interaction with the client device, the proximity of the user to the client device, and / or other features.

[0027] A non-limiting example of generating and utilizing locally interpretable semantic representations will now be provided. Assume again that the spoken utterance "Dim the lights a little" is captured in audio data detected via one or more microphones of an assistant client device. The assistant client device may process the audio data using a local speech-to-text model to generate the recognized text "Dim the lights a little". The assistant client device may check its local cache to determine whether there is a cache entry that matches the recognized text. The assistant client device may determine that there is no matching cache entry, which may be a result of this being the initial instance of the spoken utterance at the assistant client device, or because a previously matching cache entry was recently cleared from the cache. The assistant client device may utilize a device built-in NLU component to optionally further determine that the recognized text is not locally interpretable.

[0028] The client device may further send the recognized text and / or audio data to the remote assistant system. Optionally, sending the text and / or audio data to the remote assistant system is in response to determining that there is no matching cache entry and / or the recognized text is not locally interpretable.

[0029] In response to receiving the recognized text and / or audio data (which the remote assistant system can use to generate the recognized text), the remote assistant system can process the recognized text to generate a locally interpretable semantic representation of the recognized text. For example, the remote assistant system can utilize an account identifier received when sending the text to identify a corresponding remotely stored device topology for the account identifier. Based on Light C being mapped in the device topology as a default light for the assistant client device, the corresponding device topology can be utilized to resolve "Light" to "Light C."

[0030] In addition, for example, the remote assistant system may utilize one or more remote models and / or rules to determine that “make… a little dimmer” means “20% dimmer (or other percentage or value) than the current brightness intensity level”. The remote assistant system may generate a locally interpretable semantic representation by including an identifier for “Light C” and a state change representation, which is a relative phase change representation indicating how to locally resolve the target intensity level for “dimming a little”. For example, the identifier of “Light C” may be the local address of “Light C”. Additionally, for example, the state change representation may be “set intensity = (current intensity * 0.8)”, indicating that the intensity of “Light C” should be set to a value determined by identifying the current intensity value of “Light C” (e.g., the most recently reported intensity, or a locally requested and locally received intensity value) multiplied by 80%. Thus, in some embodiments, the locally interpretable semantic representation may define the state change representation more finely than a cloud-based one, so as to enable resolution of relative values, which may then be utilized to generate local control commands (e.g., utilizing a local adapter as described herein). The state change representation may optionally further include: an indication of an adapter to be utilized when generating a corresponding local control command (e.g., the adapter is specific to the manufacturer of the smart device); an indication of a channel to be utilized to send the local control command (e.g., Bluetooth or Wi-Fi); and / or an indication of a protocol suite to be utilized to generate the control command based on the local interpretable semantic representation. The local interpretable semantic representation may be, for example, [Device = C; Set Strength = (Current Strength * 0.8)].

[0031] In some embodiments, the system may determine to generate a locally interpretable semantic representation in response to: determining that the client device (or an additional client device locally connected to the client device) is able to locally control a smart device referenced in the recognized text; determining that the smart device referenced in the recognized text is capable of being locally controlled; and / or determining that a state change requested in the recognized text is capable of local control (e.g., where some states can be changed through local control, while other states can only be changed through cloud control).

[0032] Once the local interpretable semantic representation is generated, the remote assistant system may optionally send the local interpretable semantic representation to the client device for immediate implementation. In other words, the remote assistant system may send the local interpretable semantic representation so that the client device interprets the local interpretable semantic representation in response to the spoken utterance, generates a corresponding local control command, and locally sends the local control command to implement the state change of "Light C". Sending the local interpretable semantic representation to the client device for immediate implementation may be sending the cache request itself (as described below) or a separate sending. Alternatively, the remote assistant system may alternatively generate a corresponding cloud control request and send it to the smart device control system for "Light C" so that the smart device control system generates a corresponding control command and sends it to "Light C" to implement the state change.

[0033] Regardless of the manner in which the state change is achieved in response to an immediate request, the remote assistant system may send a cache request including a local interpretable semantic representation and optionally text (or its representation) and a mapping between the text (or its representation) and the local interpretable semantic representation to the client device. In response to receiving the cache request, the client device may generate a cache entry in the local cache that defines a mapping of the text to the local interpretable semantic representation. For example, the cache entry may include a mapping of the text itself to the local interpretable semantic representation, and / or a mapping of the representation of the text to the local interpretable semantic representation. In the event that the cache request itself does not include the text (or its representation), the client device may include a mapping based on the cache request received in response to the sending of the text (or corresponding audio data). In some embodiments, the remote assistant system may also include text, audio data, and / or graphics in the cache request confirming the change in the state of "Light C".

[0034] Assume further that after the cache entry is generated at the client device, another instance of the spoken utterance "Dim the lights a little" is captured in audio data detected via one or more microphones of the assistant client device. The assistant client device may process the audio data using a local speech-to-text model to generate the recognized text of "Dim the lights a little." The assistant client device may check its local cache and determine that the generated cache entry includes text (or a representation thereof) that matches the recognized text. Exact matches and / or soft matches may be utilized. In response, the assistant client device may process the local interpretable semantic representation of the cache entry to generate a corresponding local control request for local transmission to "Light C." For example, based on a local interpretable semantic representation including "Set intensity = (current intensity * 0.8)," the assistant client device may locally identify the "current intensity" of "Light C" and then determine the value of the "set intensity" by multiplying the current intensity by 0.8. Therefore, setting the "intensity" to the determined value may be a specific state change generated based on a relative state change of the local interpretable semantic representation.

[0035] In addition, the assistant client device can generate control commands based on the value of "set intensity" and optionally utilizing one or more local adapters. The control commands can be generated to conform to a protocol suite that complies with "Light C", such as BLE, Z-Wave, Zigbee, Wi-Fi, or other protocol suites. The protocol suite used in generating the control commands can be selected based on an identifier of the smart device included in the locally interpretable semantic representation and / or based on an indication of an adapter that can be included in the locally interpretable semantic representation.

[0036] In addition, the generated control command can be sent locally to "Light C" (or a corresponding local device that controls "Light C"). The control command can be sent to "Light C" (or a corresponding local device) based on a local interpretable semantic representation that identifies "Light C" (e.g., an address that identifies "Light C"), and can be sent via a local channel, which can be identified via a local interpretable semantic representation. The local channel can be, for example, a Wi-Fi radio channel, a Bluetooth radio channel, or other channel. Therefore, a state change can be achieved at "Light C" without sending any data to the remote assistant system again and / or without the remote assistant system generating a semantic representation again. This can save network resources by avoiding sending audio data and / or recognized text to the remote assistant system, and can save processing resources at the remote assistant system by avoiding the need to generate a semantic representation again at the remote system. In addition, bypassing sending to the remote assistant system and processing at the remote assistant system can significantly reduce the waiting time for implementing the requested state change at the smart device.

[0037] As described above, various embodiments utilize adapters to locally interpret a local interpretable semantic representation or a specific state change generated thereon as a corresponding control command. In those embodiments, the client device may include one or more adapters, each of which may be customized for a specific smart device, a specific smart device manufacturer, and / or a specific protocol suite. When the adapter is executed by the assistant client device, it may process the local interpretable semantic representation (and / or a specific state change generated based on the local interpretable semantic representation) to generate a specific control command, which, when sent locally to at least one corresponding smart device (or a hub / bridge or other component connected to a smart device), is customized to be directly interpretable by the corresponding smart device (or other components connected to the smart device) to implement a state change at the corresponding smart device. For example, the adapter may be implemented by JavaScript (or other interpreted programming languages), and may convert a general smart device control command generated based on a local interpretable semantic representation into a specific control command that complies with the protocol suite of the smart device. The adapter may optionally be executed in a container within an automated assistant client. In embodiments where multiple adapters are available at the assistant client device, a particular adapter may be selected for the locally interpretable semantic representation based on an identifier of the smart device included in the semantic representation and / or based on an indication of the adapter that may be included in the semantic representation.

[0038] In embodiments where the cache entry also includes text, audio data, and / or graphics confirming the change to the state of "Lights A and B," the assistant client device may also render such content in response to determining that the cache entry matches the input. The content may be rendered immediately, or optionally after a delay.

[0039] Note that in the previous example, the smart device control request is a state change request relative to the state. That is, the state of the smart device is requested to be changed in a manner that depends on the current state of the smart device. More specifically, the brightness state of lamp C is requested to be changed in a manner relative to the current brightness state of lamp C. In some embodiments, the remote assistant system may determine, in response to determining that the semantic representation is a relative state change representation, to provide a cached request including a locally interpretable semantic representation instead of a local control command. This determination may be determined based on a semantic representation that references the current state of the smart device.

[0040] In some of those embodiments, in response to determining that the semantic representation is not a relative state change representation, the remote assistant system may provide a cached request including a local control command in place of the semantic representation. For example, assume the spoken utterance is "Set the light to 500 lumens." In such an example, a locally interpretable semantic representation can be generated that identifies "Light C" and identifies the "target brightness intensity level" of the light as "500 lumens." In addition, a corresponding control request can also be generated. Since the semantic representation is not relative, the remote assistant system may optionally send a cached request including a local control command generated based on the locally interpretable semantic representation. In response, the client device may generate a cache entry that maps the text "Set the light to 500 lumens" (or its representation) to a control command. Thereafter, in response to determining that the received input matches the text of the cache entry, the client device may locally send the control command of the cache entry, thereby bypassing the generation of a control command based on the locally interpretable semantic representation.

[0041] It should also be noted that, similar to the cloud semantic representation, in the assistant client device of the user / account's ecosystem, some cache entries may include mappings of the same text to different locally interpretable semantic representations (or local control commands).

[0042] Separate examples of cloud semantic representations and locally interpretable semantic representations are provided above. However, in some embodiments, a single cache entry may include a mapping of text to both a locally interpretable semantic representation and a cloud semantic representation. For example, both lights "A" and "B" may be identified for a request, and light "A" may be controlled locally, but light "B" may only be controlled via cloud control. As a result, the remote assistant system may generate and provide a cache request that causes the text for the request to be mapped to both: (a) a locally interpretable semantic representation, which may be processed locally to effect a state change for "Light A"; and (b) a cloud semantic representation, which may be sent to the remote assistant system to effect a state change for "Light B".

[0043] In various embodiments, the remote assistant system and / or client device may determine a change in device topology in response to the occurrence of an account and / or other condition, and as a result, cause one or more cache entries of a cache of a client device of the ecosystem to be cleared (e.g., completely removed or marked as stale). The change in device topology may include, for example, adding or removing smart devices in the device topology, renaming smart devices and / or assistant client devices in the device topology, assigning devices to new rooms and / or groups in the device topology, and / or other changes to the device topology. Such a change may affect some or all semantic representations of the cache, and as a result, in response to detecting the change, at least the affected cache entries (and optionally all cache entries) may be cleared from the cache. As an example, assume that an additional light is added to the device topology and is assigned to a room with a "kitchen" tag. Upon completion of the addition, any semantic cache entries related to those lights previously assigned to "kitchen lights" will now be invalid because they will not reflect the newly added additional lights. Therefore, in response to detecting the change, at least those cache entries related to "kitchen lights" may be cleared from the cache. For example, the remote assistant system may determine such a change and therefore send a cache purge request to all assistant devices of the ecosystem to cause the local caches of all assistant client devices to be purged. Thereafter, when spoken utterances, typed inputs, or other inputs are received, new cache entries will be generated at the assistant client devices. For example, if the cache entry for the text "dim the kitchen lights" is purged, then if the user thereafter provides the spoken utterance "dim the kitchen lights", a new cache entry will be created.

[0044] In some embodiments, the assistant client device can utilize a device-built-in speech-to-text model to at least selectively process the audio data without any detection of an explicit automated assistant invocation at the assistant client device. An explicit invocation prompt is a prompt that, when detected alone, will always result in at least activation of the device-built-in speech recognition. Some non-limiting examples of explicit invocation prompts include detection of a spoken hotword with at least a threshold confidence level, launching an explicit assistant interface element (e.g., a hardware button or a graphical button on a touch screen display), a "phone squeeze" with at least a threshold intensity (e.g., as detected by a sensor in a phone bezel), and / or other explicit invocation prompts.

[0045] As an example, the assistant client device may process audio data without any detection of explicit automated assistant invocation when human presence is detected near the device (e.g., based on a passive infrared sensor and / or other sensor), in response to detecting any sound activity (e.g., using a sound activity detector), in response to detecting at least a threshold level of noise near the device, during certain days and / or times, and / or in response to other conditions. In an embodiment where the assistant client device processes audio data using a device built-in speech-to-text model without any detection of explicit automated assistant invocation, the assistant client device may determine whether any recognized text matches any of those "hot phrases" indicated as being on the assistant client device. Each "hot phrase" includes a sequence of one or more words. When text recognized from the built-in speech recognition of the audio data is determined to not match a hot phrase, the assistant client device may discard the recognized text and audio data. When text recognized from the built-in device speech recognition of the audio data is determined to match a hot phrase, the assistant client device may automatically initiate one or more corresponding actions based on the text. The user may be provided with control over which phrases are indicated as hot phrases, and optionally, explicit user acceptance may be required before a phrase is indicated as a hot phrase.

[0046] In some embodiments, a hot phrase can be a phrase that matches the text of a corresponding cache entry as described herein. In those embodiments, an action automatically initiated by an assistant client device based on a hot phrase can include sending a corresponding cloud semantic representation or a local interpretation of a corresponding locally interpretable semantic representation. In some of those embodiments, the text of a cache entry can be designated as a hot phrase for the assistant client device at the assistant client device based on a determination that one or more criteria are met. The criteria can include, for example, that the text and / or matching text has been determined to be at least a threshold number of times (over or within the entire recent duration) and / or that it exists (overall or in its entirety) at least at a threshold frequency (overall or in its entirety) in user input (e.g., typed and / or spoken) received at the assistant client device. In some versions of those embodiments, the text of a cache entry can be automatically designated as a hot phrase, optionally confirming such a designation provided via a user interface input (e.g., "By the way, I've designated X as a hot phrase. You can easily change this in your settings"). In some other versions, text may be designated as a hot phrase only after confirmation user input in response to a prompt. For example, the confirmation input in response to the prompt "you askto 'turn on the lights often', want me to designate it as a hot phrase so youno longer have to say 'OK assistant' first?" is "yes." In other embodiments, the text of any local cache entry in the cache of an assistant client device may be automatically designated as a hot phrase for the assistant client device. In embodiments where the text of a cache entry is designated as a hot phrase, the duration of a user's interaction with an assistant client device may be shortened by avoiding the need for the user to first provide an explicit invocation, such as invoking a hot word (e.g., "OK assistant"). By shortening the duration of the interaction with the assistant client device, the human-assistant client device interaction is improved, thereby enabling a greater number of smart device state changes to be achieved in a shorter duration.

[0047] The above description is only provided as an overview of some embodiments of the present disclosure. Further descriptions of those embodiments and other embodiments are described in more detail herein.

[0048] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause the execution of any of the methods described herein. Some embodiments include a client device having one or more processors that execute locally stored instructions and interact with locally stored data to perform one or more methods described herein. Some embodiments also include one or more computer-readable storage media (transitory or non-transitory) storing computer instructions that can be executed by one or more processors to implement any of the methods described herein.

[0049] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a block diagram of an exemplary environment in which embodiments disclosed herein may be implemented.

[0051] Figure 2 An exemplary environment including multiple smart devices from multiple different parties, and including multiple assistant client devices is shown.

[0052] Figure 3A shows how to handle the situation when the assistant client device lacks a cache entry that matches the text of the spoken utterance. Figure 2 An example of spoken utterance received at an assistant client device.

[0053] Figure 3B Shown in Figure 3A How to handle differently after storing cache entries with cloud semantic representations at assistant client devices Figure 3A received at the same client device Figure 3A Examples of the same spoken utterance.

[0054] Figure 4A shows how to handle the situation when the backup assistant client device lacks a cache entry that matches the text of the spoken utterance. Figure 2 The backup assistant client device receives Figure 3A and Figure 3B Examples of the same spoken utterance.

[0055] Figure 4B Shown in Figure 4A How to handle differently after storing cache entries with cloud semantic representations at alternate client devices Figure 4Areceived at the alternate client device Figure 4A Examples of the same spoken utterance.

[0056] Figure 5A shows how to handle the situation when the assistant client device lacks a cache entry that matches the text of the spoken utterance. Figure 3A and 3B An example of another spoken utterance received at the same assistant client device.

[0057] Figure 5B Shown in Figure 5A After storing cache entries with locally interpretable semantic representations at the client device of Figure 5A Received at the client device Figure 5A Examples of the same spoken utterance.

[0058] Figure 6 is a flow chart illustrating an exemplary method that may be implemented by an assistant client device according to various embodiments disclosed herein.

[0059] Figure 7 is a flow chart illustrating another exemplary method that may be implemented by an assistant client device in accordance with various embodiments disclosed herein.

[0060] Figure 8 is a flow chart illustrating an exemplary method that may be implemented by a remote assistant system according to various embodiments disclosed herein.

[0061] Fig. 9 An exemplary architecture of a computing device is shown. DETAILED DESCRIPTION

[0062] Smart connected devices (also referred to herein as smart devices or Internet of Things (IoT) devices) are proliferating, such as smart home alarms, smart door locks, smart cameras, smart lights, smart thermostats, smart scales, smart beds, smart irrigation systems, smart garage door openers, smart plugs, smart appliances, smart baby monitors, smart televisions (TVs), smart fire alarms, smart moisture detectors, smart routers, etc. Often multiple smart devices are located within a structure, such as a home - or within multiple related structures, such as a user's primary residence and the user's secondary residence and / or place of work.

[0063] In addition, there is a proliferation of assistant client devices that can each include an assistant client that can optionally interact with one or more remote automated assistant components to form a logical instance of an automated assistant. Assistant client devices can be dedicated to assistant functions (e.g., a standalone speaker and / or standalone audio / video device that includes only an assistant client and related interfaces and is dedicated to assistant functions), or can perform auxiliary functions in addition to other functions (e.g., a mobile phone or tablet that includes an assistant client as one of multiple applications). In addition, certain smart devices can also be assistant client devices. For example, some smart devices can include an assistant client and a speaker and / or microphone that at least (at least in part) serves as a user interface output and / or input device for an assistant interface for the assistant client.

[0064] Various techniques have been proposed for associating smart devices with corresponding logical instances of an automated assistant (and optionally individual assistant client devices). For example, a user, a group of users, an assistant client device, and / or a group of assistant client devices (e.g., all within a certain structure) can be linked (e.g., in one or more databases) with a plurality of different smart devices to enable interaction with the smart devices (e.g., controlling the smart devices) via the automated assistant. For example, each of a plurality of assistant client devices in a home can be linked to each of a plurality of different smart devices in the home to enable any user (or a restricted group of users) to connect with any one of the assistant client devices to interact with any one of the plurality of different smart devices.

[0065] This linking of smart devices and assistant client devices is referred to herein as a device topology and can be represented by various data structures. The device topology can be user-created and / or automatically created and can define various assistant client devices, various smart devices, identifiers of each, and / or properties of each. For example, the identifier of a device can specify the room (and / or other area) of the structure in which the device is located (e.g., living room, kitchen) and / or can specify a nickname and / or alias for the device (e.g., couch lamp, front door lock, bedroom speaker, kitchen assistant, etc.). In this way, the identifier of a device can be the name, alias, and / or location of a corresponding device that a user may associate with the corresponding device.

[0066] The device topology representation may further specify one or more device properties associated with the respective device. The device properties of the assistant client device may include, for example, one or more input and / or output modalities supported by the assistant client device and / or a preferred smart device to be controlled by the assistant client device (e.g., ambiguous smart TV commands received at assistant client device 1 should be assumed to be directed to smart TV1, but they may be assumed to be directed to smart TV2 for assistant client device 2). For example, device properties for an independent speaker-only assistant client device may indicate that it is capable of providing audible output but not visual output. Similarly, for example, device properties for the same independent speaker-only assistant client device may indicate that lighting control requests that do not specify a specific light should be interpreted as relating to lights assigned to the same room as the assistant client device. The device properties of a smart device may, for example, identify one or more states of a smart device that can be controlled, identify a party (e.g., 3P) that manufactures, distributes, and / or creates firmware for a smart device; and / or identify a unique identifier for a smart device, such as an address of the smart device and / or a fixed identifier provided by 1P or 3P. According to various embodiments disclosed herein, the device topology representation may optionally further specify: which smart devices can be locally controlled by which assistant client devices, local addresses for locally controllable smart devices (or local addresses for hubs that can directly locally control these smart devices), local signal strength between assistant client devices and smart devices, and / or other preference indicators. In addition, according to various embodiments disclosed herein, the device topology representation (or variations thereof) may be stored remotely at a remote assistant system and / or locally at each of multiple assistant client devices used in locally controlling smart devices. Optionally, each locally stored device topology may be less robust (e.g., defining fewer devices, attributes, and / or identifiers) than a remotely stored device topology due to, for example, assistant client device hardware constraints.

[0067] Now go to Figure 1 , shows an exemplary environment in which the technology disclosed herein can be implemented. The exemplary environment includes multiple assistant client devices 110 1-N (also referred to herein as “client device”), cloud automated assistant component 120, smart device system 140 A-N 、Smart devices 145 A-N and for client device 110 1-N Device topology 152. Figure 1 Client device 110 1-N and smart devices145 1-N The client devices and smart devices are represented as being at least selectively associated with each other (eg, via the device topology 152). For example, the smart device 145 1-NThe client device 110 may be located entirely within the home (eg, inside and / or outside the home). 1-N can be in the same home at least occasionally, and smart devices 145 1-N and client device 110 1-N One or more technologies (such as those described herein) may be used to link with each other. Through such linking, the client device 110 may be used according to the embodiments described herein. 1-N To control smart devices145 1-N .

[0068] Client device 110 1-N One or more (e.g., all) of the client devices 110 may execute corresponding instances of the automated assistant client. However, in some embodiments, the client device 110 1-N One or more of the above may optionally lack an instance of an automated assistant client and still include an engine and hardware components for controlling one or more smart devices. An instance of an automated assistant client may be a client device 110 that is associated with the corresponding client device 110. 1-N The operating system may be separate from the application (eg, installed "on top" of the operating system), or may alternatively be accessed through the corresponding client device 110 1-N Each instance of the automated assistant client can respond to a request made by a user via the client device 110 1-N The client device 110 may provide various requests to any of the I / O components 111, optionally interacting with the cloud automation assistant component 120. 1-N Other engines may optionally interact with the cloud automated assistant component 120 .

[0069] Client device 110 1-N One or more (eg, all) of the above may include a corresponding speech-to-text (STT) engine 112 1-N , which utilizes the corresponding locally stored STT model 152 1-N To process the audio data of the captured spoken utterance to generate corresponding recognized text of the spoken utterance. 1-N In some embodiments, the STT engine 112 detects each instance of audio data. 1-N Only on the corresponding client device 110 1-N In some embodiments, the STT engine 112 processes the audio data after detecting an explicit automated assistant call. 1-NThe audio data may be at least selectively processed, even in the absence of an explicit automated assistant invocation, but any audio data and / or recognized text determined not to include any "hot phrases" as described herein may be discarded (without further processing). In the event that a "hot phrase" is detected, one or more responsive actions may be taken based on the "hot phrase". For example, in the event that a "hot phrase" is matched to a corresponding local cache 158 1-N In the case where the text of a cache entry matches, the semantic representation of the cache entry can be further processed as described herein.

[0070] Client device 110 1-N One or more (eg, all) of the above may also optionally include a corresponding natural language understanding (NLU) engine 113 1-N , which uses the corresponding locally stored NLU model 153 1-N The text may be processed by the NLU engine 113 to attempt to locally generate a local semantic representation of the text. The text may be recognized text from the STT engine 112, or may be typed text (e.g., entered via a virtual keyboard of a touch screen), or may be text associated with a selected virtual and / or hardware button (e.g., selected text mapped to a particular virtual button displayed on a touch screen). 1-N may be able to generate valid semantic representations for various text phrases. However, as described herein, the NLU engine 113 1-N It may also be impossible to generate valid semantic representations for various other text phrases, such as various text phrases related to control of smart devices (eg, relative control of smart devices).

[0071] One or more (eg, all) client devices 110 1-N A corresponding fulfillment engine 114 may also be included 1-N Each fulfillment engine 114 1-N A corresponding cloud module 115 may be included 1-N , corresponding local module 116 1-N and / or a corresponding text-to-speech (TTS) engine 117 1-N .

[0072] Cloud Module 115 1-N Various requests may be fulfilled through interaction with the cloud automated assistant component 120. For example, the cloud module 1151 may respond to determining the text of the received input (e.g., the recognized text STT engine 112 1-N) matches the text of the cache entry of the local cache 1581, and sends the cloud semantic representation of the text mapped to the cache entry to the cloud automated assistant component 120, as described herein. The cloud module 1151 may also optionally send the cloud semantic representation generated by the NLU engine 1131 to the cloud automated assistant component 120. The cloud fulfillment engine 127 of the cloud automated assistant component 120 may then utilize the received cloud semantic representation in fulfilling the received input. For example, for a smart device control input, the cloud fulfillment engine 127 may utilize the cloud semantic representation to generate a corresponding control request to be sent to the corresponding smart device system 140. A-N Then, the smart device system 140 A-N Generate corresponding control commands and send them to corresponding smart devices 145 A-N .

[0073] Cloud Module 115 1-N You can also target the 1-N Resolve and / or not with respective local cache 158 1-N , sending data representing the received input to the cloud-based automated assistant component 120. For example, if the received input is a spoken utterance captured in audio data, the audio data and / or (e.g., from the corresponding STT engine 112) may be sent to the cloud-based automated assistant component 120. 1-N ) The locally recognized text is sent to the cloud-based automated assistant component 120. In response to at least some of those sent, the cloud-based component 120 may send the recognized text to the corresponding cache engine 118 1-N The cached request is provided to enable future inputs to be fulfilled more efficiently and / or with reduced latency. In addition, the cloud-based component 120 can additionally or alternatively respond to the received input by performing corresponding cloud fulfillment and / or providing the corresponding client device 110 with the corresponding cloud fulfillment. 1-N Providing locally interpretable semantic representations and / or local commands to be used by respective client devices 110 1-N Used in performing local execution.

[0074] Local Module 116 1-N Various requests may be fulfilled locally without interacting with the cloud automated assistant component 120. For example, the local module 1161 may respond to determining the text of the received input (e.g., from the STT engine 112 1-NThe recognized text) is matched with the text of the cache entry of the local cache 1581, and the local interpretable semantic representation of the text mapped to the cache entry is utilized when fulfilling the request locally. For example, the local module 1161 can utilize the local interpretable semantic representation when generating a local control command to be sent locally to one or more corresponding smart devices 145 via the corresponding radio 1191. A-N Optionally, and as described herein, when the local interpretable semantic representation is used for smart device control, the local module 116 1-N One or more corresponding adapters 156 may be used when generating local control commands. 1-N The local module 1161 may also optionally locally interpret any locally interpretable semantic representations generated by the NLU engine 1131 .

[0075] Text-to-speech (TTS) engine 117 1-N The TTS engine 1171 may optionally be used to generate synthesized speech to provide a response to the request. For example, the TTS engine 1171 may be responsive to determining the text of the received input (e.g., from the STT engine 112 1-N The method further comprises: matching the recognized text) with the text of the cache entry of the local cache 1581, generating corresponding synthesized speech using the response text mapped to the text in the cache entry, and rendering the synthesized speech in response to the received input. The synthesized speech can be rendered in addition to other actions performed based on the semantic representation (cloud and / or locally interpretable) of the text also mapped to the cache entry. In addition, the response text can be visually rendered in addition or alternatively, and / or the alternative content can be rendered in addition or alternatively by the fulfillment engine in response to the alternative content being mapped to the text in the cache entry.

[0076] Client device 110 1-N One or more (eg, all) of the above-mentioned cache engines 118 may also optionally include the above-mentioned corresponding cache engine 118 1-N and corresponding local cache 158 1-N . Cache Engine 118 1-N The cache generation engine 125 of the cloud automation assistant component 120 may receive respective cache requests and store the corresponding cache entries in the corresponding local cache 158. 1-N In. Local cache 158 1-N may be stored on, for example, the corresponding client device 110 1-N As described herein, in some embodiments, a cache request received from the cache generation engine 125 includes storing the cached data in the corresponding local cache 158. 1-NIn other embodiments, the cache request may lack text to be included in the cache entry and cannot be mapped to the semantic representation in the cache entry. In such embodiments, the corresponding cache engine 118 1-N The text may be determined based on it being text most recently provided to the cloud automated assistant component 120 (or recognized text of audio data most recently provided to the cloud automated assistant component 120 ).

[0077] can be obtained through the Figure 1 The one or more wide area networks (eg, the Internet) indicated at 1051 are communicatively coupled to the client device 110 1-N The cloud automated assistant component 120 is implemented on one or more computing systems (collectively referred to as a "cloud" or "remote" assistant system). For example, the cloud automated assistant component 120 can be implemented by one or more high-performance server clusters. 1-N One or more local area networks may be utilized in accessing the wide area network 1051 and / or communicating locally with each other. Such a local area network may include client devices 110 1-N between Wi-Fi networks and / or mesh networks.

[0078] The cloud automation assistant component 120 can also communicate with the smart device system 140 via one or more wide area networks. A-N The communication coupling between the cloud automation assistant component 120 and the smart device system 140 is usually composed of Figure 1 In addition, the smart device system 140 can each be connected via a Figure 1 The one or more wide area networks indicated at 1104 may be communicatively coupled to one or more smart devices 145 A-N Note that smart device 145 A-N One or more local area networks may be utilized when accessing wide area network 1053 .

[0079] Each smart device system 140 A-N The systems may be first party (1P - i.e., manufactured and / or distributed by the same party that controls the automated assistant) or third party (3P - i.e., manufactured and / or distributed by a different party) systems, and each may communicate with one or more corresponding smart devices 145 A-N For example, the first smart device system 140 A-N Can be controlled by the first 3P and can communicate with the first smart device 145 A1 The second smart device system 140 can be controlled by the second 3P and can communicate with the second smart device 145 B1 and the third smart device 145 B2 Equal coupling.

[0080] Smart device system 140 A-N Can communicate with device 145 via WAN 1053 A-N Communicate to control their corresponding smart devices 145 A-N to deliver firmware updates to their corresponding smart devices 145 A-N to determine their corresponding smart devices 145 A-N For example, a given smart device system 140 can communicate with smart device 145 A-N The smart device is communicated with the mobile application to control the smart device in response to receiving user input via the mobile application for the smart device system to initiate control of the smart device.

[0081] In addition, for example, one of the given smart device systems 140 may respond to a request from the cloud automated assistant component 120 to communicate with the smart device 145 A-N For example, according to some techniques, a user may provide a request to control a smart device via one or more I / O components 1111 of the client device 1101, such as a verbal input of "turn off the couch light" provided via a microphone of the I / O component 1111. The request (e.g., audio data capturing the verbal input or locally generated text) may be sent by the client device 1101 to the cloud automation assistant component 120 via the wide area network 1051. The cloud automation assistant component 120 may process the request to determine the smart device to be controlled based on the request, and send the control request to the smart device system 140 via the wide area network 1052. A-N The smart device system then sends the corresponding command to the smart device via the wide area network 1053. However, as described herein, such technology has disadvantages such as high latency and / or excessive consumption of network resources.

[0082] In some implementations, multiple client computing devices 110 1-N and smart devices145 A-N The client devices 110 may be associated with each other in various ways to facilitate the implementation of the techniques described herein. 1-N and smart devices145 A-N The client computing devices 110 may be associated with each other by being communicatively coupled via one or more LANs and / or via one or more peer-to-peer networks. For example, multiple client computing devices 110 may be deployed across a particular area or environment such as a home, a building, etc. 1-N Additionally or alternatively, in some embodiments, multiple client devices 110 1-N and smart devices145A-N It may be that they are client devices 110 that are at least selectively accessible by one or more users (eg, individuals, families, employees of an organization, other predefined groups, etc.). 1-N and smart devices145 A-N In some of those embodiments, the client device 110 1-N and smart devices145 1-N The ecosystems may be manually and / or automatically associated with each other in the device topology 152 .

[0083] Client device 110 1-N The instance of the automated assistant client of the cloud automated assistant component 120 can form a logical instance of an automated assistant with which the user can have a human-computer dialogue from the user's perspective through its interaction with one or more cloud automated assistant components 120. For example, the user can use any client device 1101 and the automated assistant client 1171 or use the client device 110 N and automated assistant client 117 N to participate in the same logical instance of the automated assistant. Although the automated assistant clients 1171 and 117 N The specific instance of may vary (e.g., providing different smart device controls for the same command) and / or may be provided via different I / O components 1111 and 111 N Providing user interface output and / or via different I / O components 1111 and 111 N Accept different user interface inputs (for example, I / O component 1111 may include a touch screen, and I / O component 111 N 10. Although the touch screen is not included), the user can still effectively participate in the same logical instance of the automated assistant. For the sake of brevity and simplicity, as used herein, the term "automated assistant" will refer to the automated assistant client executing on the client device 110 and, optionally, to one or more cloud automated assistant components 120 (which may be shared among multiple automated assistant clients). Figure 1 Two client devices 1101 and 110 of the collaborative ecosystem are shown in FIG. N , but it should be understood that many additional client devices may be included in the ecosystem. In addition, it should be understood that separate coordinated ecosystems of client devices will also be provided, each of which is associated with a different user (e.g., account) and / or environment, and such separate coordinated ecosystems may also interact with the cloud automated assistant component 120 (but in an interaction customized for the accounts of those separate ecosystems).

[0084] Client device 110 1-NThe client computing device may include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone assistant-centric interactive speaker, a stand-alone assistant-centric interactive display with a speaker, a smart appliance such as a smart TV, and / or a wearable device of a user including a computing device (e.g., a watch of a user with a computing device, glasses of a user with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided.

[0085] As described above, one or more client devices 110 1-N The cloud automation assistant component 120 may be at least selectively connected in processing input and / or generating output based on input and / or generating smart device control commands based on input. The cloud automation assistant component 120 may include an STT engine 121, an NLU engine 122, a TTS engine 123, a cache generation engine 125, and / or a cloud fulfillment engine 127.

[0086] As described above, for the client device 110 1-N received at and cannot be processed by the corresponding NLU engine 113 1-N Parsing and / or caching in the corresponding local cache 158 1-N The text and / or audio data corresponding to the request may be sent to the cloud automated assistant component 120 for a request that lacks a matching cache entry. The cloud automated assistant component 120 may utilize its more robust NLU engine 122 to generate a semantic representation based on processing the text for the request. The processed text may be sent by the client device, or may be recognized text generated by the STT engine 121 using audio data sent by the client device. As described herein, the generated semantic representation may include a cloud semantic representation and / or a locally interpretable semantic representation. Also as described herein, when generating the semantic representation, the NLU engine 122 may utilize a local interpretable semantic representation for the client device 110. 1-N and smart devices145 A-N Device topology 152.

[0087] The cache generation engine 125 may respond to a request from the client device 110 1-NThe cache generation engine 125 generates a cache request by sending one of the cache generation engines 125, the cache request including a cache entry, the cache entry including the generated semantic representation, and optionally including text (or its representation) and a mapping of the text to the semantic representation. The cache entry of the generated cache request may include additional and / or alternative content, such as response content to be rendered, protocol suites, adapters, and / or radio to be utilized with the locally interpretable semantic representation, and / or other content. The response content to be rendered may include text, audio data (e.g., synthesized speech generated using the TTS engine 123 for the response text) and / or graphics. The cache generation engine 125 sends the corresponding cache request to the corresponding client device 110. 1-N The corresponding cache engine 118 1-N , so that the cache entry is stored in the corresponding local cache 158 1-N middle.

[0088] When the generated semantic representation includes a cloud semantic representation, the cloud fulfillment engine 127 may process the cloud semantic representation to generate a corresponding control request, which is then sent to the corresponding smart device system 140. A-N . Intelligent device system 140 A-N In response to the control request, a corresponding control command may be generated and sent to the corresponding smart device 145 A-N .

[0089] When the generated semantic representation includes a locally interpretable semantic representation, the corresponding client device 110 A-N The corresponding local control commands may be generated and sent locally (eg, using respective local modules 116 A-N ) utilizes the locally interpretable semantic interpretation of the cached request. Alternatively, the cloud automated assistant component 120 may provide the locally interpretable semantic representation in a separate transmission, and the corresponding client device 110 A-N Corresponding local control commands may be generated and those local control commands may be sent locally (eg, using corresponding local modules 116 A-N As another alternative, the cloud automation assistant component 120 itself can generate local control commands and send the local control commands to the corresponding client device 110. A-N , to send those control commands locally.

[0090] As described above, for the client device 110 1-N received at the corresponding local cache 158 1-NThe cloud semantic representation may be sent to the cloud automated assistant component 120 (e.g., in place of the corresponding text and / or audio data) if the request matches a cache entry that includes the cloud semantic representation in the cloud. The cloud fulfillment engine 127 may process the received cloud semantic representation to generate a corresponding control request, which is then sent to the corresponding smart device system 140. A-N . Intelligent device system 140 A-N In response to the control request, a corresponding control command may be generated and sent to the corresponding smart device 145 A-N .

[0091] Now, with reference to the attached drawings, there is provided Figure 1 Additional description of the individual components. Figure 2 A home floor plan is depicted that includes a plurality of rooms 250, 252, 254, 256, 258, 260, and 262. A plurality of client devices 110 are deployed in at least some of the rooms. 1-3 Each client device 110 1-3 An instance of an automated assistant client configured with selected aspects of the present disclosure may optionally be implemented, and may optionally include one or more input devices, such as microphones, touch screens, etc., and / or one or more output devices, such as speakers, displays, etc. For example, a first client device 1101 in the form of an interactive stand-alone speaker is deployed in room 254, which is a bedroom in this example. A second client device 1102 in the form of an independent interactive speaker and display device (e.g., a display screen, a projector, etc.) is deployed in room 252, which is a living room in this example. A third client device 1103, also in the form of an interactive stand-alone speaker, is deployed in room 256.

[0092] Multiple client devices 110 1-3 The wireless router 101 and / or the local mesh network depicted in the room 252 may be communicatively connected to each other and / or other resources (e.g., smart devices and the Internet). In addition, there may be other client devices, particularly mobile devices such as smartphones, tablets, laptops, wearable devices, etc., which are carried by one or more people (e.g., user 103) in the home and may or may not be connected to the same local area network. It should be understood that Figure 2 The configuration of client devices depicted in is merely one example; more or fewer and / or different client devices may be deployed across multiple other rooms and / or areas outside of the home.

[0093] exist Figure 2 The smart device includes a first remote smart device system 140 AControlled smart lights 145 A1 The smart device further includes a second remote smart device system 140 B Controlled smart lights 145 B1 and 145 B2 . Second remote intelligent device system 140 B The first remote smart device system 140 can be controlled by A The smart device further includes a smart thermostat 145 that can be locally controlled by at least the second client device 1102. C1 For example, a smart thermostat 145 C1 Can be connected to smart thermostat 145 C1 The protocol suite and the second client device 1102 via the smart thermostat 145 C1 The control command control is provided by the Bluetooth connection with the second client device 1103. It should be understood that the configuration of the smart device 145 depicted in the figure is only an example, and more or fewer and / or different client devices can be deployed across multiple other rooms and / or areas outside the home.

[0094] Now, in describing Figure 3A , 3B , 4A, 4B, 5A and 5B will be used in various aspects Figure 2 and Figure 2 of the above description.

[0095] Figure 3A 1 shows how to process the recognized text 352A when the second assistant client device 1102 lacks a cache entry that matches the recognized text 352A of the spoken utterance 352 (where the recognized text 352A is generated using the device's built-in STT engine), and optionally, when the local NLU engine of the second assistant client device 1102 cannot effectively process the recognized text 352A. Figure 2 An example of a spoken utterance "turn the light up" received at the second assistant client device 1102. Figure 3A , the second assistant client device 1102 sends the recognized text 352A to the cloud automated assistant component 120. The sending of the recognized text 352A may be in response to the second assistant client device 1102 determining that it lacks a locally stored cache entry that matches the recognized text 352A of the spoken utterance 352, and optionally in response to the local NLU engine of the second assistant client device 1102 being unable to effectively process the recognized text 352A.

[0096] In response to receiving the recognized text 352A, the cloud automated assistant component 120 generates a cloud semantic representation 354 of the recognized text 352A. The cloud semantic representation 354 may be generated based on the recognized text received from the second assistant client device 1102 and based on a reference to a remotely stored device topology. In generating the cloud semantic representation, the cloud automated assistant component 120 may resolve the “light” in the recognized text 352A as a specific light. For example, the cloud automated assistant component 120 may utilize an account identifier received with the transmission of the text 352A to identify the corresponding remotely stored device topology for the account identifier. In addition, the cloud automated assistant component 120 may utilize an identifier of the second assistant client device 1102 received with the text 352A to identify the second assistant client device 1102 in the device topology. Further, the cloud automated assistant component 120 may resolve the “light” in the text 352A as lights 145 based on which lights are mapped in the device topology as default lights for the second assistant client device 1102. B1 and 145 B2 . Light 145 B1 and 145 B2 Can be mapped by default as second assistant client device 1102 based on previous user interface input and / or based on the identifier of room 252 to which they are all assigned in the device topology.

[0097] In generating the cloud semantic representation, the cloud automated assistant component 120 may also resolve "make ..." brighter to mean "2 / 5 (e.g., 40%) brighter than the current brightness intensity level." This may be based on references to one or more remotely stored models and / or rules accessible by the cloud automated assistant component 120. The cloud semantic representation 354 may be generated based on these resolutions and may be, for example, and as shown in the cache request 356, [device=145 B1 and 145 B2 ; brighter, relative intensity is 2 / 5], where "145 B1 and 145 B2 ” is a unique identifier of the corresponding light, and wherein “brighter, relative intensity of 2 / 5” indicates that the lights will be brightened by 2 / 5 relative to their current intensity.

[0098] The cloud automation assistant component 120 generates a corresponding control request using the generated cloud semantic representation 354. The cloud automation assistant component 120 sends the control request to the corresponding light 145. B1 and 145 B2 Smart device system 140 B In response, the smart device system 140 B Corresponding control commands may be generated and provided to the light 145B1 and 145 B2 , to make them "a little brighter" (i.e., increase the relative brightness by 40%, as indicated by the cloud semantic representation).

[0099] In addition, the cloud automated assistant component 120 generates and sends a cache request 356 including the cloud semantic representation 354 and, optionally, the recognized text 352A and / or a mapping of the recognized text to the cloud semantic representation 354 to the client device 1102. In response to receiving the cache request 356, the client device 1102 stores a corresponding cache entry including the mapping of the recognized text 352A to the cloud semantic representation 354 included in the cache request 356.

[0100] Figure 3B Shown in Figure 3A will have cloud semantic representation 354 ( Figure 3A ) is stored at the second assistant client device 1102, how to differently process the cache entry also received at the second assistant client device 1102, Figure 3A An example of another instance of the same spoken utterance 352.

[0101] exist Figure 3B In the embodiment of the present invention, the second assistant client device 1102 processes the audio data of the captured spoken utterance 352 to generate the recognized text. In addition, the second assistant client device 1102 determines whether the recognized text is consistent with the recognized text. Figure 3A In response, the second assistant client device 1102 sends the cloud semantic representation of the matching cache entry to the cloud automation assistant component 120. The cloud automation assistant component 120 can then directly process the cloud semantic representation 354 to generate a corresponding control request and send the control request to the smart device system 140. B In response, the smart device system 140 B Corresponding control commands may be generated and provided to the light 145 B1 and 145 B2 , to make them "brighter". In these and other ways, resources at the cloud automation assistant component can be saved by preventing the generation of cloud semantic representations from occurring again. In addition, control requests can be generated and sent more quickly, thereby reducing the time required to implement the light 145 B1 and 145 B2 The corresponding change waiting time. Note that Figure 3B A control request (and, as a result, a control command) may differ from Figure 3AAlthough the control request is generated based on the same semantic representation, this can be based on: the cloud state change represented by the cloud semantic representation is a relative state change, and the light 145 is generated when the control request is generated. B1 and 145 B2 The "current" state may be Figure 3A and Figure 3B Change between.

[0102] Figure 4A shows how the first assistant client device 1102 may handle a situation in which the first assistant client device 1102 lacks a cache entry that matches the text of the spoken utterance. Figure 2 The first assistant client device 1102 receives Figure 3A and 3B An example of the same spoken utterance ("make the light brighter"). Figure 4A In the example, user 103 provides spoken utterance 452 "Make the light brighter". Although spoken utterance 452 is related to Figure 3A and 3B The same spoken utterances of 352, for numbering consistency, can be marked as Figure 4A and 4B Of the 452.

[0103] exist Figure 4A , the first assistant client device 1101 sends the recognized text 452A generated by processing the audio data of the captured spoken utterance 452 using the local STT engine to the cloud automated assistant component 120. The sending of the recognized text 452A can be in response to the first assistant client device 1101 determining that it lacks a locally stored cache entry that matches the recognized text 452A of the spoken utterance 452, and optionally, in response to the local NLU engine of the first assistant client device 1101 failing to effectively process the recognized text 452A.

[0104] In response to receiving the recognized text 452A, the cloud automated assistant component 120 generates a cloud semantic representation 454 of the recognized text 452A. The cloud semantic representation 454 can be, for example, and as shown in the cache request 456, [device = 145 A1 ; brighter, relative intensity is 2 / 5], among which, "145 A1 ” is a unique identifier of the corresponding light, and wherein “brighter, relative intensity 2 / 5” indicates that the lights are to be brightened to 2 / 5 relative to their current intensity. The cloud semantic representation 454 is different from Figure 3A The cloud semantic representation 354 is that it includes lights 145 A1 Identifier, replaces light 145 B1 and 145 B2This can be based on the cloud automated assistant component 120 identifying the first assistant client device 1101 with the identifier of the first assistant client device 1101 received with the text 452A, and resolving the "light" in the text 452A to light 145 based on mapping the light as a default light for the first assistant client device 1101 in the device topology. A1 Therefore, despite the Figure 3A and 4A The same spoken utterance may be received in an assistant client device, but different cloud semantic representations may be generated based at least in part on the spoken utterance received from different assistant client devices.

[0105] The cloud automation assistant component 120 generates a corresponding control request using the generated cloud semantic representation 454. The cloud automation assistant component 120 sends the control request to the light 145 corresponding to the semantic representation 454. A1 Smart device system 140 A In response, the smart device system 140 A Corresponding control commands may be generated and provided to the light 145 A1 to make it "a little brighter".

[0106] In addition, the cloud automated assistant component 120 generates and sends a cache request 456 including the cloud semantic representation 454 and, optionally, the recognized text 452A and / or a mapping of the recognized text to the cloud semantic representation 454 to the first client device 1101. In response to receiving the cache request 456, the first client device 1101 stores a corresponding cache entry including the mapping of the recognized text 452A to the cloud semantic representation 454 included in the cache request 456.

[0107] Figure 4B It shows that in Figure 4A has cloud semantic representation 454 ( Figure 4A ) is stored in the first assistant client device 1101, how to differently process the received at the first assistant client device 1101 Figure 4A An example of another instance of the same spoken utterance 452.

[0108] exist Figure 4B In the embodiment of the present invention, the first assistant client device 1101 processes the audio data of the captured spoken utterance 452 to generate the recognized text. In addition, the first assistant client device 1101 determines whether the recognized text is consistent with the recognized text. Figure 4AIn response, the first assistant client device 1101 sends the cloud semantic representation 454 of the matching cache entry to the cloud automated assistant component 120. The cloud automated assistant component 120 can then directly process the cloud semantic representation 454 to generate a corresponding control request and send the control request to the smart device system 140. A In response, the smart device system 140 A Corresponding control commands may be generated and provided to the light 145 A1 , to make it "a little brighter".

[0109] Figure 5A 1 shows how to handle the situation when the second assistant client device 1102 lacks a cache entry that matches the recognized text 552A of the spoken utterance 552 (generated using the device's built-in STT engine) - and optionally, when the local NLU engine of the second assistant client device 1102 cannot effectively process the recognized text 452A. Figure 2 An example of another spoken utterance 552 “decrease temp a bit” received at a second assistant client device 1102 .

[0110] exist Figure 5A , the second assistant client device 1102 sends the recognized text 552A to the cloud automated assistant component 120. The sending of the recognized text 552A may be in response to the second assistant client device 1102 determining that it lacks a locally stored cache entry that matches the recognized text 552A of the spoken utterance 552, and optionally in response to the local NLU engine of the second assistant client device 1102 being unable to effectively process the recognized text 552A.

[0111] In response to receiving the recognized text 552A, the cloud automated assistant component 120 generates a locally interpretable semantic representation 554 of the recognized text 552A. In some embodiments, in response to determining that the recognized text 552A inferentially references a smart device (thermostat 145 C1 ) can be locally controlled by the second assistant client device 1102 and / or another client device in network communication with the second assistant client device 1102, the cloud automated assistant component 120 generates a locally interpretable semantic representation 554 (e.g., instead of or in addition to the cloud-based semantic representation).

[0112] A locally interpretable semantic representation 554 may be generated based on a reference to a remotely stored device topology to determine that the “decrease temperature” request is for the thermostat 145C1 (e.g., it may be the only smart device in the device topology that has "cool down" as a supported state change). Thus, text 552A may be parsed as inferentially sending thermostat 145 C1 Referenced as a target smart device. When generating the cloud semantic representation, the cloud automation assistant component 120 can also interpret "lower... a little" to mean "2 degrees less than the current setting value." This can be based on references to one or more remotely stored models and / or rules accessible to the cloud automation assistant component 120. The locally interpretable semantic representation 554 can be generated based on these interpretations, and can be further generated to enable the client device to include a relative state change representation that indicates how to locally interpret the target setting value as "2 degrees less than the current setting value." For example, the relative state change representation can be "temperature setting = (current temperature - 2 degrees)."

[0113] The locally interpretable semantic representation 556 may optionally additionally include: an indication of an adapter to be used when generating a corresponding local control command (e.g., an adapter specific to the manufacturer of the smart device); an indication of a channel to be used to send the local control command (e.g., Bluetooth or Wi-Fi); and / or an indication of a protocol suite to be used in generating the control command based on the locally interpretable semantic representation 556. For example, and as shown in the cache request 556, the locally interpretable semantic representation may be [device=145 C1 ; Temperature setting = (current temperature – 2 degrees); adapter / channel = C], where "145 C1 "It is the thermostat 145 C1 unique identifier of the thermostat (e.g., C1 local address), wherein “temperature setting = (current temperature – 2 degrees)” is a state change representation related to the current state that can be locally interpreted to resolve the current state change; and wherein “adapter / channel = C” indicates (e.g., after determining the current state change) the adapter and / or protocol suite to be utilized when generating local control commands and / or the channel to be utilized to send local control commands.

[0114] The cloud automated assistant component 120 generates and sends a cache request 556 including the cloud semantic representation 554 and optionally the recognized text 552A and / or a mapping of the recognized text to the cloud semantic representation 554 to the second client device 1102. In response to receiving the cache request 556, the second client device 1102 stores a corresponding cache entry including the mapping of the recognized text 552A to the cloud semantic representation 554 included in the cache request 556.

[0115] In some embodiments, and as the second client device 1102 and the thermostat 145C1 As shown by the dashed line between the second client device 1102, the second client device 1102 can utilize the locally interpretable semantic representation of the cached request (or a separate transmission from the cloud automated assistant component 120) to request the thermostat 145 C1 Locally generate local control commands and respond to Figure 5A spoken words, sending them locally to the thermostat 145 C1 .

[0116] In some alternative embodiments, the cloud automation assistant component 120 generates a corresponding cloud semantic representation based on the recognized text 552A. In addition, the cloud automation assistant component 120 can generate a corresponding control request using a separate cloud semantic representation. The cloud automation assistant component 120 sends the control request to the thermostat 145. C1 Corresponding intelligent device system 140 C In response, the smart device system 140 C Corresponding control commands may be generated and provided to the thermostat 145 C1 , to lower its setpoint "a little" (i.e., by 2 degrees, as indicated by the cloud semantic representation).

[0117] Figure 5B Shown in Figure 5A will have a local interpretable semantic representation 554 ( Figure 5A ) is stored in the second assistant client device 1102, how to differently process the received at the second assistant client device 1102 Figure 5A An example of another instance of the same spoken utterance 552.

[0118] exist Figure 5B In the example, the second assistant client device 1102 processes the audio data of the captured spoken utterance 552 to generate the recognized text. In addition, the second assistant client device 1102 determines that the recognized text is consistent with the Figure 4A In response, the second assistant client device 1102 identifies the local interpretable semantic representation 554 of the cache entry and uses the local interpretable semantic representation 554 to generate a corresponding control command. It should be noted that Figure 5B The control commands generated by Figure 5B control command, because the state representation of the local interpretable semantic representation 554 is a state representation related to the current state, and the thermostat 145 C1 The "current" temperature setpoint can be found at Figure 5A and Figure 5B For example, in Figure 5B, the second assistant client device 1102 can resolve the target setting value by subtracting 2 degrees from the current setting value, as indicated by the relative state change of the local interpretable semantic representation 554. The second assistant client device 1102 can then process the target setting value using the corresponding adapter to generate a control command and send the control command to the thermostat 145. C1 The address of the thermostat 145 is optionally included in the locally interpretable semantic representation 554. In response to receiving the control command, the thermostat 145 C1 The setting value can be lowered by 2 degrees.

[0119] Figure 6 is a flow chart illustrating an exemplary method 600 that may be implemented by one or more processors of an assistant client device in accordance with various embodiments disclosed herein.

[0120] At block 602, the assistant client device processes audio data capturing the spoken utterance to generate recognized text for the spoken utterance. In other implementations, instead of processing audio data to generate recognized text, the assistant client device may recognize text based on text that is directly typed (e.g., using a virtual keyboard) or based on text that corresponds to a hardware or software button press (e.g., text for "Dim Lights" is mapped to a graphical software button labeled "Dim").

[0121] In box 604, the assistant client device determines whether the text matches a local cache entry. If so, the client device proceeds to box 606 and selects a semantic representation from the matching local cache entry. In box 608, the system then uses the semantic representation. Box 608 may include sub-boxes 608A and / or 608B. In sub-box 608A, the client device locally processes any of the selected semantic representations as locally interpretable semantic representations to generate local control commands and sends the control commands via a local channel. In sub-box 608B, the client device sends any of the selected semantic representations as cloud semantic representations to a remote system. For example, they may be sent to a remote system, which may then execute boxes 812 and 814 of method 800 (described below). Figure 8 ). In some embodiments, box 606 may include additional sub-boxes, such as a sub-box in which the assistant client device renders any response content included in the local cache entry. For example, the local cache entry may include response text, and the assistant client device may perform local TTS processing on the response text and audibly render the resulting synthesized speech. As another example, the local cache entry may additionally or alternatively include a response graphic, and the graphic may be visually rendered on a display of the assistant client device.

[0122] If the determination of block 604 is no, the assistant client device proceeds to optional block 610 and determines whether the text can be parsed by local NLU (e.g., using a local NLU engine and a local model). If so, the system proceeds to block 612, generates a (locally interpretable and / or cloud) semantic representation, then proceeds to block 608, and uses the generated semantic representation.

[0123] If the determination at box 610 is no (or the determination at 604 is no and 610 is omitted), the client device proceeds to box 614 and sends the audio data and / or text to the remote assistant system. Optionally, box 614 can occur before and / or during execution of boxes 604 and / or 610, and in response to a "yes" determination at either box 604 and 610, further processing at the remote system is optionally paused (e.g., in response to a pause transmission from the client device). For example, if box 614 occurs before and / or during execution of boxes 604 and / or 610, and the assistant client device makes a "yes" determination at boxes 604 and / or 610, a pause request can be sent to the remote assistant system to pause any further processing by the remote processing system related to the sent audio data and / or text. In these and other ways, if the determination at boxes 604 and / or 610 is "no", processing at the remote assistant system can be initiated more quickly to reduce waiting time, while if the determination is "yes", such processing can also be paused to save remote assistant system resources.

[0124] At block 616, the client device receives a cache request from the remote assistant system, the cache request including the semantic representation of the text and optionally the text itself (and / or its representation). Block 616 optionally includes subblock 616A, where the assistant client device optionally proceeds to block 608A to process one or more locally interpretable semantic representations of the cache request (or separate transmission), if any. In other words, at block 616, the assistant client device may immediately act on any locally interpretable semantic representation of the cache request to implement any local smart device action in response to the current spoken utterance or other current user input.

[0125] At block 618 , the client device stores, in a local cache and in response to receiving the cache request, a cache entry that includes a mapping of text to a semantic representation of the cache request.

[0126] Box 618 optionally includes sub-box 618A, which can occur at about the time of box 618 or at a later time. In sub-box 618A, the client device optionally assigns the text in the cache entry for use in an invocation-free action. For example, the client device can assign the text as a "hot phrase," as described herein. Assigning the text as a "hot phrase" can optionally be based on determining that one or more criteria are met. When text recognized by device built-in speech recognition from audio data that occurs independently of any explicit automated assistant invocation is determined to match a hot phrase, the assistant client device can automatically initiate one or more corresponding actions (e.g., actions based on the semantic representation of the corresponding cache entry) based on the text.

[0127] Figure 7 is a flow chart illustrating another exemplary method 700 that may be implemented by one or more processors of an assistant client device according to various embodiments disclosed herein. The method 700 illustrates Figure 6 In a particular embodiment of method 600, determining that the text matches the text of the cache entry (ie, Figure 6 604 is “yes”) and wherein the semantic representation of the cache entry includes a locally interpretable semantic representation.

[0128] At block 702, the assistant client device processes the audio data capturing the spoken utterance to generate recognized text for the spoken utterance. In other implementations, instead of processing the audio data to generate recognized text, the assistant client device may recognize the text based on whether it was directly typed (e.g., using a virtual keyboard) or based on whether it corresponds to a hardware or software button press (e.g., the text for "Dim Lights" is mapped to a graphical software button labeled "Dim").

[0129] At box 704, the assistant client device determines that the text matches the text of the cache entry. Box 704 optionally includes sub-box 704A, wherein the client device optionally determines that the text is assigned to (e.g., at the assistant client device) a call-free action. For example, box 704A may optionally be performed when processing of box 702 occurs without an explicit automated assistant invocation. Further, in such an example, one or more subsequent boxes of method 700 may only be performed if it is determined that the text is assigned to a call-free action (e.g., locally on the client device). In other words, box 704A may be performed when processing of box 702 occurs without an explicit automated assistant invocation, and when it is determined that the text is not assigned to a call-free action, it may be discarded without executing other boxes of method 700 based on the text – whereas when it is determined that the text is assigned to a call-free action, other boxes of method 700 will be executed based on the text.

[0130] At block 706 , the client device selects a locally interpretable semantic representation based on the text in the cache entry that is mapped to the cache entry.

[0131] In box 708, the client device determines whether the state change of the local interpretable semantic representation is a state change related to the current state. If not, the client device proceeds to box 712 and uses the state change of the local interpretable semantic representation. For example, if the state change indicates a fixed value, the fixed value can be used as the state change. If the determination at box 708 is "yes", the client device proceeds to box 710 and parses the state change based on the current state value. For example, if the state change is related to the current state, it can be parsed by identifying the current state of one or more smart devices and then determining the state change based on the identified current state. For example, if the state change is "set value = (current temperature + 3 degrees)", it can be parsed by identifying "current temperature".

[0132] At block 712 , the client device generates a local control command using the state change and optionally using an adapter identified based on the cache entry.

[0133] At block 714, the client device sends a local command on a local channel optionally selected based on the cache entry (e.g., explicitly identified in the cache entry, or selected based on the protocol suite, smart device, and / or radio / channel identified in the cache entry). The sent local command may be addressed to the smart device based on those addresses identified in or resolved thereby the locally interpretable semantic representation.

[0134] At block 718, the client device determines whether there are any additional unprocessed semantic representations in the cache entry. If not, the client device proceeds to block 720 and method 700 ends. If yes, the client device proceeds to block 722 and determines whether the next unprocessed semantic representation can be interpreted locally. If yes, the client device returns to block 708. If no (i.e., it is a cloud semantic representation), the client device proceeds to Figure 6 The method 600 of FIG. 608B is then executed, and then the method 608B is returned to block 718 after executing block 608B.

[0135] Figure 8 8 is a flow chart illustrating an exemplary method 800 that may be implemented by one or more processors of a remote assistant system in accordance with various implementations disclosed herein.

[0136] At block 802, the system receives a request from an assistant client device. At block 804, the system determines whether the request is a semantic cloud representation. If so, the system performs blocks 812 and 814 (described below) of method 800 without performing any other blocks of method 800.

[0137] If not, the system proceeds to block 808 and generates a semantic representation of the spoken utterance based on the representation of the spoken utterance in the request. In other implementations, the representation may be typed utterance, or pressed text mapped to a hardware element or software graphical element.

[0138] In optional block 810, the system generates confirmation content for rendering at the assistant client device in response to the request.

[0139] At optional block 812, the system generates a control request based on the cloud semantic representation, which is generated based on the request. Optional block 812 may include optional block 812A, wherein if the state representation of the semantic representation is a relative / current state-related state representation, the system generates the control request based on the current state of the smart device.

[0140] At optional block 814 , the system sends the generated control request to the smart device remote system.

[0141] At block 816 , the system generates a cache request that includes the (locally interpretable and / or cloud) semantic representation and optionally a mapping of the text to the semantic representation.

[0142] At block 818, the system sends the cache request to the assistant client device. Block 818 optionally includes subblock 818A, where, if the semantic representation of the cache request is a locally interpretable semantic representation, the system optionally generates a separate request for implementing the locally interpretable semantic representation in response to the current request.

[0143] Fig. 9 9 is a block diagram of an exemplary computing device 910 that may optionally be used to perform one or more aspects of the techniques described herein. The computing device 910 generally includes at least one processor 914 that communicates with a number of peripheral devices via a bus subsystem 912. These peripheral devices may include a storage subsystem 924 (including, for example, a memory subsystem 925 and a file storage subsystem 926), a user interface output device 920, a user interface input device 922, and a network interface subsystem 916. The input devices and output devices allow a user to interact with the computing device 910. The network interface subsystem 916 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0144] User interface input devices 922 may include a keyboard, a pointing device such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into computing device 910 or onto a communication network.

[0145] The user interface output device 920 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 910 to a user or to another machine or computing device.

[0146] Storage subsystem 924 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 924 may include logic to perform selected aspects of one or more methods described herein.

[0147] These software modules are typically executed by the processor 914 alone or in combination with other processors. The memory 925 used in the storage subsystem 924 may include multiple memories, including a main random access memory (RAM) 930 for storing instructions and data during program execution and a read-only memory (ROM) 932 for storing fixed instructions. The file storage subsystem 926 can provide permanent storage for program and data files, and may include a hard drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functions of certain embodiments may be stored in the storage subsystem 924 by the file storage subsystem 926 or may be stored in other machines accessible by the processor 914.

[0148] The bus subsystem 912 provides a mechanism for the various components and subsystems of the computing device 910 to communicate with each other in an intended manner. Although the bus subsystem 912 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0149] Computing device 910 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the varied nature of computers and networks, Fig. 9 The description of the computing device 910 illustrated in FIG. 1 is intended only as a specific example for the purpose of illustrating some embodiments. Many other configurations of the computing device 910 may have more Fig. 9 The computing devices illustrated in the figure may have more or fewer components.

[0150] Where certain embodiments discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information only when explicit authorization to do so is received from the relevant user.

[0151] In some embodiments, a method is provided, including receiving, at a remote assistant system and from an assistant client device, a representation of a spoken utterance captured at an assistant client device. The spoken utterance is a request related to one or more states of one or more smart devices linked to the assistant client device. As an example, the request may be a request to change the state of the smart device (i.e., to change the state from a current state value to a new state value). The representation of the spoken utterance includes capturing audio data of the spoken utterance and / or text of the spoken utterance generated at the client device using a speech-to-text model stored locally on the client device. The method further includes at the remote system and in response to receiving the representation of the spoken utterance: generating a semantic representation of the spoken utterance based on the representation of the spoken utterance; generating at least one control request based on the semantic representation of the spoken utterance to be sent to at least one smart device remote system that controls one or more smart devices, wherein at least one control request is different from the semantic representation; sending at least one control request to at least one smart device remote system so that at least one smart device remote system provides one or more corresponding commands to one or more smart devices; and sending a cache request including the semantic representation to the assistant client device. Sending the cache request to the assistant client device causes the assistant client device to store a cache entry including a mapping of the text to the semantic representation in the cache of the assistant client device.

[0152] These and other implementations of the technology may optionally include one or more of the following features.

[0153] In some embodiments, the method further includes, after sending the cache request to the assistant client device: capturing additional audio data at the assistant client device, the additional audio data capturing the additional spoken utterance; processing the additional audio data to generate current text corresponding to the additional spoken utterance using a speech-to-text model stored locally on the assistant client device; determining that the current text matches the text of the cache entry; and in response to determining that the current text matches the text of the cache entry, and in response to the cache entry including a mapping of the text to a semantic representation: sending the semantic representation to the remote assistant system. In some of those embodiments, processing the additional audio data to generate the current text is performed when no explicit automated assistant invocation is detected at the assistant client device. In some versions of those embodiments, sending the semantic representation to the remote assistant system is further responsive to: determining that the current text satisfies one or more criteria for enabling the matching text to be operational in the absence of an explicit automated assistant invocation at the assistant client device. In some additional or alternative versions, the method further includes: receiving, at a remote assistant system, a semantic representation sent from an assistant client device; based on the semantic representation, generating at least one alternative control request to send to at least one smart device remote system that controls one or more smart devices, wherein the at least one alternative control request is different from the at least one control request; and sending at least one alternative control request to the at least one smart device remote system so that the at least one smart device remote system changes one or more states of the one or more smart devices. Optionally, in those alternative or additional embodiments: generating at least one control request is further based on at least one current state of the one or more smart devices when generating the at least one control request; generating at least one alternative control request is further based on at least one alternative current state of the one or more smart devices at different times when generating the at least one alternative control request; based on at least one current state being different from at least one alternative current state, the at least one alternative control request is different from the at least one control request.

[0154] In some embodiments, generating the semantic representation of the spoken utterance is further based on a device topology for an account of an assistant client device, wherein the device topology defines an assistant client device, one or more smart devices, additional assistant client devices, and additional smart devices. In some of those embodiments, generating the semantic representation of the spoken utterance further based on the device topology includes: resolving ambiguous terms in the text of the spoken utterance as references to the one or more smart devices based on the device topology.

[0155] In some embodiments, the semantic representation includes: a state change representation of one or more state changes; and a corresponding unique identifier for each of the one or more smart devices. In some versions of those embodiments, the state change representation is a relative state change representation that defines one or more state changes in a relative manner rather than an absolute manner. In some additional or alternative versions of those embodiments, the method further includes: determining a change in device topology at a remote assistant system or assistant client device; and in response to determining a change in device topology: causing cache entries to be cleared from a cache of the assistant device. Determining a change in device topology may optionally include: determining that the change is to one or more attributes defined in the device topology for the assistant client device, one or more smart devices, and / or one or more rooms assigned to the assistant client device and / or one or more smart devices; and optionally, causing the cache entry to be cleared from the cache further in response to determining that the cache entry is affected by the one or more attributes.

[0156] In some embodiments, the method further includes receiving an additional representation of the spoken utterance at the remote assistant system and from an additional assistant client device. The spoken utterance is captured at the additional assistant client device, and both the additional assistant client device and the assistant client device are members of the same device topology. In some of those embodiments, the method further includes, at the remote system and in response to receiving the additional representation of the spoken utterance: generating an alternative semantic representation of the spoken utterance based on the representation of the spoken utterance, wherein the alternative semantic representation of the spoken utterance is different from the semantic representation based on the additional representation captured at the additional assistant client device and the representation captured at the assistant client device; and sending an additional cache request including the alternative semantic representation to the additional assistant client device. Sending the additional cache request to the additional assistant client device causes the additional assistant client device to store an additional cache entry in an additional cache on the additional assistant client device, the additional cache entry including a mapping of text to the alternative semantic representation.

[0157] In some embodiments, a method is provided, comprising: receiving, at a remote assistant system and from an assistant client device, a representation of a spoken utterance captured at an assistant client device. The spoken utterance is a request to change at least a given state of a given smart device linked to the assistant client device. The representation of the spoken utterance includes capturing audio data of the spoken utterance and / or text of the spoken utterance generated at the client device using a speech-to-text model stored locally on the client device. The method further includes, at the remote system and in response to receiving the representation of the spoken utterance: determining that a given smart device can be locally controlled by an assistant client device and / or by an additional client device having a local connection to the assistant client device; generating a semantic representation of the spoken utterance, the semantic representation including a locally interpretable semantic representation, and sending a cache request including the semantic representation to the assistant client device. The locally interpretable semantic representation can be locally interpreted by the assistant client device and / or the additional client device to generate a corresponding control command, which can be sent on a local channel to change at least a given state of the given smart device. Generating a locally interpretable semantic representation for at least one smart device includes including the locally interpretable semantic representation in the semantic representation in response to determining that the given smart device can be locally controlled. Sending a cache request to the assistant client device causes the assistant client device to store a cache entry including a mapping of text to semantic representations in a cache of the assistant client device.

[0158] These and additional implementations of the technology may optionally include one or more of the following features.

[0159] In some embodiments, the method further includes, after sending the cache request to the assistant client device: capturing additional audio data at the assistant client device, the additional audio data capturing the additional spoken utterance; processing the additional audio data using a speech-to-text model stored locally on the assistant client device to generate a current text corresponding to the additional spoken utterance; determining that the current text matches the text of the cache entry; and in response to determining that the current text matches the text of the cache entry, and in response to the cache entry including a mapping of the text to the semantic representation: processing the local interpretable semantic representation at the assistant client device to generate a given control command; and sending the given control command via a local channel to cause a change in at least a given state of a given smart device. In some versions of those embodiments, the local interpretable semantic representation includes: an identifier of the given smart device; and a current state-related state change representation, the current state-related state change representation defining a corresponding current value relative to a given state of the given smart device, changing at least a given state of the given smart device. In some of those versions, processing the local interpretable semantic representation to generate the given control command includes: determining a current value of a given state of the given smart device at the assistant client device; generating the given control command based on applying the current value to the current state-related state change representation. Optionally, the locally interpretable semantic representation further includes an indication of a manufacturer of the smart device, and generating a given control command optionally includes utilizing an adapter stored locally on the assistant client device in response to an adapter assigned to the indication of the manufacturer of the smart device. Optionally, the method further includes selecting a local channel to send the given control command based on a local channel indicated directly or indirectly in the locally interpretable semantic representation. For example, based on a local channel assigned to the indication of the manufacturer of the smart device included in the locally interpretable semantic indication and / or assigned to a protocol suite included in the locally interpretable semantic representation.

[0160] In some embodiments, processing the additional audio data to generate the current text is performed without detecting an explicit automated assistant invocation at the assistant client device. In some of those embodiments, sending the given control command is further responsive to determining that the current text satisfies one or more criteria for enabling matching text to be operated without an explicit automated assistant invocation at the assistant client device.

[0161] In some embodiments, generating the semantic representation of the spoken utterance is further based on a device topology for an account of an assistant client device. The device topology defines an assistant client device, a given smart device, an additional assistant client device, and an additional smart device. In some of those embodiments, generating the semantic representation of the spoken utterance includes resolving ambiguous terms in the text of the spoken utterance as referring to the given smart device.

[0162] In some embodiments, the method further includes: determining, at the remote assistant system or the assistant client device, a change in device topology; and in response to determining the change in device topology: causing the cache entry to be purged from the cache of the assistant client device. In some of those embodiments, determining the change in device topology includes: determining that the change is to one or more properties defined in the device topology for the assistant client device, a given smart device, and / or one or more rooms assigned to the assistant client device and / or the given smart device; and causing the cache entry to be purged from the cache further in response to determining that the cache entry is affected by the one or more properties.

[0163] In some embodiments, a method is provided, including storing a cache entry in a cache on a client device, the cache entry including a mapping of text to a semantic representation. The semantic representation includes a locally interpretable semantic representation that can be interpreted locally by the client device. The method further includes capturing additional audio data at the client device, the additional audio data capturing the spoken utterance. The method further includes processing the additional audio data to generate a current text corresponding to the spoken utterance using a speech-to-text model stored locally on the client device. The method further includes determining a text match between the current text and the cache entry. The method further includes, in response to determining that the current text matches the text of the cache entry, and in response to the cache entry including a mapping of text to the semantic representation: at the client device, processing the locally interpretable semantic representation to generate a given control command; and sending the given control command via a local channel to cause at least a given state of a given smart device to be changed.

Claims

1. A method implemented by one or more processors of a client device, the method comprising: storing in a cache on the client device: a cache entry comprising a mapping of text to a semantic representation, wherein the semantic representation comprises a locally interpretable semantic representation locally interpretable by the client device; capturing additional audio data at the client device, the additional audio data capturing spoken utterances; processing the additional audio data to generate current text corresponding to the spoken utterance using a speech-to-text model stored locally on the client device; Determining that the current text matches the text of the cache entry; and In response to determining that the current text matches the text of the cache entry, and in response to the cache entry including a mapping of the text to the semantic representation: processing the locally interpretable semantic representation at the client device to generate a given control command, wherein the given control command is generated in response to determining that the locally interpretable semantic representation cannot be locally interpreted by the given smart device; as well as The given control command is sent through the local channel to change at least a given state of the given smart device.

2. The method of claim 1, wherein: The local interpretable semantic representation includes: An identifier of the given smart device; and A current state-related state change representation defines a change in at least a given state of the given smart device relative to a corresponding current value of a given state of the given smart device.

3. The method of claim 2, wherein: Processing the locally interpretable semantic representation to generate the given control command comprises: determining, at the assistant client device, a current value of a given state of the given smart device; The given control command is generated based on applying the current value to the current state related state change representation.

4. The method of claim 3, wherein: The locally interpretable semantic representation further includes an indication of a manufacturer of the given smart device.

5. The method of claim 4, wherein: Generating the given control command includes utilizing the adapter in response to an indication locally stored at the client device that the adapter is assigned to a manufacturer of the given smart device.

6. The method of claim 4, further comprising: Based on a manufacturer's indicated local channel assigned to the given smart device, the local channel is selected to transmit the given control command.

7. The method according to any one of claims 1 to 6, wherein: In the absence of detecting an explicit automated assistant invocation at the client device, processing the audio data to generate the current text is performed.

8. The method of claim 7, wherein: Sending the given control command is further responsive to determining that the current text satisfies one or more criteria for enabling matching text to be operational without an explicit automated assistant invocation at the assistant client device.

9. A method implemented by one or more processors of a client device, the method comprising: storing a cache entry in a cache on the client device, the cache entry comprising a mapping of text to a semantic representation, wherein the semantic representation is locally interpretable by the client device to generate a control command for at least one given smart device of one or more smart devices included in a device topology, wherein the at least one control command is different from the semantic representation; After storing the cache entry, identifying a change in the device topology; In response to identifying a change in the device topology, determining whether to update the cache entry based on the change in the device topology; and In response to determining to update the cache entry: Generate and store an update cache entry that replaces the cache entry in the cache on the client device, wherein the update cache entry includes an update mapping, the update mapping being: a mapping of the text to an updated semantic representation based on a change in the device topology, updating the mapping of text to the semantic representation based on changes to the device topology, or A mapping of the update text to the update semantic representation.

10. The method of claim 9, further comprising: In response to generating and storing the update cache entry: The cache entry is cleared at least on the client device.

11. The method of claim 9, wherein: The device topology is at least partially user created and includes a corresponding identifier for each of the smart devices included in the device topology and includes one or more corresponding attributes for each of the smart devices included in the device topology.

12. The method of claim 9, wherein: The change of the device topology includes adding a new intelligent device to the device topology.

13. The method of claim 9, wherein: The change of the device topology includes renaming the given smart device, and wherein the update mapping of the update cache entry is a mapping of the update text to the semantic representation or a mapping of the update text to the update semantic representation.

14. The method of claim 9, wherein: The alteration of the device topology comprises assigning an additional smart device to a room or group to which at least the given smart device has been assigned, and wherein the update mapping of the update cache entry is a mapping of the text to the update semantic representation or a mapping of the update text to the update semantic representation.

15. The method of claim 9, wherein: Determining whether to update the cache entry based on the change in the device topology includes determining whether the change in the device topology affects the text and / or the semantic representation of the cache entry.

16. A method implemented by one or more processors, comprising: designating text of a cache entry stored in a cache of the assistant client device as a hot phrase, wherein designating the text as the hot phrase is based on determining that one or more criteria associated with the text are satisfied, and wherein the cache entry maps the text to a semantic representation locally interpretable by the assistant client device; After designating the text as the hot phrase: processing, at the assistant client device and utilizing an on-device speech-to-text model, audio data detected at the assistant client device and capturing a spoken utterance, wherein processing the audio data utilizing the on-device speech-to-text model generates recognized text of the spoken utterance, and is performed without detecting any explicit automated assistant invocation; determining, at the assistant client device, whether the recognized text of the spoken utterance matches the text of the cache entry, wherein determining whether the recognized text generated without detecting any explicit automated assistant invocation matches the text of the cache entry is in response to the text of the cache entry being designated as the hot phrase; In response to determining that the recognized text matches the text of the cache entry, and the text of the cache entry is designated as the hot phrase and mapped to the semantic representation: generating output for controlling one or more smart devices using the semantic representation of the cache entry; and The output is sent.

17. The method of claim 16, wherein: Determining that one or more criteria associated with the text are satisfied includes determining that the text and / or matching text has been determined to be present in the user input at least a threshold number of times.

18. The method of claim 17, wherein: Determining that one or more criteria associated with the text are satisfied includes determining that the text and / or matching text has been determined to be present in the user input with at least a threshold frequency.

19. The method of claim 16, wherein: Determining that one or more criteria associated with the text are satisfied includes determining that the text and / or matching text has been determined to be present in the user input with at least a threshold frequency.

20. The method of claim 16, wherein: Designating the text of the cache entry as a hot phrase automatically occurs in response to determining that the one or more criteria are satisfied.

21. The method of claim 16, further comprising: providing a prompt based on determining that one or more criteria associated with the text are satisfied; as well as receiving a confirmation user input in response to providing the prompt; Wherein, designating the text of the cache entry as the hot phrase is in response to receiving the confirming user input in response to providing the prompt.

22. The method of claim 16, in, Generating the output for controlling one or more smart devices using the semantic representation of the cache entry includes: generating a control command different from the semantic representation; and Wherein, sending the output includes sending the control command to the one or more smart devices via a local channel.

23. At least one computing device, comprising one or more processors and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the one or more processors, cause the one or more processors to perform the method as described in any one of claims 1-22.

24. A non-transitory computer-readable storage medium storing computer-executable instructions, which, when executed by one or more processors of at least one computing device, cause the one or more processors to perform the method of any one of claims 1-22.

Images