Accommodate differences in device status reporting from third-party servers

Abstract

Embodiments herein relate to information describing one or more internal states of a technical system. The methods described herein are provided for characterizing the reliability of various different third-party servers, at least when reporting third-party device states, and protocols for accommodating device ecosystems that are affected by such reliability. Latency can affect the accuracy of device states represented by assistant devices. Certain servers can be characterized as being particularly delayed when reporting updated device states in response to user requests, and as a result, third-party servers can be associated with metrics that characterize the relative latency of the third-party servers. When a metric fails to meet a particular threshold, servers and / or clients associated with an "ecosystem" of third-party devices can affirmatively operate to retrieve device state updates, rather than passively waiting for updates from corresponding third-party servers.

Description

Background Art

[0001] A human may engage in a human-computer dialogue with an interactive software application, referred to herein as an "automated assistant" (also referred to as a "digital agent," "chat program," "interactive personal assistant," "intelligent personal assistant," "assistant application," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when they interact with an automated assistant) may provide commands and / or requests to the automated assistant using verbal natural language input (i.e., spoken utterances) and / or by providing textual (e.g., typed) natural language input, which may in some cases be converted to text and then processed. The automated assistant may respond to the request by providing responsive user interface output, which can include audible user interface output and / or visual user interface output.

[0002] In many cases, users can provide verbal utterances to cause the automated assistant to perform various actions to facilitate control of a particular device. Certain devices can become available through third-party entities that can exhibit various levels of reliability in providing operational data to the user's automated assistant on a regular basis. For example, a third-party entity that uses an unreliable push of device status updates to the automated assistant's server may cause inaccurate status data to be provided to the user and cause the user to mistakenly rely on inaccurate status data. Therefore, users who always rely on the interface of their automated assistant device to determine the status of certain devices may be presented with information that inaccurately represents one or more internal states of the device and / or one or more internal states of the broader technical system in which the device resides. Users may accidentally keep certain devices running or not running based on such inaccurate status data, resulting in the consumption of undesirable resources. In addition, users may deploy insignificant requests to modify the settings of devices whose settings have been appropriately modified, thereby wasting network bandwidth and processing resources. For example, users who rely on their automated assistants to determine the operating status of their lights or furnaces may not know the latest operating data that is otherwise available at the third-party server. Such recent operating data can cause users to be presented with inaccurate information about the internal states of lights and / or furnaces. Thus, a user may cause certain requests to be sent to a third-party server for controlling lights and / or furnaces based on inaccurate status data. For example, a user may issue a verbal command to their automated assistant to turn off a set of lights in their living room, even though the lights may already be off. Thus, the verbal command will still be processed and sent, for example, to the automated assistant server and the third-party server, which may then reject the command given that the lights are already off. This insignificant transaction occurring on many devices may result in a waste of network traffic and power consumption. Summary of the invention

[0003] The embodiments set forth herein relate to information describing one or more internal states of a technical system. The embodiments set forth herein are provided for characterizing the reliability of various third-party servers at least in terms of reporting third-party device states to an automated assistant, a client device, and / or another server device. In addition, some embodiments relate to protocols adapted to eliminate the negative impact of such delays on the automated assistant and / or the corresponding device ecosystem. A specific ecosystem such as a network of computing devices connected to a local area network can exhibit various different states during the operating life of the device. Some ecosystems can include at least one assistant device, through which a user can control various devices within its ecosystem by providing a request to an automated assistant, which can be accessed via the assistant device. When the state of a device within the ecosystem is modified according to a request and / or other command, the automated assistant can track the state of the device. For example, in order for an automated assistant to ascertain the state of a specific device such as a smart light bulb at home, a third-party server associated with the smart light bulb can provide a state update to a first-party server associated with the automated assistant (e.g., a first-party server relative to the automated assistant and the third-party smart light bulb).

[0004] However, differences in the reliability of third-party servers that provide status updates reflecting the accurate status of devices in a timely manner can affect the accuracy of status information provided by an automated assistant to one or more users. For example, a particular smart device manufacturer may not provide status updates to a first-party server and / or assistant device (or may provide them with significant delays) in response to a user manually controlling a smart device (e.g., turning off a light switch of a smart light bulb) without assistance from an automated assistant. As a result, if the user subsequently requests the status of all of their devices within their respective ecosystems, the status of a particular smart device may be inaccurately presented. Such inaccuracy can result in a waste of energy and / or computing resources. For example, a user who has manually adjusted a smart light bulb and then checks the status (i.e., state) of the smart light bulb before going to bed may be provided with inaccurate and misleading status information about the smart light bulb, and therefore may not realize that the smart light bulb remains on throughout the night. If the user is provided with more accurate status information, the user will be able to turn off the light bulb before going to bed via the assistant device, thereby saving energy resources at the home and / or any other affected device. Alternatively, when the status of the smart light bulb is indicated as off, but the light is actually on, the user can provide a command to turn on their smart light bulb. Such commands provided by the user can result in a waste of computing and / or network resources, since at least speech-to-text processing and / or network transmission will need to be completed, only for the corresponding server device to reject the command if the communication is already connected.

[0005] The embodiments discussed herein allow the automated assistant to actively request the device state of the device within the ecosystem according to one or more different metrics that characterize the reliability of one or more server devices and / or other support systems associated with those devices. The support system can represent a server device and / or a client device that is responsible for managing the device state of the device and reporting those device states to other servers, clients, applications, and / or any other devices or modules. In some embodiments, a delay-based metric can be generated for one or more support systems corresponding to one or more devices that can be controlled via the automated assistant. The delay-based metric can be generated using time data that characterizes the amount of time that occurs between a request (e.g., from an automated assistant) to change the device state and reporting the updated device state to the automated assistant, the first-party server device, and / or the client device associated with the automated assistant. The delay-based metric can then be used, for example, when a third-party server is typically unreliable in actively transmitting an updated state to the automated assistant, to make a decision about whether to actively request an updated device state. In addition, the delay-based metric can also be used to make a decision about whether or when not to actively request an updated device state, for example, when a third-party server has proven to be reliable in actively reporting an updated device state to the automated assistant.

[0006] In some embodiments, one or more metrics can be generated for determining how a particular server device reports a device state change with certainty. For example, a smart device connected within an ecosystem is controllable via an automated assistant and a third-party application. If a user chooses to control the particular smart device via a third-party application, the server device can receive a request from the third-party application and cause the device state change to occur. However, the third-party server may not diligently provide an updated state to the automated assistant. For example, while some third-party servers may actively provide a state update to the automated assistant immediately in response to a third-party application causing a device state change, some third-party servers may not be active. For example, some third-party servers may not provide device state updates to the automated assistant, and / or may provide device state updates after a certain amount of delay. One or more metrics of a third-party server can be generated based on such omissions and / or delays exhibited by the third-party server.

[0007] In some implementations, a metric in the one or more metrics can be determined based on a probability that a particular third-party server will proactively provide status updates to the automated assistant and / or a probability that the automated assistant will have accurate status data (such as provided by a third-party client device and / or a particular third-party server device) at any given time. When a particular metric does not meet a threshold, the automated assistant can choose to query the third-party server for status updates of certain devices more frequently. Alternatively, when a particular metric meets a particular threshold, the automated assistant can bypass querying the third-party server for status change updates, at least in view of the third-party server's history of proactively providing such device status updates.

[0008] In some embodiments, one or more metrics can affect a variety of different operations performed by the automated assistant. For example, a user can provide an action (e.g., a tap or slide action) at a touch display panel of the assistant device to cause the assistant device to present the operating status of multiple different devices associated with the "ecosystem" corresponding to the user. The operating status can be presented as a graphical display element that characterizes the operating state of the device (e.g., a graphic depicting a light that glows brightly to indicate that a particular smart bulb is turned on; another graphic depicting a padlock in a closed position to indicate that an alarm system has been secured; and another graphic depicting a thermometer and indicating the current temperature of a room in the user's home). In response to receiving an input action, the assistant device and / or a corresponding server device can cause one or more metrics to be accessed. Each of the one or more metrics can correspond to a specific device (e.g., an alarm system, a thermostat, and a light). Furthermore, in response to an action, each metric can be used to determine whether to query a third-party server for the current state of each particular device, or to bypass querying the third-party server for the current state of each particular device (at least assuming that the value of the metric provides some basis (e.g., satisfies a threshold), or fails to provide some basis from which to assume that the third-party server device supporting the particular device is reliable in proactively providing updates about changes in the operating state of the particular device). Thus, in some instances, the assistant device can query one or more third-party server devices based on one or more metrics, and / or bypass querying one or more third-party server devices based on one or more queries.

[0009] As another example, in some embodiments, the automated assistant can execute a routine, which can be a combination of operations performed according to the settings of the routine. The routine can be, for example, a "goodnight" routine, which can be established by the user to combine various automated assistant operations that are typically requested by the user before going to bed into a single request. The operations corresponding to the "goodnight" routine can include, for example, one or more of the following: (1) turning off various smart lights in the home, (2) turning on the security alarm in the home, and (3) playing a certain amount of audio that the user usually likes to listen to when in bed. During the execution of the routine, the automated assistant can access device status data corresponding to the security alarm and the smart light. The accessed device status data can be current device status data maintained by the automated assistant based on previous communications with a third-party server. In addition, the automated assistant can access one or more metrics corresponding to a server device for a smart light bulb and a server device for an alarm system. The device status data can indicate that the light has been turned off and the security alarm has been activated. However, the automated assistant can determine whether to query each server device to confirm whether the state is correct based on the one or more metrics. For example, a metric of a server device of a smart light bulb can satisfy a threshold, and based on (e.g., in response to) the metric satisfying the threshold, the automated assistant can bypass querying the server device for the state of the light bulb. In other words, the automated assistant can determine that a particular server device has a history of proactively providing state updates consistently and / or in a timely manner, and therefore, the metric can indicate that the data characterizing the state of the device is accurate.

[0010] However, in some instances, a metric corresponding to a server device supporting a device and / or system (e.g., an alarm system) may not satisfy a threshold. Therefore, based on (e.g., in response to) a metric associated with the alarm system not satisfying a threshold, the automated assistant can proactively query the corresponding server device to confirm whether the device state data is accurate. When the device state is inaccurate, and therefore the alarm system is determined to have an inaccurate state, the automated assistant can undertake an operation to request a change in the state of the security alarm from the corresponding server device during the execution of the routine. Optionally, and based on (e.g., in response to) one or more metrics, the automated assistant can then request status verification of the alarm system from the server device. In other words, because the server device may not have a history of consistently and / or promptly providing device state updates after a request for a device state change, the automated assistant can proactively confirm that the device state change has actually taken effect.

[0011] Over time, one or more previously generated metrics can be modified according to changes in the pattern of how a particular active third-party server device reports device status. For example, a third-party server device can become more active over time, or a third-party server device can become less active over time. Factors affecting this may include: changes in the total network traffic to a particular third-party server device, an increase in the use of devices and / or accounts by users, implementation methods of software upgrades of third-party servers, expansion or reduction of available network servers, updates pushed, lack of support for previous versions of certain third-party devices, and / or any other factors that can affect how active a server device is in reporting device status. As an example, a metric for a particular third-party server device may not initially meet a specific threshold, but, due to certain enhancements issued by a third-party entity, the third-party server device can become more active over time. As a result, a first-party server, an automated assistant, and / or any other device responsible for tracking such a metric can modify the corresponding metric of a third-party server device to reflect the trend that the third-party server device becomes more active. This can enable an automated assistant to reduce the frequency of status requests provided to a third-party server device, at least in part due to a metric reflecting the reliability of the third-party server device that reports the device status. Furthermore, this can eliminate waste of network resources that might otherwise be consumed by indiscriminately providing device state requests, at least with respect to the latency and / or accuracy of any previously provided device state changes.

[0012] In some embodiments, one or more metrics and / or one or more reference values ​​can be generated to correspond to a particular device and a particular context of the particular device (e.g., time of day, transaction type, type of assistant device, type of third-party server device, type of third-party client device, a particular user associated with the particular device and / or command). As an example, a first metric for a particular device can characterize the reliability of a third-party server device supported during weekends (or any time range), and a second metric for a particular device can characterize the reliability of a third-party server device supported during weekdays (or any other time range). In this way, when an assistant device is responsible for presenting the operating status of a particular device, the assistant device and / or assistant server device can identify the current context and then select a metric based on the current context (e.g., a particular time of day). The selected metric can then be used to determine whether to query a supporting third-party server device for updated status data, or to determine whether the supporting third-party server device is reliable in proactively providing such status updates.

[0013] It will be apparent from the above discussion that the embodiments described herein can enable the information presented by the automated assistant device to more accurately represent the internal state of a technical system, such as the state of one or more of a smart light bulb, a furnace, or an electrical safety alarm (to give just a few examples). This information can help a user perform technical tasks, including those associated with a technical system whose internal state is represented by the presented information. An example of such a task is controlling a technical system to assume or remain in a specific internal state. The provision of inaccurate state data to a user by an automated assistant device via a user interface of the automated assistant device can be reduced or avoided. In addition, the reliance of the automated assistant device on such inaccurate state data can also be reduced or avoided when implementing operations in a technical system.

[0014] The above description is provided as an overview of some embodiments of the present disclosure. Further descriptions of these and other embodiments are described in more detail below.

[0015] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described above and / or elsewhere herein. Other embodiments may include a system of one or more computers and / or one or more robots including one or more processors operable to execute the stored instructions to perform methods such as one or more of the methods described above and / or elsewhere herein.

[0016] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A and Figure 1B A diagram illustrating a secondary server device adapting to changes in reliability and / or reliability patterns of a third party server device when reporting status updates for a particular client device.

[0018] Figure 2A and Figure 2B Illustrated is a diagram of an assistant server device that adapts the reliability of a third-party server device in reporting state changes to a client device that provides access to an automated assistant.

[0019] Figure 3A system is illustrated for characterizing the reliability of various third-party servers, at least in reporting third-party device status, and a protocol for accommodating an ecosystem of devices affected by such reliability.

[0020] Figure 4 Illustrated is a method for determining whether to query a third-party server for status updates corresponding to a particular third-party client device based on one or more metrics.

[0021] Figure 5 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0022] Figure 1A and Figure 1B The views 100 and 140 respectively illustrate the assistant server device 104 adapting to changes in reliability and / or reliability mode of a third-party server device 102 when reporting status updates of a particular client device 108 (e.g., a thermostat). The client device 108 can be connected to a local area network provided via a wireless network source 116 such as a Wi-Fi router. Other client devices can also be connected to the local area network, such as a tablet device (and / or any computing device (i.e., client device 110) and assistant device 114, which can communicate with the assistant server device 104, the client device 110 can be controlled by a user 106 and can include a third-party application 120, through which the user 106 can control the client device 108.

[0023] The user 106 can interact with the third-party application 120 via its interface 118 in order to control the client device 108. The interface 118 can be presented at a display panel 128 of the client device 110, and can present controls to the user for modifying the operating state of the client device 108. For example, the third-party application 120 can present a first control interface 124 and a second control interface 126. The first control interface 124 can include various graphical control elements through which the user 106 can modify one or more operations and / or settings of the client device 108. The second control interface 126 can include one or more graphical control elements from which the user 106 can control one or more operations and / or settings of the client device 108.

[0024] For example, when the client device 108 is a thermostat, the interface 118 can provide a graphical control element from which the user 106 can change the temperature setting of the client device 108. The interface 118 can also include a current temperature reading (e.g., 65 degrees), which can present the current temperature in the room 112 in which the user is located. The current temperature of the room 112 can be 65 degrees, and the user can provide input to the graphical control element of the interface 118 to adjust the room 112 temperature to 72 degrees. In response to the user 106 providing the interface 118 or otherwise interacting with the interface 118, the third-party application 120 can send a device request 136 to the third-party server device 102 via the local area network. The device request 136 can cause the operating state of the client device 108 to change in order to implement the setting change requested by the user 106 via the third-party application 120.

[0025] The third-party server device 102 can be responsible for updating the server device and / or the client device with changes in the operational state of the client device 108. For example, the user 106 can access the assistant device 114 in order to determine the state of the client device 108. Based on the state data available to the assistant device 114 and / or the auxiliary server device 104, the state of the client device 108 can be presented at the display panel 134 of the assistant device 114. However, if the third-party server device 102 does not actively provide state updates to the assistant device 114, the assistant device 114 may not have accurate state data to present to the user 106. In other words, when the metrics corresponding to the third-party server device 102 indicate that the third-party server device 102 is not reliable, the assistant device 114 can at least in response to user input, such as a request for the state of the client device 108, choose to query the third-party server device 102 for an updated state. Alternatively, when the metrics corresponding to the third-party server device 102 indicate that the third-party server device 102 is reliable, the assistant device 114 can choose to rely on locally stored state data and bypass querying the third-party server device 102 for an updated state. In this manner, the metrics will alleviate the need to query the third party server device 102 each time a user 106 requests a status update for a particular client device.

[0026] However, in some embodiments, the assistant server device 104 can determine the latency of the third-party server device 102 in reporting the status changes exhibited by the client device 108. One or more metrics can be generated based on the latency of the third-party server device 102, and the one or more metrics can thereafter be used to determine whether to: proactively query the third-party server device 102 for status updates regarding the client device 108, and / or bypass providing such queries, at least facilitating reliance on the third-party server device 102 to proactively transmit status data to the assistant server device 104.

[0027] For example, in response to the user 106 sliding the graphical control element 138 to change a setting of the client device 108 via the interface 118, the device request 136 can be sent to the third-party server device 102. As a result, the third-party server device 102 can exhibit a certain amount of delay when providing corresponding status updates to the assistant server device 104 and / or the assistant device 114. For example, the delay can be, but is not limited to, a period of time between the user 106 changing the setting of the client device 108 and the assistant server device 104 receiving the updated status data.

[0028] like Figure 1B As shown in , in response to the assistant device 114 and / or the assistant server device 104 not actively receiving the status update from the third-party server device 102, the third-party server device 102 can receive the status request 138 from the assistant server device 104. In other words, based on the metric generated from the delay from the third-party server device 104, the auxiliary server device 104 can actively determine to issue the status request 138 to the third-party server device 102, or wait for the third-party server device 102 to actively send the status update to the auxiliary server device 104. In some embodiments, the metric can be generated based on the delay corresponding to the time period between the time when the device setting of the client device 108 is changed via the third-party application 120 and the time when the auxiliary server device 104 and / or the assistant device 114 has received the status data from the third-party server device 102. Alternatively or in addition, the metric can be generated based on the delay between the time when the third-party server device 102 sent the most recent status update to the auxiliary server device 104 and the time when the third-party server device 102 received the request from the third-party application 120 and / or the user 106. In some embodiments, the state change of the client device 108 can be performed via the client device 110 and / or directly at an interface of the client device 108. For example, the client device 108 can include one or more interfaces that can be controlled via touch input, gesture input, verbal input, and / or any other input that can be provided by the user 106. In some embodiments, the delay from which the metric is generated can be based on a time period between the time when the client device 108 receives a request from the user 106 at an interface physically integrated into the client device 108 and the time when the assistant server device 104 and / or assistant device 114 receives a state update from the third-party server device 102 after the request from the user 106.

[0029] Figure 2A and Figure 2BThe views 200 and 240 of the assistant server device 204 that adapt the reliability of the third-party server device 202 in reporting the state change at the client device 208 are respectively illustrated. The user 206 can control the client device 208 using one or more spoken utterances 238, which can be provided to the automated assistant interface of the assistant device 214. The assistant device 214 can communicate with the client device 208 via at least a local area network such as a Wi-Fi network provided by a Wi-Fi router 216 and / or a wide area network such as the Internet. In response to receiving the spoken utterances 238, the assistant device 214 can generate audio data that can characterize the spoken utterances 238, and provide the audio data to the auxiliary server device 204 for processing. Alternatively or additionally, the assistant device 214 can locally process the audio data. Processing the audio data can include, for example, converting the audio data into text (or other representations), generating at least natural language understanding (NLU) data from the converted representation, and identifying one or more intent requests reflected by the spoken utterances 238 based on the NLU data.

[0030] Based on the processing of the audio data by the assistant server device 204 and / or the assistant device 214, the assistant server device 204 can send a device request 236 to the third-party server device 202. In response to receiving the device request 236, the third-party server device 202 can communicate with the client device 208 to facilitate fulfilling one of the multiple requests from the user 206. For example, the spoken utterance 238 can include natural language content, such as "Assistant, please set my thermostat to 72°." Based on the natural language content, the device request 236 can be recognized as an intent to modify a setting of the client device 208 and specifically modify the temperature setting of the client device 208 (e.g., a thermostat).

[0031] The user 206 can provide the spoken utterance 238 after returning home, entering the room 212 of their home, and seeing the current temperature reading presented at the display panel 234 of the assistant device 214. The assistant device 214 can present an interface 230 of the assistant application, which can provide one or more graphical control elements 232 for controlling one or more settings of the client device 208. In addition, the interface 230 can present status data corresponding to the client device 208. For example, before the user 206 provides the spoken utterance 238, the interface 230 of the assistant application can indicate that the settings of the client device 208 are set to 65 degrees. The status data can be based on data provided by the third-party server device 202 before the user provides the spoken utterance 238. The assistant server device 204 and / or the assistant device 214 can determine one or more metrics that characterize the reliability of the third-party server device 202 and / or the client device 208 in providing such status data updates. Based on these one or more metrics, assistant server device 204 and / or assistant device 214 can determine whether to actively request status data from third-party server device 202 and / or wait for third-party server device 202 and / or client device 208 to provide status updates.

[0032] In some implementations, one or more metrics can be based on a delay exhibited by the third party server device 202 when reporting a status update after a user has provided a spoken utterance 238. The spoken utterance can be provided to facilitate controlling a third party client device 208 in communication with the third party server device 202. For example, the amount of delay can be based on an amount of time delay between when the assistant device 214 receives the spoken utterance 238 from the user 206 and when the assistant device 214 receives the status update corresponding to the spoken utterance 238. Alternatively or in addition, the amount of delay can be based on a time delay between when the assistant server device 204 provides the device request 236 to the third party server device 202 and when the assistant server device 204 receives the status data 242 from the third party server device 202.

[0033] In some embodiments, one or more metrics characterizing the reliability of a third-party server device 202 and / or a client device 208 can be based on one or more measurements of delay and / or a comparison of the delay with one or more reference values. For example, an automated assistant and / or an assistant server device 204 can interact with various third-party server devices 202 corresponding to various third-party entities. Each third-party entity can cause a certain amount of delay at their respective third-party server devices and / or their respective client devices. In some embodiments, a reference value can characterize the average amount of delay caused by one or more third-party entities. Therefore, a metric value can indicate a higher reliability of a specific third-party server device that exhibits less delay than the average delay of one or more third-party entities. In addition, another metric value can indicate a lower reliability of a specific third-party server device that exhibits more delay than the average delay of one or more third-party entities. In some embodiments, a reference value can be based on a plurality of different types of delay characteristics (e.g., each delay characteristic can characterize one or more different types of interactions involving a third-party server device and / or a third-party client device, characterize the type of device, characterize the time of day) or a single type of delay measurement.

[0034] When the third-party server device 202 corresponds to a metric indicating lower reliability relative to other third-party entities, the assistant server device 204 can provide a status request 244 to the third-party server device 202 after providing the device request 236. Alternatively or in addition, when the third-party server device 202 corresponds to a different metric indicating higher reliability relative to other third-party entities, the assistant server device 204 can bypass providing the status request 244 to the third-party server device 202 after providing the device request 236. In response to receiving the status request 244, the third-party server device 202 can generate updated status data and provide the status data to the assistant server device 204 and / or the assistant device 214. In response to receiving the updated status data, the assistant server device 204 and / or the assistant device 214 can cause the display panel 234 of the assistant device 214 to render an interface 230 representing the updated status data 242. For example, because the user 206 previously requested that the temperature setting of the client device 208 be set to 72 degrees, the interface 230 can represent the most recent setting based on the updated status data received from the third-party server device 202. In this manner, use of such metrics allows assistant device 214 and / or assistant server device 204 to determine to proactively retrieve updated status data or rely on third party server device 202 to proactively provide updated status data.

[0035] By operating assistant server device 204 and / or assistant device 214 according to one or more metrics characterizing the reliability of different third-party devices and / or entities, computing resources can be reserved. For example, when assistant server device 204 recognizes the reliability of third-party server devices, network bandwidth can be reserved, and thus bypasses sending frequent status update requests. In addition, for those third-party entities that can be indicated as unreliable, at least by actively requesting such data, assistant server device 204 and / or assistant device 214 will have more accurate data about other client devices. As a result, assistant server device 204 and / or assistant device 214 will be able to make more informed decisions about controlling certain client devices. For example, a user can rely on inaccurate data about the state of a client device and send a request to an automated assistant in order to modify the state of a client device. As a result, a request can cause assistant device 214 to implement instructions that cause a specific client device to operate inefficiently, perform redundant operations, and / or otherwise waste electricity and / or other computing resources. By adopting the embodiments provided herein, such wasteful operations can be eliminated.

[0036] Figure 3 A system 300 is shown for characterizing the reliability of various third-party servers and protocols for adapting to the device ecosystem affected by such reliability, at least when reporting third-party device status to an automated assistant 304. The automated assistant 304 can operate as part of an assistant application provided at one or more computing devices such as a client device 318 and / or a server device 302. A user can interact with the automated assistant 304 via an assistant interface, which can be a microphone, a camera, a touch screen display, a user interface, and / or any other device that can provide an interface between a user and an application. For example, a user can initialize the automated assistant 304 by providing verbal, textual, and / or graphical input to the assistant interface 320 to cause the automated assistant 304 to perform a function (e.g., provide data, control peripherals, access an agent, generate input and / or output, etc.). The client device 318 can include a display device, which can be a display panel including a touch interface, which is used to receive touch input and / or actions for allowing a user to control an application of the client device 318 via the touch interface. In some embodiments, the client device 318 can lack a display device, thereby providing an audible user interface output instead of a graphical user interface output. In addition, the client device 318 can provide a user interface such as a microphone for receiving spoken natural language input from the user. In some implementations, the client device 318 can include a touch interface and can lack a camera, but can optionally include one or more other sensors.

[0037] Client device 318 and / or other third-party client devices 318 can communicate with server device 302 via network 340, such as the Internet. Additionally, client device 318 and other computing devices 434 can communicate with each other via a local area network (LAN), such as a Wi-Fi network. Client device 318 can offload computing tasks to server device 302 in order to conserve computing resources at client device 318. For example, server device 302 can host automated assistant 304, and client device 318 can send input received at one or more assistant interfaces 320 to server device 302. However, in some implementations, automated assistant 304 can be hosted at client device 318 as client-side automated assistant 322.

[0038] In various implementations, all or less than all aspects of automated assistant 304 can be implemented on client device 318. In some of those implementations, aspects of automated assistant 304 are implemented via client automated assistant 322 of client device 318, and can interface with server device 302, which can implement other aspects of automated assistant 304. Server device 302 can optionally serve multiple users and their associated assistant applications via multiple threads. In implementations where all or less than all aspects of automated assistant 304 are implemented via client automated assistant 322 at client device 318, client automated assistant 322 can be an application separate from the operating system of client device 318 (e.g., installed “on top” of the operating system) - or can alternatively be implemented directly by the operating system of client device 318 (e.g., an application that is considered to be the operating system, but integrated with the operating system).

[0039] In some implementations, automated assistant 304 and / or client automated assistant 322 can include an input processing engine 306 that can employ a number of different modules for processing input and / or output of client device 318 and / or server device 302. For example, input processing engine 306 can include speech processing engine 308 that can process audio data received at assistant interface 320 to recognize text contained in the audio data. Audio data can be sent from, for example, client device 318 to server device 302 in order to preserve computing resources at client device 318.

[0040] The processing for converting audio data into text can include a speech recognition algorithm, which can use a neural network and / or a statistical model to identify an audio data group corresponding to a word or phrase. The text converted from the audio data can be parsed by a data parsing engine 310 and available to an automated assistant as text data, which can be used to generate and / or identify command phrases, intentions, actions, slot values ​​and / or any other content specified by the user. In some embodiments, the output data provided by the data parsing engine 310 can be provided to a parameter module 312 to determine whether the user has provided an input corresponding to a specific intention, action and / or routine that can be performed by an automated assistant 304 and / or can be accessed via an automated assistant 304 or an application or agent. For example, assistant data 316 can be stored in a server device 302 and / or a client device 318 as assistant device data 316, and can include data defining one or more actions that can be performed by automated assistant 304 and / or client automated assistant 322, and parameters necessary for performing the action.

[0041] In some embodiments, the server device 302, the automated assistant 304, and / or the client device 318 can track the reliability of the third-party server device 350 and / or the third-party client device 336. Specifically, the reliability of the third-party server device 350 and / or the third-party client device 336 can be tracked in terms of delays in reporting status updates, accuracy of specific status updates, and / or any other characteristics that can indicate the reliability of the server device and / or the client device. For example, the user can provide input to the assistant interface 320, which can cause the automated assistant 304 to provide output via the output generation engine 314. The output can be based on data provided by the parameter engine 312 that can process input from the user. The output can be provided to the third-party server device 315 to affect the operation and / or settings of the third-party client device 336. For example, the third-party client device 336 can be one of a plurality of third-party client devices connected to the user's home or otherwise associated with the user. The user can control the third-party client device 336 via the automated assistant 304 by allowing the automated assistant 304 to communicate with the third-party client device 336 and / or the third-party server device 350. The output from the output generation engine 314 can be provided to the third-party server device 350. For example, the output from the server device 302 can characterize the intention requested by the user, such as the alarm system control intention. The input request engine 354 can receive the output from the server device 302, and determine that the output has been provided by the server device 302, and the output is based on the input of the specific user interacting with the corresponding automated assistant. In addition, the client command engine 356 can generate one or more client commands based on the output of the server device 302. Based on one or more processes of the input request engine 354 and / or the client command engine 356, the third-party server device 350 can send one or more commands to the third-party client device 336 that the user intends to control with its input. When the third-party client device 336 receives one or more commands, the command engine 342 of the third-party client device 336 can process one or more commands. Command data can be generated by the command engine 342 and used by the setting engine 344 to determine one or more settings and / or one or more operations to be modified according to one or more commands. Any changes to the settings, operations, and / or any other features of the third-party client device 336 can be characterized in state data stored as local device data 346. When the state data has been generated at the third-party client device 336, the third-party client device 336 can confirm the completion of the execution of one or more commands for the third-party server device 350.

[0042] In some embodiments, when the third-party server device 350 sends one or more commands to the third-party client device 336 based on the input from the user to the automated assistant, the third-party server device 350 can also send a confirmation to the server device 302 and / or the client device 318. The confirmation can indicate to the server device 302 and / or the client device 318 that the third-party server device 350 has acknowledged the input from the user and has acted on the input from the user. Thereafter, the third-party server device 315 can determine that the third-party client device 336 has successfully executed one or more commands, and in response to determining that the third-party client device 336 has successfully executed one or more commands, the third-party server device 350 can update any client device data 358 to characterize the updated operating state of the third-party client device 336. In addition, depending on the third-party entity responsible for the third-party server device 350, the third-party server device 250 can send status data to the server device 302. However, different third-party server devices can provide such status updates according to a variety of different protocols.

[0043] In some implementations, third-party server device 350 can operate according to a protocol in which client state engine 352 of third-party server device 350 provides requests for confirmation that certain commands were executed by third-party client device 336. Based on such confirmations from third-party client device 336, client state engine 352 can generate various sets of state data that can be stored at third-party server device 350 as client device data 358. However, third-party server device 350 may or may not operate according to another protocol for providing state updates to other stakeholders, such as server device 302, automated assistant 304, and / or client device 318.

[0044] The third-party interaction engine 326 of the server device 302 can collect various data about various third-party server devices 350 to facilitate eliminating status inaccuracies and / or delays presented at the client device 318. For example, time data corresponding to interactions between the server device 302 and the third-party server devices 350 can be collected by the third-party interaction engine 326. The third-party interaction engine 326 is capable of identifying a specific time when a request is transmitted from the server device 302, a specific time when a request is received from a user at the client device 318, a specific time when a request is received at the server device 302 from the client device 318, a specific time when the third-party server device 350 receives an instruction from the server device 302, a specific time when the third-party server device 350 sends command data to one or more third-party client devices 336, a specific time when one or more third-party client devices 336 confirms receipt of the command data to the third-party server device 350, a specific time when the third-party server device 350 provides an acknowledgment of receipt of the instruction from the server device 302, a specific time when the third-party server device 350 requests a status update from the third-party client device 336, a specific time when the third-party server device 350 receives status data from the third-party client device 336, a specific time when the server device 302 receives client status data from the third-party server device 250, and / or any other temporal characteristics that can characterize a transaction occurring relative to at least one device.

[0045] The server device 302 can include a reliability engine 324 that can use the temporal data generated by the third-party interaction engine 326 to generate one or more metrics 328 based on one or more temporal characteristics of any transaction discussed herein. The one or more metrics can characterize the reliability of one or more third-party server devices 350, at least in terms of proactively providing status updates to the automated assistant 304, the server device 302, and / or the client device 318. In addition, the third-party interaction engine 326 can use the one or more metrics 328 to make decisions about whether and / or how to interact with a particular third-party server device 350. For example, in response to determining that a user has provided a request to change an operational setting of a third-party client device 336, the server device 302 can determine whether to query the third-party server device 350 immediately after communicating the user request to the third-party server device 350, or wait for the third-party server device 350 to provide updated status data corresponding to the third-party client device 336.

[0046] In some embodiments, the reliability engine 324 can determine and / or generate one or more reference values ​​360, and the third-party interaction engine 326 can compare the reference values ​​360 to the metrics 328. The reference values ​​360 can be based on one or more transactions corresponding to the third-party server device 350 and / or the third-party client device 336. In addition, the reference values ​​360 can indicate the reliability of a particular third-party entity, at least in terms of how reliable the third-party entity is in services that employ active reporting of changes in the state of the server device and / or the client device. In some embodiments, the reference values ​​and / or metrics can characterize the reliability of multiple different third-party entities, the reliability of multiple different third-party servers, the reliability of multiple different third-party client devices, the reliability of certain types of third-party entities, and / or any other third-party entities and / or devices that can interact with the automated assistant.

[0047] As an example, and in some embodiments, the reference value can characterize the average amount of delay exhibited by one or more different third-party server devices when reporting client device status updates after fulfilling a user request to the automated assistant. In other words, when the average delay of multiple different third-party server devices 350 is 1 hour, at least one of the reference values ​​360 is, for example, 1 hour. Therefore, when the automated assistant is interacting with a third-party server device 350 associated with a third-party entity, the third-party interaction engine 326 can access a metric corresponding to the third-party entity and compare the metric with the reference value. When the metric corresponds to an amount of delay less than the reference value or the average, the third-party interaction engine 326 can bypass actively querying the third-party server device 350 for client device status updates. However, when the metric corresponds to an amount of delay greater than the reference value or the average, the third-party interaction engine 326 can actively query the third-party server device 350 for client device status updates. It should be noted that the reference value can correspond to an amount of time less than or greater than microseconds, milliseconds, seconds, hours, days, and / or any other unit of time. For example, the reference value can be, but is not limited to, 500 milliseconds, and thus comparison of the metric to that particular reference can provide a basis from which to query the third party server device 350 for status updates, or to bypass providing querying the third party server device 350 for status updates.

[0048] Figure 4The method 400 for determining whether to query a third-party server for a status update corresponding to a specific third-party client device based on one or more metrics is illustrated. The method 400 can be performed by one or more computing devices, applications, and / or any other device or module that can interact with a third-party server device and / or a third-party client device. The method 400 can include an operation 402 for determining whether updated device status data of a specific device has been received. When updated device status data of a specific device has been received, the method 400 can proceed from operation 402 to operation 404. Operation 404 can include determining a time delay or amount of time from the most recent request to modify the operating state of a specific device. In other words, operation 404 can include determining a time period between the time when the updated device status data of the specific device is received and the time when the most recent request to modify the operating state of the specific device is provided to a third-party server device, a specific client device, and / or any other associated device.

[0049] Method 400 can also include optional operation 406 that determines the accuracy of the stored operating state relative to the updated state data. Optional operation 406 can be periodically determined by a server device and / or a client device communicating with a third-party server device and / or a third-party client device. In some embodiments, optional operation 406 can be determined randomly, and / or determined in response to a request from a user for device status, so as to assess how reliable a specific third-party server device is in actively providing status updates. For example, a user may have modified and / or initialized the operation of a third-party client device using an interface of a third-party client device or a third-party application corresponding to a third-party client device. Depending on how active a third-party entity is in providing status updates, an automated assistant or an assistant-related device may or may not receive any status updates about the user's actions. Therefore, if a server device checks the state of a third-party client device, and the state indicated by the third-party server device is different from the state stored at the server device, the server device can generate data to characterize this inaccuracy of the device state.

[0050] In addition, if the server device checks the state of the third-party client device, and the state indicated by the third-party server device is the same as the state stored at the server device, the server device can also use data to capture this accuracy. In this way, the server device and / or the client device can characterize the probability that the state data for a specific third-party client device stored by the server device and / or the client device is accurate at any given time. Therefore, if the server device determines that the probability that the state data of a specific third-party client device is accurate is high, and the user requests the third-party client device to operate according to the same state data, the server device can bypass requesting the third-party server device to modify the operation of the third-party client device. As a result, this can save network bandwidth and computing resources because many insignificant or otherwise unnecessary transactions between the client and the server will be avoided.

[0051] However, if the server device determines that the probability that the state data of a particular third-party client device is accurate is low, and the server device receives a request from the user to operate the third-party client device according to the same specific state data, the server device can transmit the request to the third-party server device regardless, even though the server device is under the impression that the third-party client device is already operating as requested by the user. In response, the third-party server device can process the request and indicate that the third-party client device is already operating according to the same specific state data. In response, when the third-party client device has not yet operated according to the same specific state data stored by the server device, the third-party server device can process the request and cause the third-party client device to operate to facilitate fulfillment of the request from the user.

[0052] Method 400 can further include an operation 408 of generating and / or modifying one or more metrics corresponding to the reliability of the third-party server device. One or more metrics can be based on the time delay and / or time period determined in operation 404, and / or the accuracy determined in optional operation 406. In other words, in some embodiments, one or more metrics can be based on the amount of delay exhibited by one or more third-party server devices and / or the accuracy determined for a particular third-party server device. In some embodiments, the metric can be one or more different values ​​selected from a range of different values. Alternatively or in addition, the metric can be a discrete value, such as 1 or 0, for example, when the time delay is greater than, equal to, or less than a specific reference value, the metric can be defined as 1. In addition, when the time delay is less than, equal to, or greater than a specific reference value, the metric can be defined as 0. In a non-limiting example, when the time delay is 2 seconds and the reference value is 5 seconds (i.e., when the time delay is less than the reference value), the metric can be assigned 1. Alternatively, in another non-limiting example, when the time delay is 7 seconds and the reference value is 5 seconds (i.e., when the time delay is greater than the reference value), the metric can be assigned 0.

[0053] In some embodiments, one or more metrics can be based on data generated in operation 406 and / or operation 404, and / or any other data that can indicate the reliability of the third-party server device. One or more metrics can be selected from a range of different values, and can optionally be based on one or more reference values. In some embodiments, one or more metrics can be discrete values ​​or binary values, such as 1 or 0, indicating that the third-party server device responsible for providing the status data about one or more third-party client devices is reliable or unreliable in actively providing status data. In some embodiments, one or more metrics can be used to determine whether to query updated status data to a specific third-party server device.

[0054] Method 400 can proceed from operation 408 to operation 402 to determine whether any further updated device status data of a specific device has been received. When in operation 402, no updated status data of a specific device is received at a given time, method 400 can proceed from operation 402 to operation 410. Operation 410 can include determining whether one or more device metrics provide a basis for querying a third-party server device about status data of a specific device. For example, one or more metrics can be compared with one or more reference values ​​to determine whether there is a basis for providing query status data to a third-party server device. In some embodiments, one or more metrics can be processed to determine whether one or more metrics meet a specific threshold, and / or whether one or more metrics are greater than, equal to, and / or less than one or more specific reference values. When one or more metrics do not provide a basis for querying status data to a third-party server device, method 400 can proceed from operation 410 to operation 402. Alternatively, when one or more metrics do provide a basis for providing query status data to a third-party server device, method 400 can proceed from operation 410 to operation 412. By completing method 400, at least completing operations 402, 410 and 412, the assistant server and / or assistant device is able to determine whether and / or when to query the third-party server device for updated operating status data of the client device, at least when no operating status update has been received within a specific time period.

[0055] Operation 412 can include an operation of accessing the state data of a specific device via a third-party server device. In order to access the state data, the server device can send a request to the third-party server device, and in response, the third-party server device can provide the state data back to the requesting server device and / or the corresponding client device. Method 400 further includes an operation 414 of determining whether the automated assistant has received a request to modify the operating state of the specific device. When the automated assistant has not yet received a request to modify the operating state of the specific device, method 400 can proceed from operation 414 to operation 402. However, when the assistant has received a request to modify the operating state of the specific device, method 400 can proceed from operation 414 to operation 416. Operation 416 can include causing the specific device to have a modified operating state according to the request received by the automated assistant. In addition, method 400 can proceed from operation 416 to operation 402, wherein a determination is made as to whether the updated state data of the specific device affected by the request from operation 414 and / or operation 416 has been received.

[0056] The reliability of the third-party server device can be determined to vary over time, such as by exhibiting a reliability pattern corresponding to the time of day or the day of the week. For example, using the technology described herein, the time delay associated with a specific third-party server can be determined to be greater during the evening / morning and / or weekend than at other times of the day (or greater for other times of the day than other times such as evening / morning and / or weekend). Based on these determinations, the first-party server device can modify its behavior regarding actively seeking an operational status update for a specific third-party client device associated with the third-party server. In determining the time of the day / days of the week that the third-party server is satisfactorily reliable in actively providing status updates, the first-party server can reduce (or eliminate) its initiative to request a status update from the third-party server. Accordingly, in determining the time of the day / days of the week that the third-party server is not satisfactorily reliable in actively providing status updates, the first-party server can increase its initiative to request a status update from the third-party server. This can help the first-party server to be efficient in its use of network resources and computing resources at the first-party server and the third-party server, while also maintaining accurate information at the first-party server about the internal operating state of one or more third-party client devices associated with the third-party server.

[0057] Figure 5 5 is a block diagram of an example computer system 510. The computer system 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524, for example including a memory 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computer system 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0058] The user interface input device 522 may include a keyboard, a pointing device such as a mouse, a trackball, a touch pad or a graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computer system 510 or over a communication network.

[0059] User interface output device 520 can include display subsystem, printer, fax machine, or non-visual display such as audio output device.Display subsystem can include cathode ray tube (CRT), flat panel device such as liquid crystal display (LCD), projection device, or some other mechanism for creating visual image.Display subsystem can also provide non-visual display such as via audio output device.Usually, the use of term "output device" is intended to include all possible types of devices and modes for outputting information from computer system 510 to a user or another machine or computer system.

[0060] The storage subsystem 524 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include logic for performing selected aspects of the method 400 and / or for implementing one or more of the third-party server device 102, assistant server device 104, third-party application 120, assistant device 114, client device 110, third-party server device 202, assistant server device 204, assistant device 214, server device 302, client device 318, third-party server device 350, and / or third-party client device 336.

[0061] These software modules are typically executed by the processor 514 alone or in conjunction with other processors. The memory 525 used in the storage subsystem 524 can include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 can provide permanent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments can be stored in the storage subsystem 524 by the file storage subsystem 526, or in other machines accessible by the processor 514.

[0062] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computer system 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0063] Computer system 510 can be of various types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5The description of the computer system 510 depicted in FIG. 5 is intended only as a specific example for the purpose of illustrating some embodiments. Many other configurations of the computer system 510 may have more Figure 5 The computer system depicted in FIG.

[0064] In cases where the systems described herein collect personal information about users (or "participants," as they are often referred to herein) or may use personal information, users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current geographic location) or to control whether and / or how content that may be more relevant to the user is received from a content server. In addition, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. For example, the identity of a user may be processed so that no personally identifiable information can be determined for the user, or the user's geographic location may be summarized as to where the geographic location information was obtained (such as to a city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, users may control how information about users is collected and / or used.

[0065] In some embodiments, a method implemented by one or more processors is described as including operations such as receiving first state data from a third-party server device indicating a state of a third-party client device controllable via an assistant device, wherein the assistant device includes an automated assistant interface, and a user interacts with the automated assistant via the automated assistant interface to control the third-party client device. The method can further include determining a metric based on the amount of time delay between receiving the first state data and a previous time associated with a request for modifying an operating state of the third-party client device based on receiving the first state data. The method can further include, after determining the metric based on the amount of time delay: determining whether to provide a state request to the third-party server device for retrieving second state data indicating a current state of the third-party client device. The method can further include, when the metric indicates not to query the third-party server device for at least some basis of the current operating state of the third-party client device: based on the metric, bypassing providing the state request to the third-party server device. The method can further include, when the metric indicates querying the third-party server device for at least some basis of the current operating state of the third-party client device: based on the metric, providing the state request to the third-party server device, and in response to the third-party server device receiving the state request, receiving second state data characterizing the current operating state of the third-party client device.

[0066] In some of the described embodiments, the method can further include, after determining the metric based on the amount of time delay: determining that the user has provided an action to the interface of the assistant device or another client device to facilitate control of the third-party client device, wherein determining whether to provide the status request is based on determining that the user has provided an action to the interface to facilitate control of the third-party client device. In some embodiments, the action is a tactile action and / or a spoken word, and the interface includes a touch display panel and / or a microphone. In some embodiments, the method can further include generating output data representing a current operating state of the third-party client device in response to receiving the second state data; and causing the output data to be rendered via the assistant device or another client device. In some embodiments, determining the metric based on the amount of time delay includes: modifying a previously generated metric based on the amount of time delay between receiving the first state data and a previous time associated with the state of the third-party client device.

[0067] In some embodiments, modifying a previously generated metric based on a time delay amount includes adapting the previously generated metric to limit a basis for querying a third-party server device for a current operating state of a third-party client device when the time delay amount is less than a reference time delay amount. In some embodiments, modifying a previously generated metric based on a time delay amount includes adapting the previously generated metric to expand a basis for querying a third-party server device for a current operating state of a third-party client device when the time delay amount is greater than a reference time delay amount. In some embodiments, the reference value is based on a time value corresponding to other requests from one or more other users to other third-party server devices to facilitate modifying the operating state of other third-party client devices. In some embodiments, determining the metric includes accessing metric data characterizing the metric and other metrics, and the assistant device communicates with a plurality of other third-party client devices, and one or more of the plurality of other third-party client devices is associated with at least one other metric among the other metrics.

[0068] In some embodiments, the request is based on a spoken utterance provided by a user to the automated assistant interface, and the natural language content of the spoken utterance identifies an automated assistant routine corresponding to a plurality of different actions, and the plurality of different actions include an action for converting the state of the third-party client device to the current operating state. In some embodiments, determining a metric includes accessing another metric, the other metric being based on whether a third-party server device has previously provided a state update to the automated assistant in response to a user requesting a state change of the third-party client device via a third-party hardware interface and / or a third-party application communicating with the third-party client device. In some embodiments, the other metric is further based on another amount of time between the user interacting with the third-party hardware interface and / or the third-party application and the third-party server device providing a state update to the automated assistant and / or the assistant device. In some embodiments, determining whether to query the third-party server device for an indication of the current operating state of the third-party client device is further based on another metric, and the third-party server device is at least partially controlled by a third-party entity, which is different from the entity that at least partially controls the automated assistant.

[0069] In some embodiments, the metric and / or another metric is based on an interaction between one or more other users and one or more other third-party client devices communicating with the third-party server device. In some embodiments, the one or more other third-party client devices are: different from the third-party client device and connected to a network separate from the network to which the third-party client device is connected. In some embodiments, the request is received by the third-party server device in response to a user interacting with a third-party application that controls the third-party client device and / or the third-party server device. In some embodiments, the method can further include generating a graphical user interface element representing the current operating state of the third-party client device in response to receiving the second state data; and causing the graphical user interface of the assistant device or another client device to render the graphical user interface element. In some embodiments, the metric is based on a specific time delay between a first request initiated at the third-party server device to change the operating state of the client device and a second request for the current state from the automated assistant. In some embodiments, the request identifies at least one action of the automated assistant, and at least one action includes an action for converting the state of the third-party client device to the current operating state.

[0070] In some embodiments, a method implemented by one or more processors is described as including operations, such as determining that a user has requested an automated assistant to cause a modification to an operating state of a client device based on processing audio data corresponding to a spoken utterance. In some embodiments, the method can further include providing a request to implement a modification to the operating state of the client device to a server device and / or a client device in response to determining that the user has requested an automated assistant to cause a modification to the operating state of the client device. In some embodiments, the method can further include receiving first state data from a server device and / or a client device based on providing the request to the server device and / or the client device, wherein the first state data characterizes the modified operating state of the client device. In some embodiments, the method can further include determining, based on receiving the first state data, a time delay amount that characterizes a time period between determining that the user has requested an automated assistant to cause a modification to the operating state of the client device and receiving the first state data from the server device and / or the client device. In some embodiments, the method can further include, after determining the time delay amount: determining that the user has subsequently requested the automated assistant to cause a specific modification to the current operating state of the client device. In some embodiments, the method can further include, in response to determining that the user has subsequently requested the automated assistant to cause a specific modification to the current operating state of the client device, providing another request to the server device and / or the client device to implement the specific modification to the current operating state of the client device. In some embodiments, the method can further include, based on the time delay amount and in response to determining that the user has requested the automated assistant to cause a specific modification to the current operating state of the client device, determining whether the time delay amount indicates at least some basis for querying the server device and / or the client device for the second state data. In some embodiments, the method can further include, when the time delay amount indicates at least some basis for not querying the server device and / or the client device for the second state data: based on the time delay amount, bypassing providing the state request to the server device and / or the client device. In some embodiments, the method can further include, when the time delay amount indicates at least some basis for querying the server device and / or the client device for the second state data: based on the time delay amount, providing the state request to the server device and / or the client device, and in response to the server device and / or the client device receiving the state request, receiving the second state data representing the update of the operating state of the client device.

[0071] In some embodiments, the method can further include, after determining the time delay amount: causing the user interface of the separate client device to provide content representing the current operating state of the client device. In some embodiments, the user interface of the separate client device is a display panel and provides graphical content representing the current operating state of the client device. In some embodiments, the method can further include, when the time delay amount indicates at least some basis for querying the server device and / or the client device for the second state data: causing the display panel to render other graphical content representing the update of the operating state of the client device. In some embodiments, the time delay amount represents a specific time period between: providing a request to the server device and / or the client device to implement a modification of the operating state of the client device, and receiving the first state data from the server device and / or the client device.

[0072] In some embodiments, a method implemented by one or more processors is described as including operations such as causing a client device to render content representing a first state of another client device, wherein the client device and the other client device are connected to a public local area network, and the other client device is controlled using an automated assistant accessible at least via the client device. In some embodiments, the method can further include, after causing the client device to render the content, determining that the client device has received a command for causing the other client device to operate according to a second state. In some embodiments, the method can further include, in response to determining that the client device has received a command for causing the other client device to operate according to the second state, providing a request to a third-party server device to cause the other client device to operate according to the second state. In some embodiments, the method can further include, after sending the request to the third-party server device, receiving state data from the third-party server device, wherein the state data represents the operating state of the other client device. In some embodiments, the method can further include, when the state data indicates that the client device is operating according to the second state: at least in response to the server device causing the other client device to present any modified state relative to any previous state of the other client device, modifying or bypassing modification of one or more metrics to characterize the server device as reliable in providing accurate state data. In some embodiments, the method can further include, when the state data indicates that the client device is not operating according to the second state: at least in response to the server device causing another client device to exhibit any modified state relative to any previous state of the other client device, modifying or generating one or more metrics to indicate that the server device is unreliable in providing accurate state data.

[0073] In some embodiments, wherein modifying or generating one or more metrics to indicate that the server device is unreliable in providing accurate state data includes: in response to determining that a subsequent command for causing another client device to operate according to a particular state is received, modifying or generating one or more metrics to indicate a basis for actively querying the server device. In some embodiments, the method can further include modifying or bypassing modifying one or more metrics to characterize the server device as being reliable in providing accurate state data includes: in response to determining that a subsequent command for causing another client device to operate according to a particular state is received, modifying or bypassing modifying one or more metrics to indicate a basis for bypassing actively querying the server device. In some embodiments, causing the client device to render content characterizing the first state of another client device includes causing a graphical user interface of the client device to render one or more graphical elements at the graphical user interface.

[0074] In some embodiments, the method can further include determining a device type that characterizes another client device, wherein modifying or generating one or more metrics is based on the device type that characterizes the other client device. In some embodiments, the method can further include determining a specific time when the request is provided to the third-party server device, wherein modifying or generating one or more metrics is based on the specific time when the request is provided to the third-party server device. In some embodiments, determining that the client device has received the command includes determining a command type for causing the other client device to operate according to the second state, wherein modifying or generating one or more metrics is based on the command type. In some embodiments, determining the command type includes determining whether the command is initialized via an automated assistant, a specific interface of the other client device, or a separate interface of a peripheral device associated with the other client device.

[0075] In some embodiments, the method implemented by one or more processors is described as including operations such as causing a client device to store state data representing another client device, wherein the client device and the other client device are connected to a public local area network, and the other client device is controlled using an automated assistant accessible at least via the client device. The method can further include providing a server request to determine the current operating state of the other client device to a server device after storing the state data at the client device. In some embodiments, the method can further include providing a client request to determine the current operating state of the other client device to another client device after storing the state data at the client device. In some embodiments, the method can further include receiving server state data from a server device based on sending a server request to the server device, wherein the server state data represents a specific state of another client as indicated by the server device. In some embodiments, the method can further include receiving client state data from another client device based on sending a client request to another client device, wherein the client state data represents another specific state of another client as indicated by another client device. In some implementations, the method can further include, when the server state data and the client state data indicate a common operational state of the other client device: causing the client device to store updated state data characterizing the common operational state of the other client device.

[0076] In some embodiments, the method can further include, when the server state data and the client state data indicate a common operating state of another client device: modifying or bypassing modification of one or more metrics to characterize the server device as reliable in providing accurate state data. In some embodiments, the method can further include, when the server state data and the client state data fail to indicate a common operating state of another client device: causing the client device to store other updated state data characterizing another specific state of another client as indicated by the other client device. In some embodiments, the method can further include, when the server state data and the client state data fail to indicate a common operating state of another client device: modifying or generating one or more metrics to indicate that the server device is unreliable in providing accurate state data. In some embodiments, the method can further include determining a device type characterizing another client device, wherein modifying or generating one or more metrics is based on the device type characterizing another client device. In some embodiments, the method can further include determining a specific time when the server request is provided to the server device, wherein modifying or generating one or more metrics is based on the specific time when the server request is provided to the server device. In some embodiments, the method can further include determining an amount of time between a first timestamp corresponding to the state of the other client device and a second timestamp corresponding to the server state data received from the server device.

[0077] Although several described methods have been described and illustrated herein, various other devices and / or structures for performing the functions described herein and / or obtaining the results and / or one or more advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications using this teaching. Those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein using no more than routine experiments. Therefore, it should be understood that the aforementioned embodiments are presented only by way of example, and within the scope of the appended claims and their equivalents, the embodiments may be practiced in a manner different from that specifically described and claimed. The embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if these features, systems, articles, materials, kits, and / or methods are not mutually contradictory, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors, the method comprising: receiving, from a third-party server device, first status data indicating a status of a third-party client device controllable via an assistant device, wherein the assistant device includes an automated assistant interface via which a user interacts with an automated assistant in order to control the third-party client device; based on receiving the first state data, determining a metric based on an amount of time delay between receiving the first state data and a previous time associated with a request to modify an operational state of the third party client device; After determining the metric based on the amount of time delay: determining whether to provide a status request for retrieving second status data indicating a current status of the third-party client device to the third-party server device; When the metric indicates not to query the third-party server device for at least some benchmarks of the current operating state of the third-party client device: Based on the metric, bypassing providing the status request to the third party server device; as well as When the metric indicates querying the third-party server device for at least some benchmarks of the current operating state of the third-party client device: providing the status request to the third party server device based on the metric, and In response to the third-party server device receiving the status request, the second status data characterizing the current operating status of the third-party client device is received.

2. The method according to claim 1, further comprising: After determining the metric based on the amount of time delay: determining that a user has provided an action to an interface of the assistant device or another client device to facilitate controlling the third-party client device; Wherein, determining whether to provide the status request is based on determining that the user provided the action to the interface to facilitate controlling the third-party client device.

3. The method according to claim 2, wherein: The actions are tactile actions and / or spoken words, and the interface includes a touch display panel and / or a microphone.

4. The method according to claim 1, further comprising: In response to receiving the second state data, generating output data representing a current operational state of the third-party client device; as well as The output data is caused to be rendered via the assistant device or another client device.

5. The method according to claim 1, wherein: Determining the metric based on the amount of time delay comprises: A previously generated metric is modified based on an amount of time delay between receiving the first status data and the previous time associated with the status of the third party client device.

6. The method according to claim 5, wherein: Modifying the previously generated metric based on the time delay amount includes adapting the previously generated metric to limit a basis for querying the third party server device for the current operating status of the third party client device when the time delay amount is less than a reference time delay amount.

7. The method according to claim 6, wherein: Modifying the previously generated metric based on the time delay amount includes adapting the previously generated metric to expand a basis for querying the third-party server device for the current operating status of a third-party client device when the time delay amount is greater than the reference time delay amount.

8. The method according to claim 6, wherein: The reference time delay amount is based on time values ​​corresponding to other requests from one or more other users to other third-party server devices to facilitate modifying the operating state of other third-party client devices.

9. The method according to claim 1, in, Determining the metric includes accessing metric data characterizing the metric and other metrics, and Wherein, the assistant device communicates with a plurality of other third-party client devices, and one or more of the plurality of other third-party client devices are associated with at least one of the other metrics.

10. The method according to claim 1, in, The request is based on a spoken utterance provided by a user to the automated assistant interface, and the natural language content of the spoken utterance identifies an automated assistant routine corresponding to a plurality of different actions, and The multiple different actions include an action for converting the state of the third-party client device to the current operating state.

11. The method according to claim 1, further comprising: in, Determining the metric includes accessing another metric based on whether the third-party server device has previously provided a status update to the automated assistant in response to the user requesting a state change of the third-party client device via a third-party hardware interface and / or a third-party application that communicates with the third-party client device.

12. The method according to claim 11, wherein: The another metric is further based on another amount of time between the user interacting with the third-party hardware interface and / or the third-party application and the third-party server device providing the status update to the automated assistant and / or the assistant device.

13. The method according to claim 11, in, determining whether to query the third party server device for an indication of a current operational status of the third party client device is further based on the other metric; as well as Wherein the third party server device is at least partially controlled by a third party entity, and the third party entity is different from the entity that at least partially controls the automated assistant.

14. The method according to claim 11, wherein: The metric and / or the further metric is based on interactions between one or more other users and one or more other third-party client devices in communication with the third-party server device.

15. The method according to claim 14, wherein: The one or more other third-party client devices: Different from the third-party client device and connected to a network separate from the network to which the third-party client device is connected.

16. The method according to any one of claims 1 to 15, in, The request is received by the third party server device in response to the user interacting with a third party application that controls the third party client device and / or the third party server device.

17. The method according to any one of claims 1 to 15, further comprising: In response to receiving the second state data, generating a graphical user interface element representing a current operational state of the third-party client device; as well as A graphical user interface of the assistant device or another client device is caused to render the graphical user interface element.

18. The method according to any one of claims 1 to 15, wherein: The metric is based on a time delay between a first request initiated at the third-party server device to change an operational state of the client device and a second request from the automated assistant for a current state.

19. The method according to any one of claims 1 to 15, wherein: The request identifies at least one action for the automated assistant, and The at least one action includes an action for converting the state of the third-party client device to the current operating state.

20. A method implemented by one or more processors, the method comprising: determining, based on processing the audio data corresponding to the spoken utterance, that the user has requested the automated assistant to cause a modification to an operational state of the client device; responsive to determining that the user has requested the automated assistant to cause the modification to the operational state of the client device, providing a request to a server device and / or the client device to effectuate the modification to the operational state of the client device; receiving first state data from the server device and / or the client device based on providing the request to the server device and / or the client device, wherein the first state data characterizes a modified operational state of the client device; determining, based on receiving the first state data, an amount of time delay characterizing a period of time between determining that the user has requested the automated assistant to cause the modification to the operational state of the client device and receiving the first state data from the server device and / or the client device; After determining the time delay amount: determining that the user has subsequently requested the automated assistant to cause a modification to a current operational state of the client device; responsive to determining that the user has subsequently requested the automated assistant to cause a modification to the current operating state of the client device, providing to the server device and / or the client device another request to effectuate the modification to the current operating state of the client device; determining, based on the amount of time delay and in response to determining that the user has requested the automated assistant to cause a modification of the current operational state of the client device, whether the amount of time delay indicates at least some basis for querying the server device and / or the client device for second state data; When the amount of time delay indicates that at least some reference of the second state data is not queried from the server device and / or the client device: Based on the amount of time delay, bypassing providing a status request to the server device and / or the client device; and When the amount of time delay indicates querying the server device and / or the client device for at least some basis for the second state data: providing the status request to the server device and / or the client device based on the amount of time delay, and In response to the server device and / or the client device receiving the status request, the second status data representing an update of the operational status of the client device is received.

21. The method according to claim 20, further comprising: After determining the time delay amount: A user interface of a separate client device is caused to provide content representative of the current operating state of the client device.

22. The method according to claim 21, in, the user interface of the separate client device is a display panel and provides graphical content representative of the current operating state of the client device, and Wherein, the method further comprises: When the amount of time delay indicates querying the server device and / or the client device for at least some basis for the second state data: The display panel is caused to render other graphical content representative of the operational status update of the client device.

23. The method according to any one of claims 20 to 22, wherein: The amount of time delay characterizes a period of time between providing a request to the server device and / or the client device to effectuate a modification of the operational state of the client device and receiving the first state data from the server device and / or the client device.

24. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 23.

25. A computer-readable storage medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 23.

26. A system comprising one or more processors for executing the method of any one of claims 1 to 23.

Images