Adapting differences in device status reports for third party servers

Abstract

The invention relates to adapting differences in device status reports for third party servers. Embodiments herein relate to information describing one or more internal states of a technical system. The ways described herein are provided for characterizing the reliability of various different third-party servers at least when reporting third-party device status, as well as protocols that accommodate device ecosystems affected by such reliability. The delay can affect the accuracy of the device state represented by the assistant device. When an updated device state is reported in response to a user request, certain servers can be characterized as particularly delayed, and as a result, third-party servers can be correlated with metrics characterizing the relative latency of the third-party servers. When the metric cannot meet a particular threshold, a server and / or client associated with the'ecosystem 'of the third-party device can positively operate to retrieve device state updates rather than passively waiting for updates from the corresponding third-party server.

Description

[0001] Division Explanation

[0002] This application is a divisional application of Chinese Patent Application No. 201980089841.5 with an application date of February 8, 2019. Background Art

[0003] Humans can participate in human-computer conversations with interactive software applications referred to herein as "automation assistants" (also known as "digital agents", "chatbots", "interactive personal assistants", "intelligent personal assistants", "assistant applications", "conversational agents", etc.). For example, humans (who can be referred to as "users" when they interact with an automation assistant) can provide commands and / or requests to an automation assistant using oral natural language input (i.e., spoken utterances) and / or by providing text (e.g., typed) natural language input, and the oral natural language input can in some cases be converted to text and then processed. The automation assistant can respond to requests by providing responsive user interface output, which can include audible user interface output and / or visual user interface output.

[0004] In many cases, a user is able to provide verbal utterances to cause an automated assistant to perform various actions to facilitate control of a particular device. Some devices can become available through a third-party entity, which can exhibit various levels of reliability in regularly providing operational data to the user's automated assistant. For example, a third-party entity that unreliably pushes device status updates to the server of the automated assistant may cause inaccurate status data to be provided to the user and cause the user to erroneously rely on the inaccurate status data. Thus, a user who always relies on the interface of their automated assistant device to determine the status of some devices may be presented with information that inaccurately represents one or more internal states of the device and / or one or more internal states of the broader technical system in which the device resides. The user may inadvertently keep some devices running or not running based on such inaccurate status data, resulting in consumption of undesired resources. Additionally, the user may deploy inconsequential requests to modify the settings of a device whose settings have already been appropriately modified, thereby wasting network bandwidth and processing resources. For example, a user who relies on their automated assistant to determine the operational status of their lights or furnace may be unaware of the most recent operational data that is otherwise available at a third-party server. Such recent operational data can cause the user to be presented with inaccurate information regarding the internal state of the lights and / or furnace. Thus, the user may cause certain requests to be sent to the third-party server for controlling the lights and / or furnace based on the inaccurate status data. For example, the user can issue a verbal command to their automated assistant to turn off a group of lights in their living room, even though the group of lights may already be off. Thus, the verbal command will still be processed and, for example, sent to the automated assistant server and the third-party server, which may then reject the command in view of the fact that the lights are already off. Such inconsequential transactions that occur on many devices can result in waste of network traffic and power consumption. SUMMARY OF THE INVENTION

[0005] The embodiments described herein relate to information describing one or more internal states of a technical system. The embodiments described herein are provided for characterizing the reliability of various third-party servers, at least in reporting third-party device status to an automated assistant, a client device, and / or another server device. Additionally, some embodiments relate to protocols adapted to eliminate the negative impact of such latency on the automated assistant and / or the corresponding device ecosystem. A particular ecosystem of a network of computing devices, such as those connected to a local area network, can exhibit various different states over the operational lifetime of the devices. Some ecosystems can include at least one assistant device through which a user can control various devices within their ecosystem by providing requests to an automated assistant that is accessible via the assistant device. When modifying the state of a device within the ecosystem according to a request and / or other command, the automated assistant can track the state of the device. For example, in order for an automated assistant to ascertain the state of a particular device, such as a smart light bulb in a home, a third-party server associated with the smart light bulb can provide a status update to a first-party server associated with the automated assistant (e.g., a first-party server relative to the automated assistant and the third-party smart light bulb).

[0006] However, differences in the reliability of third-party servers that timely provide status updates reflecting the accurate state of a device can affect the accuracy of the status information provided by the automated assistant to one or more users. For example, a particular smart device manufacturer may not provide a status update (or may provide a significant delay) to the first-party server and / or the assistant device in response to a user manually controlling the smart device (e.g., turning off the light switch of a smart light bulb) without assistance from the automated assistant. As a result, if a user subsequently requests the status of all of their devices within their respective ecosystem, the status of a particular smart device may be inaccurately presented. Such inaccuracies can result in a waste of energy and / or computing resources. For example, a user who has manually adjusted a smart light bulb and then checks the status (i.e., state) of the smart light bulb before going to bed may be provided with inaccurate and misleading status information regarding the smart light bulb and may thus not realize that the smart light bulb remains on throughout the night. If the user had been provided with more accurate status information, the user would have been able to turn off the bulb before going to sleep via the assistant device, thereby conserving energy resources at home and / or any other affected devices. Alternatively, when the state of the smart light bulb is indicated as off but the light is actually on, the user may provide a command to turn on their smart light bulb. Such a command provided by the user can result in a waste of computing and / or network resources because at least speech-to-text processing and / or network transmission will need to be done and only then can the corresponding server device reject the command in case the light is already on.

[0007] The embodiments discussed herein allow an automated assistant to proactively request the device state of devices within an ecosystem based on one or more different metrics representing the reliability of one or more server devices and / or other support systems associated with those devices. The support systems can represent server devices and / or client devices that are responsible for managing the device state of the devices and reporting those device states to other servers, clients, applications, and / or any other device or module. In some embodiments, a latency-based metric can be generated for one or more support systems corresponding to one or more devices that can be controlled via the automated assistant. The latency-based metric can be generated using time data that represents the amount of time that elapses between a request to change the device state (e.g., from the automated assistant) and the reporting of the updated device state to the automated assistant, a first-party server device, and / or a client device associated with the automated assistant. The latency-based metric can then be used, for example, to make a decision on whether to proactively request an updated device state when a third-party server is generally unreliable in proactively delivering updated states to the automated assistant. Additionally, the latency-based metric can also be used to make a decision on whether or when not to proactively request an updated device state, for example, when a third-party server has proven to be reliable in proactively reporting updated device states to the automated assistant.

[0008] In some embodiments, one or more metrics can be generated for determining how positively a particular server device reports a change in device state. For example, smart devices connected within the ecosystem are controllable via the automated assistant and third-party applications. If a user chooses to control the particular smart device via a third-party application, the server device can receive a request from the third-party application and cause a change in the state of the device. However, the third-party server may not diligently provide the updated state to the automated assistant. For example, while some third-party servers may immediately proactively provide a status update to the automated assistant in response to a third-party application causing a change in device state, some third-party servers may not be proactive. For example, some third-party servers may not provide a device state update to the automated assistant and / or may provide a device state update after a certain amount of latency. One or more metrics for the third-party server can be generated based on such omission and / or latency exhibited by the third-party server.

[0009] In some embodiments, a metric in one or more metrics can be determined based on the probability that a status update will be provided to the automated assistant proactively by a particular third-party server and / or the probability that the automated assistant will have accurate status data (as provided by a third-party client device and / or a particular third-party server device) at any given time. When a particular metric does not meet a threshold, the automated assistant can choose to query the third-party server more frequently for status updates of certain devices. Alternatively, when a particular metric meets a particular threshold, the automated assistant can bypass querying the third-party server for status change updates, at least given that the third-party server has a history of proactively providing such device status updates.

[0010] In some embodiments, one or more metrics can affect various different operations performed by the automated assistant. For example, a user can provide gestures (e.g., tap or swipe gestures) at a touch display panel of an assistant device to cause the assistant device to present the operational states of a plurality of different devices associated with the user's "ecosystem". The operational states can be presented as graphical display elements characterizing the operational states of the devices (e.g., a graphic of a light bulb depicting bright illumination to indicate that a particular smart bulb is on; another graphic of a padlock depicting a closed position to indicate that an alarm system is armed; and yet another graphic depicting a thermometer and indicating the current temperature of a room in the user's home). In response to receiving an input gesture, the assistant device and / or a corresponding server device can cause one or more metrics to be accessed. Each metric in the one or more metrics can correspond to a particular device (e.g., an alarm system, a thermostat, and a light). Additionally, in response to the gesture, each metric can be used to determine whether to query the third-party server for the current status of each particular device or to bypass querying the third-party server for the current status of each particular device (at least assuming that the value of the metric provides some basis (e.g., meets a threshold), or does not provide some basis, from which it is assumed that the third-party server device supporting the particular device is reliable in proactively providing updates on changes in the operational state of the particular device). Thus, in some instances, the assistant device can query one or more third-party server devices based on one or more metrics and / or bypass querying one or more third-party server devices based on one or more queries.

[0011] As another example, in some embodiments, the automated assistant is capable of executing routines, which can be a combination of operations performed according to the settings of the routine. The routine can be, for example, a "good night" routine, which can be established by the user to combine various automated assistant operations that are typically requested by the user before going to sleep into a single request. The operations corresponding to the "good night" routine can include, for example, one or more of the following: (1) turning off various smart lights in the home, (2) turning on the home security alarm, and (3) playing a certain amount of audio that the user typically likes to listen to while in bed. During the execution of the routine, the automated assistant can access device status data corresponding to the security alarm and the smart lights. The accessed device status data can be the current device status data maintained by the automated assistant based on previous communications with a third-party server. In addition, the automated assistant can access one or more metrics corresponding to the server device for the smart bulb and the server device for the alarm system. The device status data can indicate that the electric lights have been turned off and the security alarm has been activated. However, the automated assistant can determine whether to query each server device to confirm whether the status is correct based on one or more metrics. For example, the metric of the server device for the smart bulb can meet a threshold, and based on (e.g., in response to) the metric that meets the threshold, the automated assistant can bypass querying the server device for the status of the bulb. In other words, the automated assistant can determine that a particular server device has a history of actively providing status updates consistently and / or in a timely manner, and thus, the metric can indicate that the data representing the device status is accurate.

[0012] However, in some instances, the metrics corresponding to the server device supporting the device and / or system (e.g., the alarm system) may not meet the threshold. Therefore, based on (e.g., in response to) the metric associated with the alarm system not meeting the threshold, the automated assistant can actively query the corresponding server device to confirm whether the device status data is accurate. When the device status is inaccurate and thus the alarm system is determined to have an inaccurate status, the automated assistant can undertake the operation of requesting a change in the status of the security alarm from the corresponding server device during the execution of the routine. Optionally, and based on (e.g., in response to) one or more metrics, the automated assistant can subsequently request status verification of the alarm system from the server device. In other words, because the server device may not have a history of consistently and / or timely positively providing device status updates after a request for a change in device status, the automated assistant can actively confirm that the device status change has actually taken effect.

[0013] Over time, one or more previously generated metrics can be modified based on changes in the pattern of how an active specific third-party server device reports device status. For example, the third-party server device can become more active over time, or the third-party server device can become less active over time. Factors that can affect this can include: changes in the total network traffic to the specific third-party server device, an increase in user usage of the device and / or account, how software upgrades of the third-party server are implemented, expansion or reduction of available network servers, pushed updates, lack of support for previous versions of certain third-party devices, and / or any other factors that can affect how active the server device is in reporting device status. As an example, a metric for a specific third-party server device may initially not meet a specific threshold, but, due to certain enhancements released by a third-party entity, the third-party server device can become more active over time. As a result, the first-party server, automated assistant, and / or any other device responsible for tracking such a metric can modify the corresponding metric for the third-party server device to reflect the trend of the third-party server device becoming more active. This can enable the automated assistant to reduce the frequency of status requests provided to the third-party server device, at least in part due to the metric reflecting the reliability of the third-party server device reporting device status. Additionally, this can eliminate waste of network resources that might otherwise be consumed by indiscriminately provided device status requests, at least relative to the latency and / or accuracy of any previously provided device status changes.

[0014] In some embodiments, one or more metrics and / or one or more reference values can be generated corresponding to a specific device and a specific context of the specific device (e.g., time of day, type of transaction, type of assistant device, type of third-party server device, type of third-party client device, specific user associated with the specific device and / or command). As an example, a first metric for a specific device can characterize the reliability of the third-party server device supported during the weekend (or any time range), and a second metric for the specific device can characterize the reliability of the third-party server device supported during a weekday (or any other time range). In this way, when the assistant device is responsible for presenting the operational status of the specific device, the assistant device and / or the assistant server device can identify the current context and subsequently select a metric based on the current context (e.g., a specific time of day). The selected metric can then be used to determine whether to query the supporting third-party server device for updated status data or to determine that the supporting third-party server device is reliable in proactively providing such status updates.

[0015] It will be apparent from the above discussion that the embodiments described herein can enable the information presented by an automated assistant device to more accurately represent the internal state of a technical system, such as the state of a technical system including one or more of a smart light bulb, a furnace, or an electrical safety alarm (only giving some examples). This information can assist a user in performing technical tasks, including those technical tasks associated with the technical system whose internal state is represented by the presented information. An example of such a task is controlling the technical system to assume or remain in a particular internal state. The provision of inaccurate status data by the automated assistant device to the user via the user interface of the automated assistant device can be reduced or avoided. In addition, when implementing operations in the technical system, the reliance of the automated assistant device on such inaccurate status data can also be reduced or avoided.

[0016] The above description is provided as an overview of some embodiments of the present disclosure. Further descriptions of these and other embodiments are provided below in more detail.

[0017] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform one or more methods such as those described above and / or elsewhere herein. Other embodiments may include a system including one or more computers and / or one or more robots including one or more processors operable to execute the stored instructions to perform one or more methods such as those described above and / or elsewhere herein.

[0018] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are expected to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter that appear at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1A and Figure 1B A view illustrating an auxiliary server device adapting to changes in the reliability and / or reliability mode of a third-party server device when reporting a status update of a particular client device.

[0020] Figure 2A and Figure 2B A view illustrating an assistant server device adapting to the reliability of a third-party server device in reporting a status change to a client device providing access to an automated assistant.

[0021] Figure 3A system is illustrated for characterizing the reliability of various different third-party servers and protocols for adapting a device ecosystem affected by such reliability, at least when reporting the status of third-party devices.

[0022] Figure 4 A method is illustrated for determining whether to query a third-party server for a status update corresponding to a particular third-party client device based on one or more metrics.

[0023] Figure 5 is a block diagram of an example computer system. Detailed Description

[0024] Figure 1A and Figure 1B Views 100 and 140 are respectively illustrated when reporting a status update of a particular client device 108 (e.g., a thermostat), where an assistant server device 104 adapts to changes in the reliability and / or reliability mode of a third-party server device 102. The client device 108 can be connected to a local area network provided by a wireless network source 116 such as a Wi-Fi router. Other client devices can also be connected to the local area network, such as a tablet device (and / or any computing device (i.e., client device 110)) and an assistant device 114, which can communicate with the assistant server device 104. The client device 110 can be controlled by a user 106 and can include a third-party application 120. The user 106 can control the client device 108 through the third-party application 120.

[0025] The user 106 can interact with the third-party application 120 via an interface 118 of the third-party application 120 to control the client device 108. The interface 118 can be presented at a display panel 128 of the client device 110 and can present controls to the user for modifying the operating state of the client device 108. For example, the third-party application 120 can present a first control interface 124 and a second control interface 126. The first control interface 124 can include various graphical control elements through which the user 106 can modify one or more operations and / or settings of the client device 108. The second control interface 126 can include one or more graphical control elements from which the user 106 can control one or more operations and / or settings of the client device 108.

[0026] For example, when the client device 108 is a thermostat, the interface 118 can provide graphical control elements, and the user 106 can change the temperature setting of the client device 108 from the graphical control elements. The interface 118 can also include a current temperature reading (e.g., 65 degrees), which can present the current temperature in the room 112 where the user is located. The current temperature of the room 112 can be 65 degrees, and the user can provide an input to the graphical control element of the interface 118 to adjust the temperature of the room 112 to 72 degrees. In response to the user 106 providing input to or otherwise interacting with the interface 118, the third-party application 120 can send a device request 136 to the third-party server device 102 over the local area network. The device request 136 can cause a change in the operating state of the client device 108 to effect the setting change requested by the user 106 via the third-party application 120.

[0027] The third-party server device 102 can be responsible for updating the server device and / or the client device with the change in the operating state of the client device 108. For example, the user 106 can access the assistant device 114 to determine the state of the client device 108. Based on the status data available to the assistant device 114 and / or the auxiliary server device 104, the state of the client device 108 can be presented at the display panel 134 of the assistant device 114. However, if the third-party server device 102 does not proactively provide status updates to the assistant device 114, the assistant device 114 may not have accurate status data to present to the user 106. In other words, when a metric corresponding to the third-party server device 102 indicates that the third-party server device 102 is unreliable, the assistant device 114 can, at least in response to a user input, such as a request for the status of the client device 108, select to query the third-party server device 102 for an updated status. Alternatively, when a metric corresponding to the third-party server device 102 indicates that the third-party server device 102 is reliable, the assistant device 114 can select to rely on locally stored status data and bypass querying the third-party server device 102 for an updated status. In this way, the metric will alleviate the need to query the third-party server device 102 every time the user 106 requests a status update for a particular client device.

[0028] However, in some embodiments, the assistant server device 104 can determine the latency of the third-party server device 102 when reporting a change in the state exhibited by the client device 108. One or more metrics can be generated based on the latency of the third-party server device 102, and the one or more metrics can thereafter be used to determine whether to: proactively query the third-party server device 102 for a status update regarding the client device 108, and / or bypass providing such a query, at least facilitating reliance on the third-party server device 102 to proactively transfer status data to the auxiliary server device 104.

[0029] For example, in response to a user 106 sliding a graphical control element 138 to change a setting of the client device 108 via the interface 118, a device request 136 can be sent to the third-party server device 102. As a result, when providing a corresponding status update to the assistant server device 104 and / or the assistant device 114, the third-party server device 102 can exhibit a certain amount of latency. For example, the latency can be, but is not limited to, the time period between the user 106 changing the setting of the client device 108 and the assistant server device 104 receiving the updated status data.

[0030] As Figure 1B shown, in response to the assistant device 114 and / or the assistant server device 104 not actively receiving a status update from the third-party server device 102, the third-party server device 102 can receive a status request 138 from the assistant server device 104. In other words, based on a metric generated from the latency of the third-party server device 104, the assistant server device 104 can actively determine to issue a status request 138 to the third-party server device 102, or wait for the third-party server device 102 to actively send a status update to the assistant server device 104. In some embodiments, the metric can be generated based on a latency corresponding to the time period between a device setting of the client device 108 being changed via the third-party application 120 and the assistant server device 104 and / or the assistant device 114 having received status data from the third-party server device 102. Alternatively or additionally, the metric can be generated based on the latency between the time the third-party server device 102 sent a most recent status update to the assistant server device 104 and the time the third-party server device 102 received a request from the third-party application 120 and / or the user 106. In some embodiments, a status change of the client device 108 can be performed via the client device 110 and / or directly at the interface of the client device 108. For example, the client device 108 can include one or more interfaces that can be controlled via touch input, gesture input, voice input, and / or any other input that can be provided by the user 106. In some embodiments, the latency from which the metric is generated can be based on the time period between the time a request from the user 106 is received at an interface physically integrated into the client device 108 of the client device 108 and the time the assistant server device 104 and / or the assistant device 114 receive a status update from the third-party server device 102 after the request from the user 106.

[0031] Figure 2A and Figure 2BViews 200 and 240 are respectively illustrated showing the reliability of assistant server device 204 in reporting a state change at client device 208 to a third-party server device 202. User 206 can use one or more spoken utterances 238 to control client device 208, and the spoken utterances can be provided to the automated assistant interface of assistant device 214. Assistant device 214 can communicate with client device 208 via at least a local area network such as a Wi-Fi network provided by Wi-Fi router 216 and / or a wide area network such as the Internet. In response to receiving spoken utterance 238, assistant device 214 can generate audio data that can characterize spoken utterance 238 and provide the audio data to auxiliary server device 204 for processing. Alternatively or additionally, assistant device 214 can locally process the audio data. Processing the audio data can include, for example, converting the audio data into text (or other representation), generating at least natural language understanding (NLU) data from the converted representation, and identifying one or more intent requests reflected by spoken utterance 238 based on the NLU data.

[0032] Based on the processing of the audio data by assistant server device 204 and / or assistant device 214, assistant server device 204 can send a device request 236 to third-party server device 202. In response to receiving device request 236, third-party server device 202 can communicate with client device 208 to facilitate fulfilling one of a plurality of requests from user 206. For example, spoken utterance 238 can include natural language content such as "Assistant, please set my thermostat to 72°" (°). Based on this natural language content, device request 236 can be made to identify the intent to modify the settings of client device 208 and specifically modify the temperature setting of client device 208 (e.g., the thermostat).

[0033] User 206 can provide verbal utterance 238 after arriving home, entering room 212 of their home, and seeing the current temperature reading presented at display panel 234 of assistant device 214. Assistant device 214 can present an interface 230 of an assistant application, which can provide one or more graphical control elements 232 for controlling one or more settings of client device 208. Additionally, interface 230 can present status data corresponding to client device 208. For example, before user 206 provides verbal utterance 238, interface 230 of the assistant application can indicate that the settings of client device 208 are set to 65 degrees. This status data can be based on data provided by third-party server device 202 before the user provides verbal utterance 238. Assistant server device 204 and / or assistant device 214 can determine one or more metrics characterizing the reliability of third-party server device 202 and / or client device 208 in providing such status data updates. Based on these one or more metrics, assistant server device 204 and / or assistant device 214 can determine whether to actively request status data from third-party server device 202 and / or wait for third-party server device 202 and / or client device 208 to provide a status update.

[0034] In some embodiments, one or more metrics can be based on the latency exhibited by third-party server device 202 when reporting a status update after the user has provided verbal utterance 238. The verbal utterance can be provided to facilitate control of third-party client device 208 that communicates with third-party server device 202. For example, the amount of latency can be based on the time delay between when assistant device 214 receives verbal utterance 238 from user 206 and when assistant device 214 receives the status update corresponding to verbal utterance 238. Alternatively or additionally, the amount of latency can be based on the time delay between when assistant server device 204 provides device request 236 to third-party server device 202 and when assistant server device 204 receives status data 242 from third-party server device 202.

[0035] In some embodiments, one or more metrics characterizing the reliability of the third-party server device 202 and / or the client device 208 can be based on one or more measurements of latency and / or a comparison of the latency to one or more reference values. For example, the automated assistant and / or the assistant server device 204 can interact with various different third-party server devices 202 corresponding to various different third-party entities. Each third-party entity can cause a certain amount of latency at their respective third-party server devices and / or their respective client devices. In some embodiments, the reference value can characterize the average amount of latency caused by one or more third-party entities. Thus, a metric value can indicate a higher reliability of a particular third-party server device that exhibits less latency than the average latency of one or more third-party entities. Additionally, another metric value can indicate a lower reliability of a particular third-party server device that exhibits more latency than the average latency of one or more third-party entities. In some embodiments, the reference value can be based on multiple different types of latency characteristics (e.g., each latency characteristic can characterize one or more different types of interactions involving the third-party server device and / or the third-party client device, the type of device, the time of day) or a single type of latency measurement.

[0036] When the third-party server device 202 corresponds to a metric indicating lower reliability relative to other third-party entities, the assistant server device 204 can provide a status request 244 to the third-party server device 202 after providing the device request 236. Alternatively or additionally, when the third-party server device 202 corresponds to a different metric indicating higher reliability relative to other third-party entities, the assistant server device 204 can bypass providing the status request 244 to the third-party server device 202 after providing the device request 236. In response to receiving the status request 244, the third-party server device 202 can generate updated status data and provide the status data to the assistant server device 204 and / or the assistant device 214. In response to receiving the updated status data, the assistant server device 204 and / or the assistant device 214 can cause the display panel 234 of the assistant device 214 to render an interface 230 characterizing the updated status data 242. For example, since the user 206 previously requested that the temperature setting of the client device 208 be set to 72 degrees, the interface 230 can characterize this most recent setting based on the updated status data received from the third-party server device 202. In this way, using such metrics allows the assistant device 214 and / or the assistant server device 204 to determine whether to actively retrieve updated status data or rely on the third-party server device 202 to actively provide updated status data.

[0037] By operating the assistant server device 204 and / or the assistant device 214 according to one or more metrics representative of the reliability of different third-party devices and / or entities, computing resources can be conserved. For example, when the assistant server device 204 acknowledges the reliability of a third-party server device, network bandwidth can be conserved and thus bypass sending frequent status update requests. Additionally, for those third-party entities that can be indicated as unreliable, the assistant server device 204 and / or the assistant device 214 will have more accurate data about other client devices, at least by proactively requesting such data. As a result, the assistant server device 204 and / or the assistant device 214 will be able to make more informed decisions regarding controlling certain client devices. For example, a user may rely on inaccurate data regarding the status of a client device and issue a request to the automated assistant to modify the status of the client device. As a result, the request may cause the assistant device 214 to implement instructions that cause the particular client device to operate inefficiently, perform redundant operations, and / or otherwise waste power and / or other computing resources. By employing the embodiments provided herein, such wasteful operations can be eliminated.

[0038] Figure 3 A system 300 is shown that is used to characterize the reliability of various different third-party servers and adapt a device ecosystem affected by such reliability, at least when reporting third-party device status to an automated assistant 304. The automated assistant 304 can operate as part of an assistant application provided at one or more computing devices such as client device 318 and / or server device 302. A user can interact with the automated assistant 304 via an assistant interface, which can be a microphone, camera, touch screen display, user interface, and / or any other device capable of providing an interface between the user and the application. For example, the user can initialize the automated assistant 304 by providing verbal, text, and / or graphical input to the assistant interface 320 to cause the automated assistant 304 to perform functions (e.g., provide data, control peripherals, access agents, generate input and / or output, etc.). The client device 318 can include a display device, which can be a display panel including a touch interface for receiving touch input and / or gestures for allowing a user to control an application of the client device 318 via the touch interface. In some embodiments, the client device 318 can lack a display device, thus providing an audible user interface output without providing a graphical user interface output. Additionally, the client device 318 can provide a user interface such as a microphone for receiving verbal natural language input from the user. In some embodiments, the client device 318 can include a touch interface and can lack a camera, but can optionally include one or more other sensors.

[0039] The client device 318 and / or other third-party client devices 318 can communicate with the server device 302 via a network 340 such as the Internet. Additionally, the client device 318 and other computing devices 434 can communicate with each other via a local area network (LAN) such as a Wi-Fi network. The client device 318 can offload computing tasks to the server device 302 to conserve computing resources at the client device 318. For example, the server device 302 can host the automation assistant 304, and the client device 318 can send inputs received at one or more assistant interfaces 320 to the server device 302. However, in some embodiments, the automation assistant 304 can be hosted at the client device 318 as a client automation assistant 322.

[0040] In various embodiments, all or less than all aspects of the automation assistant 304 can be implemented on the client device 318. In some of those embodiments, aspects of the automation assistant 304 are implemented via the client automation assistant 322 of the client device 318 and can interface with the server device 302 that can implement other aspects of the automation assistant 304. The server device 302 can optionally serve multiple users and their associated assistant applications via multiple threads. In embodiments where all or less than all aspects of the automation assistant 304 are implemented via the client automation assistant 322 at the client device 318, the client automation assistant 322 can be an application separate from the operating system of the client device 318 (e.g., installed "on top" of the operating system) - or can alternatively be implemented directly by the operating system of the client device 318 (e.g., considered an application of the operating system but integrated with the operating system).

[0041] In some embodiments, the automation assistant 304 and / or the client automation assistant 322 can include an input processing engine 306 that can employ multiple different modules for processing the input and / or output of the client device 318 and / or the server device 302. For example, the input processing engine 306 can include a speech processing engine 308 that can process audio data received at the assistant interface 320 to identify the text contained in the audio data. The audio data can be sent from, for example, the client device 318 to the server device 302 to conserve computing resources at the client device 318.

[0042] Processing for converting audio data into text can include a speech recognition algorithm that can employ a neural network and / or a statistical model to identify groups of audio data corresponding to words or phrases. The text converted from the audio data can be parsed by a data parsing engine 310 and made available as text data to an automated assistant, which can be used to generate and / or identify command phrases, intents, actions, slot values, and / or any other content specified by the user. In some embodiments, the output data provided by the data parsing engine 310 can be provided to a parameter module 312 to determine whether the user has provided input corresponding to a particular intent, action, and / or routine that can be performed by the automated assistant 304 and / or an application or agent accessible via the automated assistant 304. For example, assistant data 316 can be stored as assistant device data 316 at the server device 302 and / or the client device 318 and can include data defining one or more actions that can be performed by the automated assistant 304 and / or the client automated assistant 322, as well as the parameters necessary to perform the actions.

[0043] In some embodiments, the server device 302, the automated assistant 304, and / or the client device 318 are capable of tracking the reliability of the third-party server device 350 and / or the third-party client device 336. Specifically, the reliability of the third-party server device 350 and / or the third-party client device 336 can be tracked in terms of the latency in reporting status updates, the accuracy of specific status updates, and / or any other characteristics that can indicate the reliability of the server device and / or the client device. For example, a user can provide input to the assistant interface 320, which can cause the automated assistant 304 to provide an output via the output generation engine 314. The output can be based on data provided by the parameter engine 312 that can process the input from the user. The output can be provided to the third-party server device 315 to affect the operation and / or settings of the third-party client device 336. For example, the third-party client device 336 can be one of a plurality of third-party client devices connected in the user's home or otherwise associated with the user. The user can control the third-party client device 336 via the automated assistant 304 by permitting the automated assistant 304 to communicate with the third-party client device 336 and / or the third-party server device 350. The output from the output generation engine 314 can be provided to the third-party server device 350. For example, the output from the server device 302 can characterize the intention requested by the user, such as an alarm system control intention. The input request engine 354 can receive the output from the server device 302 and determine that the output has been provided by the server device 302 and that the output is based on the input from a specific user interacting with its corresponding automated assistant. Additionally, the client command engine 356 can generate one or more client commands based on the output of the server device 302. Based on one or more processes of the input request engine 354 and / or the client command engine 356, the third-party server device 350 can send one or more commands to the third-party client device 336 that the user intends to control with its input. When the third-party client device 336 receives the one or more commands, the command engine 342 of the third-party client device 336 can process the one or more commands. Command data can be generated by the command engine 342 and used by the setting engine 344 to determine one or more settings and / or one or more operations to be modified according to the one or more commands. Any changes to the settings, operations, and / or any other characteristics of the third-party client device 336 can be characterized as status data that is stored as local device data 346. When the status data has been generated at the third-party client device 336, the third-party client device 336 can confirm the completion of the execution of the one or more commands for the third-party server device 350.

[0044] In some embodiments, when the third-party server device 350 sends one or more commands to the third-party client device 336 based on an input from the user to the automated assistant, the third-party server device 350 is also capable of sending an acknowledgement to the server device 302 and / or the client device 318. The acknowledgement can indicate to the server device 302 and / or the client device 318 that the third-party server device 350 has acknowledged the input from the user and has acted on the input from the user. Thereafter, the third-party server device 315 can determine that the third-party client device 336 has successfully executed one or more commands, and in response to determining that the third-party client device 336 has successfully executed one or more commands, the third-party server device 350 can update any client device data 358 to characterize the updated operational state of the third-party client device 336. Additionally, depending on the third party entity responsible for the third-party server device 350, the third-party server device 250 can send status data to the server device 302. However, different third-party server devices can provide such status updates according to a variety of different protocols.

[0045] In some embodiments, the third-party server device 350 is capable of operating according to a protocol, where the client state engine 352 of the third-party server device 350 provides requests for acknowledgement that certain commands have been executed by the third-party client device 336. Based on such acknowledgements from the third-party client device 336, the client state engine 352 can generate various sets of status data, which can be stored as client device data 358 at the third-party server device 350. However, the third-party server device 350 can operate according to or fail to operate according to another protocol for providing status updates to other interested parties such as the server device 302, the automated assistant 304, and / or the client device 318.

[0046] The third-party interaction engine 326 of the server device 302 can collect various data regarding various different third-party server devices 350 to facilitate the elimination of status inaccuracies and / or delays presented at the client device 318. For example, time data corresponding to the interaction between the server device 302 and the third-party server device 350 can be collected by the third-party interaction engine 326. The third-party interaction engine 326 can identify the specific time when a request is transmitted from the server device 302, the specific time when a request is received from the user at the client device 318, the specific time when a request is received from the client device 318 at the server device 302, the specific time when the third-party server device 350 receives an instruction from the server device 302, the specific time when the third-party server device 350 sends command data to one or more third-party client devices 336, the specific time when one or more third-party client devices 336 confirm the receipt of the command data to the third-party server device 350, the specific time when the third-party server device 350 provides confirmation of the receipt of an instruction from the server device 302, the specific time when the third-party server device 350 requests a status update from the third-party client device 336, the specific time when the third-party server device 350 receives status data from the third-party client device 336, the specific time when the server device 302 receives client status data from the third-party server device 250, and / or any other time characteristics that can characterize a transaction occurring with respect to at least one device.

[0047] The server device 302 can include a reliability engine 324 that can use the time data generated by the third-party interaction engine 326 to generate one or more metrics 328 based on one or more time characteristics of any transaction discussed herein. The one or more metrics can characterize the reliability of one or more third-party server devices 350, at least in terms of actively providing status updates to the automated assistant 304, the server device 302, and / or the client device 318. Additionally, the third-party interaction engine 326 can use the one or more metrics 328 to make decisions regarding whether and / or how to interact with a specific third-party server device 350. For example, in response to determining that the user has provided a request to change the operation settings of the third-party client device 336, the server device 302 can determine whether to query the third-party server device 350 immediately after communicating the user request to the third-party server device 350 or to wait for the third-party server device 350 to provide updated status data corresponding to the third-party client device 336.

[0048] In some embodiments, the reliability engine 324 can determine and / or generate one or more reference values 360, and the third-party interaction engine 326 can compare the reference values 360 with the metrics 328. The reference values 360 can be based on one or more transactions corresponding to the third-party server device 350 and / or the third-party client device 336. Additionally, the reference values 360 can indicate the reliability of a particular third-party entity, at least in terms of how reliable the third-party entity is in a service that actively reports changes in the state of the server device and / or the client device. In some embodiments, the reference values and / or metrics can characterize the reliability of multiple different third-party entities, the reliability of multiple different third-party servers, the reliability of multiple different third-party client devices, the reliability of certain types of third-party entities, and / or any other third-party entity and / or device that can interact with the automated assistant.

[0049] As an example, and in some embodiments, the reference value can characterize the average amount of latency exhibited by one or more different third-party server devices when reporting client device status updates after fulfilling a user request of the automated assistant. In other words, when the average latency of multiple different third-party server devices 350 is 1 hour, at least one of the reference values 360 is, for example, 1 hour. Thus, when the automated assistant is interacting with a third-party server device 350 associated with a third-party entity, the third-party interaction engine 326 can access the metric corresponding to that third-party entity and compare the metric with the reference value. When the metric corresponds to a latency amount that is less than the reference value or the average value, the third-party interaction engine 326 can bypass actively querying the third-party server device 350 for client device status updates. However, when the metric corresponds to a latency amount that is greater than the reference value or the average value, the third-party interaction engine 326 can actively query the third-party server device 350 for client device status updates. It should be noted that the reference value can correspond to an amount of time that is less than or greater than any number of microseconds, milliseconds, seconds, hours, days, and / or any other unit of time. For example, the reference value can be, but is not limited to, 500 milliseconds, and thus the comparison of the metric with this particular reference can provide a basis for either querying the third-party server device 350 for a status update or bypassing providing a query for a status update to the third-party server device 350.

[0050] Figure 4FIG. illustrates a method 400 for determining whether to query a third-party server for a status update corresponding to a particular third-party client device based on one or more metrics. The method 400 may be performed by one or more computing devices, applications, and / or any other device or module capable of interacting with the third-party server device and / or the third-party client device. The method 400 can include an operation 402 of determining whether device status data for a particular device has been received. When device status data for a particular device has been received, the method 400 can proceed from operation 402 to operation 404. Operation 404 can include determining the amount of time delay or time elapsed since the most recent request to modify the operational state of the particular device. In other words, operation 404 can include determining the time period between the time when the device status data for the particular device was received and the time when the most recent request to modify the operational state of the particular device was provided to the third-party server device, the particular client device, and / or any other associated device.

[0051] The method 400 can also include an optional operation 406 of determining the accuracy of the stored operational state relative to the updated status data. The optional operation 406 can be periodically determined by a server device and / or a client device communicating with the third-party server device and / or the third-party client device. In some embodiments, the optional operation 406 can be determined randomly and / or in response to a request from a user for the device status in order to evaluate how reliable a particular third-party server device is in proactively providing status updates. For example, a user may have modified and / or initialized the operation of a third-party client device using the interface of the third-party client device or a third-party application corresponding to the third-party client device. Depending on how proactive the third party is in providing status updates, an automated assistant or assistant-related device may or may not receive any status updates regarding the user's actions. Thus, if the server device checks the status of the third-party client device and the status indicated by the third-party server device is different from the status stored at the server device, the server device can generate data to characterize this inaccuracy of the device status.

[0052] In addition, if the server device checks the status of a third-party client device and the status indicated by the third-party server device is the same as the status stored at the server device, the server device can also use the data to capture the accuracy. In this way, the server device and / or the client device can characterize the probability that the status data stored by the server device and / or the client device for a specific third-party client device is accurate at any given time. Therefore, if the server device determines that the probability that the status data of a specific third-party client device is accurate is high and the user requests the third-party client device to operate based on the same status data, the server device can bypass the request to the third-party server device to modify the operation of the third-party client device. As a result, this can save network bandwidth and computing resources because many insignificant or otherwise unnecessary transactions between the client and the server will be avoided.

[0053] However, if the server device determines that the probability that the status data of a specific third-party client device is accurate is low and the server device receives a request from the user to operate the third-party client device based on the same specific status data, even though the server device has the impression that the third-party client device has already operated as requested by the user, the server device can still transmit a request to the third-party server device anyway. In response, the third-party server device can process the request and indicate that the third-party client device has operated based on the same specific status data. In response, when the third-party client device has not operated based on the same specific status data stored by the server device, the third-party server device can process the request and cause the third-party client device to operate to facilitate fulfilling the request from the user.

[0054] Method 400 can further include operation 408 of generating and / or modifying one or more metrics corresponding to the reliability of third-party server devices. The one or more metrics can be based on the time delay and / or time period determined in operation 404, and / or the accuracy determined in optional operation 406. In other words, in some embodiments, the one or more metrics can be based on the amount of delay exhibited by one or more third-party server devices and / or the accuracy determined for a particular third-party server device. In some embodiments, the metric can be one or more different values selected from a range of different values. Alternatively or additionally, the metric can be a discrete value, such as 1 or 0. For example, when the time delay is greater than, equal to, or less than a particular reference value, the metric can be defined as 1. Further, when the time delay is less than, equal to, or greater than a particular reference value, the metric can be defined as 0. In one non-limiting example, when the time delay is 2 seconds and the reference value is 5 seconds (i.e., when the time delay is less than the reference value), the metric can be assigned 1. Alternatively, in another non-limiting example, when the time delay is 7 seconds and the reference value is 5 seconds (i.e., when the time delay is greater than the reference value), the metric can be assigned 0.

[0055] In some embodiments, the one or more metrics can be based on the data generated in operation 406 and / or operation 404, and / or any other data that can indicate the reliability of third-party server devices. The one or more metrics can be selected from a range of different values and can optionally be based on one or more reference values. In some embodiments, the one or more metrics can be discrete values or binary values, such as 1 or 0, indicating whether the third-party server device responsible for providing status data about one or more third-party client devices is reliable or unreliable in actively providing status data. In some embodiments, the one or more metrics can be used to determine whether to query a particular third-party server device for updated status data.

[0056] Method 400 can proceed from operation 408 to operation 402 to determine whether any further updated device status data of a particular device has been received. When, at operation 402, no updated status data of a particular device is received at a given time, method 400 can proceed from operation 402 to operation 410. Operation 410 can include determining whether one or more device metrics provide a basis for querying a third-party server device about the status data of a particular device. For example, one or more metrics can be compared with one or more reference values to determine whether there is a basis for providing a query for status data to the third-party server device. In some embodiments, one or more metrics can be processed to determine whether one or more metrics meet a particular threshold, and / or whether one or more metrics are greater than, equal to, and / or less than one or more particular reference values. When one or more metrics do not provide a basis for querying status data from the third-party server device, method 400 can proceed from operation 410 to operation 402. Alternatively, when one or more metrics do provide a basis for providing a query for status data to the third-party server device, method 400 can proceed from operation 410 to operation 412. By completing method 400, at least operations 402, 410, and 412, at least when no operation status update has been received within a particular time period, the assistant server and / or assistant device can determine whether and / or when to query the third-party server device for updated operation status data of the client device.

[0057] Operation 412 can include an operation of accessing the status data of a particular device via the third-party server device. To access the status data, the server device can send a request to the third-party server device, and in response, the third-party server device can provide the status data back to the requesting server device and / or the corresponding client device. Method 400 further includes operation 414 of determining whether the automated assistant has received a request to modify the operation status of a particular device. When the automated assistant has not received a request to modify the operation status of a particular device, method 400 can proceed from operation 414 to operation 402. However, when the assistant has received a request to modify the operation status of a particular device, method 400 can proceed from operation 414 to operation 416. Operation 416 can include causing the particular device to have a modified operation status according to the request received by the automated assistant. Additionally, method 400 can proceed from operation 416 to operation 402, where a determination is made as to whether updated status data of the particular device affected by the request from operation 414 and / or operation 416 has been received.

[0058] The reliability of a third-party server device can be determined to vary over time, e.g., by exhibiting a reliability pattern corresponding to time of day or day of the week. For example, using the techniques described herein, the latency associated with a particular third-party server can be determined to be greater during evenings / mornings and / or weekends than during other times of the day (or greater for other times of the day than other times such as evenings / mornings and / or weekends). Based on these determinations, a first-party server device can modify its behavior regarding actively seeking operation state updates of a particular third-party client device associated with the third-party server. On a day(s) of the week / time(s) of day in which the third-party server is determined to be satisfactorily reliable in actively providing state updates, the first-party server can reduce (or eliminate) its aggressiveness in requesting state updates from the third-party server. Correspondingly, on a day(s) of the week / time(s) of day in which the third-party server is determined to be unsatisfactorily reliable in actively providing state updates, the first-party server can increase its aggressiveness in requesting state updates from the third-party server. This can help the first-party server be efficient in its use of network resources and computing resources at both the first-party server and the third-party server, while also maintaining accurate information at the first-party server regarding the internal operation state of one or more third-party client devices associated with the third-party server.

[0059] Figure 5 is a block diagram of an example computer system 510. The computer system 510 generally includes at least one processor 514 that communicates with a plurality of peripheral devices via a bus subsystem 512. These peripheral devices can include a storage subsystem 524, e.g., including a memory 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computer system 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0060] The user interface input device 522 can include a keyboard, a pointing device such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into a display, an audio input device such as a speech recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computer system 510 or onto a communication network.

[0061] The user interface output device 520 can include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem can include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating visual images. The display subsystem can also provide a non-visual display such as via an audio output device. Generally, the use of the term "output device" is intended to include all possible types of devices and ways of outputting information from the computer system 510 to a user or another machine or computer system.

[0062] The storage subsystem 524 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 can include logic for performing selected aspects of the method 400 and / or for implementing one or more of the third party server device 102, the assistant server device 104, the third party application 120, the assistant device 114, the client device 110, the third party server device 202, the assistant server device 204, the assistant device 214, the server device 302, the client device 318, the third party server device 350, and / or the third party client device 336.

[0063] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 can include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 can provide permanent storage of program and data files and can include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical disk drive, or a removable media cartridge. Modules implementing the functionality of certain embodiments can be stored in the storage subsystem 524 by the file storage subsystem 526 or in other machines accessible by the processor 514.

[0064] The bus subsystem 512 provides a mechanism for enabling the various components and subsystems of the computer system 510 to communicate with each other as expected. Although the bus subsystem 512 is schematically shown as a single bus, alternative embodiments of the bus subsystem can use multiple buses.

[0065] The computer system 510 can be of various types including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5The description of the computer system 510 depicted is only intended as a specific example for the purpose of illustrating some embodiments. Many other configurations of the computer system 510 may have more or fewer components than the Figure 5 computer system depicted.

[0066] In cases where the systems described herein collect personal information about a user (or as often referred to herein as a "participant") or can use personal information, the user may be provided with an opportunity to control whether programs or features collect user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or user's current geographic location) or to control whether and / or how content that may be more relevant to the user is received from a content server. Additionally, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the identity of a user may be processed so that no personally identifiable information for the user can be determined, or the user's geographic location may be generalized to the place where the geographic location information was obtained (such as to a city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, the user can control how information about the user is collected and / or used.

[0067] In some embodiments, a method implemented by one or more processors is set forth as including operations such as receiving, from a third-party server device, first status data indicative of a state of a third-party client device controllable via an assistant device, where the assistant device includes an automated assistant interface via which a user interacts with the automated assistant to control the third-party client device. The method can further include determining, based on receiving the first status data, a measure of a time delay amount between receiving the first status data and a previous time associated with a request to modify an operational state of the third-party client device. The method can further include, after determining the measure based on the time delay amount: determining whether to provide a status request to the third-party server device to retrieve second status data indicative of a current state of the third-party client device. The method can further include, when the measure indicates at least some basis for not querying the third-party server device for a current operational state of the third-party client device: bypassing providing the status request to the third-party server device based on the measure. The method can further include, when the measure indicates at least some basis for querying the third-party server device for a current operational state of the third-party client device: providing the status request to the third-party server device based on the measure, and in response to the third-party server device receiving the status request, receiving second status data characterizing the current operational state of the third-party client device.

[0068] In some of the ways, the method can further include, after determining a metric based on a time delay amount: determining that the user has provided a gesture to the interface of the assistant device or another client device to facilitate controlling a third-party client device, wherein determining whether to provide the status request is based on determining that the user has provided a gesture to the interface to facilitate controlling a third-party client device. In some embodiments, the gesture is a tactile gesture and / or an oral utterance, and the interface includes a touch display panel and / or a microphone. In some embodiments, the method can further include generating output data representing the current operating state of the third-party client device in response to receiving second status data; and causing the output data to be rendered via the assistant device or another client device. In some embodiments, determining a metric based on a time delay amount includes: modifying a previously generated metric based on the time delay amount between receiving the first status data and a previous time associated with the status of the third-party client device.

[0069] In some embodiments, modifying a previously generated metric based on a time delay amount includes adapting the previously generated metric to limit the basis for querying the third-party server device for the current operating state of the third-party client device when the time delay amount is less than a reference time delay amount. In some embodiments, modifying a previously generated metric based on a time delay amount includes adapting the previously generated metric to expand the basis for querying the third-party server device for the current operating state of the third-party client device when the time delay amount is greater than a reference time delay amount. In some embodiments, the reference value is based on a time value corresponding to other requests from one or more other users to other third-party server devices to facilitate modifying the operating state of other third-party client devices. In some embodiments, determining the metric includes: accessing metric data characterizing the metric and other metrics, and the assistant device communicates with a plurality of other third-party client devices, and one or more of the plurality of other third-party client devices are associated with at least one of the other metrics.

[0070] In some embodiments, the request is based on an oral utterance provided by a user to an automated assistant interface, and the natural language content of the oral utterance identifies an automated assistant routine corresponding to a plurality of different actions, and the plurality of different actions includes an action for transitioning the state of a third-party client device to a current operating state. In some embodiments, determining the metric includes accessing another metric, the other metric being based on whether the third-party server device has previously provided a status update to the automated assistant in response to a request by the user via a third-party hardware interface and / or a third-party application communicating with the third-party client device for a change in the state of the third-party client device. In some embodiments, the other metric is further based on another amount of time between the user's interaction with the third-party hardware interface and / or the third-party application and the third-party server device providing a status update to the automated assistant and / or the assistant device. In some embodiments, determining whether to query the third-party server device for an indication of the current operating state of the third-party client device is further based on the other metric, and the third-party server device is at least partially controlled by a third-party entity that is different from the entity that at least partially controls the automated assistant.

[0071] In some embodiments, the metric and / or the other metric is based on the interaction between one or more other users and one or more other third-party client devices communicating with the third-party server device. In some embodiments, the one or more other third-party client devices: are different from the third-party client device and are connected to a network separate from the network to which the third-party client device is connected. In some embodiments, the request is received by the third-party server device in response to the user's interaction with a third-party application that controls the third-party client device and / or the third-party server device. In some embodiments, the method can further include generating a graphical user interface element representing the current operating state of the third-party client device in response to receiving second status data; and causing the graphical user interface of the assistant device or another client device to render the graphical user interface element. In some embodiments, the metric is based on a specific time delay between a first request initiated at the third-party server device to change the operating state of the client device and a second request for the current state from the automated assistant. In some embodiments, the request identifies at least one action of the automated assistant, and the at least one action includes an action for transitioning the state of the third-party client device to the current operating state.

[0072] In some embodiments, a method implemented by one or more processors is described as including operations such as determining, based on processing audio data corresponding to spoken utterances, that a user has requested an automated assistant resulting in a modification to an operating state of a client device. In some embodiments, the method can further include providing, in response to determining that a user has requested an automated assistant resulting in a modification to an operating state of a client device, a request to a server device and / or the client device to effect the modification to the operating state of the client device. In some embodiments, the method can further include receiving, based on providing the request to the server device and / or the client device, first status data from the server device and / or the client device, where the first status data characterizes the modified operating state of the client device. In some embodiments, the method can further include determining, based on receiving the first status data, a quantity of time delay characterizing a period of time between determining that a user has requested an automated assistant resulting in a modification to an operating state of a client device and receiving the first status data from the server device and / or the client device. In some embodiments, the method can further include, after determining the quantity of time delay: determining that the user has subsequently requested an automated assistant resulting in a particular modification to a current operating state of the client device. In some embodiments, the method can further include providing, in response to determining that the user has subsequently requested an automated assistant resulting in a particular modification to a current operating state of the client device, another request to a server device and / or the client device to effect the particular modification to the current operating state of the client device. In some embodiments, the method can further include determining, based on the quantity of time delay and in response to determining that a user has requested an automated assistant resulting in a particular modification to a current operating state of the client device, whether the quantity of time delay indicates at least some basis for querying a server device and / or the client device for second status data. In some embodiments, the method can further include, when the quantity of time delay indicates no at least some basis for querying a server device and / or the client device for second status data: bypassing providing a status request to the server device and / or the client device based on the quantity of time delay. In some embodiments, the method can further include, when the quantity of time delay indicates at least some basis for querying a server device and / or the client device for second status data: providing a status request to the server device and / or the client device based on the quantity of time delay, and receiving second status data characterizing an update to the operating state of the client device in response to the server device and / or the client device receiving the status request.

[0073] In some embodiments, the method can further include, after determining the amount of time delay: causing the user interfaces of the separate client devices to provide content representative of the current operating state of the client devices. In some embodiments, the user interfaces of the separate client devices are display panels and provide graphical content representative of the current operating state of the client devices. In some embodiments, the method can further include, when the amount of time delay indicates at least some basis for querying the server device and / or the client device for second state data: causing the display panel to render additional graphical content representative of an update to the operating state of the client device. In some embodiments, the amount of time delay represents a specific period of time between: providing a request to the server device and / or the client device to effect a modification to the operating state of the client device, and receiving first state data from the server device and / or the client device.

[0074] In some embodiments, a method implemented by one or more processors is set forth as including operations such as causing a client device to render content representative of a first state of another client device, where the client device and the other client device are connected to a common local area network and the other client device is controlled using an automated assistant that is at least accessible via the client device. In some embodiments, the method can further include, after causing the client device to render the content, determining that the client device has received a command to cause the other client device to operate according to a second state. In some embodiments, the method can further include, in response to determining that the client device has received a command to cause the other client device to operate according to a second state, providing a request to a third-party server device to cause the other client device to operate according to the second state. In some embodiments, the method can further include, after sending the request to the third-party server device, receiving status data from the third-party server device, where the status data represents the operating state of the other client device. In some embodiments, the method can further include, when the status data indicates that the client device is operating according to the second state: modifying or bypassing a modification of one or more metrics to characterize the server device as being reliable in providing accurate status data, at least in response to the server device causing the other client device to exhibit any modified state relative to any previous state of the other client device. In some embodiments, the method can further include, when the status data indicates that the client device is not operating according to the second state: modifying or generating one or more metrics to indicate that the server device is unreliable in providing accurate status data, at least in response to the server device causing the other client device to exhibit any modified state relative to any previous state of the other client device.

[0075] In some embodiments, modifying or generating one or more metrics to indicate that the server device is unreliable in providing accurate status data includes: in response to determining that a subsequent command is received for causing another client device to operate according to a specific status, modifying or generating one or more metrics to indicate the basis for actively querying the server device. In some embodiments, the method can further include modifying or bypassing the modification of one or more metrics to cause the server device to be characterized as reliable in providing accurate status data includes: in response to determining that a subsequent command is received for causing another client device to operate according to a specific status, modifying or bypassing the modification of one or more metrics to indicate the basis for bypassing the active query of the server device. In some embodiments, causing the client device to render content characterizing a first status of another client device includes causing the graphical user interface of the client device to render one or more graphical elements at the graphical user interface.

[0076] In some embodiments, the method can further include determining the device type characterizing another client device, wherein the modification or generation of one or more metrics is based on the device type characterizing another client device. In some embodiments, the method can further include determining the specific time at which the request is provided to the third-party server device, wherein the modification or generation of one or more metrics is based on the specific time at which the request is provided to the third-party server device. In some embodiments, determining that the client device has received a command includes determining the type of command for causing another client device to operate according to a second status, wherein the modification or generation of one or more metrics is based on the type of command. In some embodiments, determining the type of command includes determining whether the command is initialized via an automated assistant, a specific interface of another client device, or a separate interface of a peripheral device associated with another client device.

[0077] In some embodiments, a method implemented by one or more processors is described as including operations such as causing a client device to store state data representative of another client device, where the client device and the other client device are connected to a common local area network, and the other client device is controlled using an automated assistant that is at least accessible via the client device. The method can further include, after storing the state data at the client device, providing a server request to a server device to determine a current operational state of the other client device. In some embodiments, the method can further include, after storing the state data at the client device, providing a client request to the other client device to determine a current operational state of the other client device. In some embodiments, the method can further include receiving, from the server device, server state data based on sending the server request to the server device, where the server state data represents a particular state of the other client as indicated by the server device. In some embodiments, the method can further include receiving, from the other client device, client state data based on sending the client request to the other client device, where the client state data represents another particular state of the other client as indicated by the other client device. In some embodiments, the method can further include, when the server state data and the client state data indicate a common operational state of the other client device: causing the client device to store updated state data representative of the common operational state of the other client device.

[0078] In some embodiments, the method can further include, when the server state data and the client state data indicate a common operating state of another client device: modifying or bypassing modification of one or more metrics to characterize the server device as being reliable in providing accurate state data. In some embodiments, the method can further include, when the server state data and the client state data do not indicate a common operating state of another client device: causing the client device to store other updated state data characterizing another specific state of another client as indicated by another client device. In some embodiments, the method can further include, when the server state data and the client state data do not indicate a common operating state of another client device: modifying or generating one or more metrics to indicate that the server device is unreliable in providing accurate state data. In some embodiments, the method can further include determining the device type characterizing another client device, wherein the modification or generation of one or more metrics is based on the device type characterizing another client device. In some embodiments, the method can further include determining a specific time at which a server request is provided to the server device, wherein the modification or generation of one or more metrics is based on the specific time at which the server request is provided to the server device. In some embodiments, the method can further include determining the amount of time between a first timestamp corresponding to the state of another client device and a second timestamp corresponding to the server state data received from the server device.

[0079] Although several of the foregoing embodiments have been described and illustrated herein, various other devices and / or structures can be utilized for performing the functions and / or obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all of the parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend upon the one or more specific applications of the teachings herein. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. Accordingly, it is to be understood that the foregoing embodiments are presented by way of example only, and that the embodiments can be practiced otherwise than as specifically described and claimed within the scope of the appended claims and their equivalents. Embodiments of the present disclosure relate to each and every separate feature, system, article, material, kit, and / or method described herein. Additionally, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, any combination of two or more of such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors, the method comprising: causing a client device to render content representative of a first state of another client device, wherein the client device and the other client device are connected to a common local area network, and the other client device is controlled using an automated assistant that is at least accessible via the client device; after causing the client device to render the content, determining that the client device has received a command to cause the other client device to operate according to a second state; in response to determining that the client device has received the command to cause the other client device to operate according to the second state, providing a request to a third-party server device to cause the other client device to operate according to the second state; after sending the request to the third-party server device, receiving status data from the third-party server device, wherein the status data represents an operating state of the other client device; when the status data indicates that the client device is operating according to the second state: modifying or bypassing a modification of one or more metrics to cause the third-party server device to be characterized as reliable in providing accurate status data, at least in response to the third-party server device causing the other client device to exhibit a modified state relative to any previous state of the other client device, wherein the one or more metrics indicate the reliability of the third-party server device in providing accurate status data; when the status data indicates that the client device is not operating according to the second state: modifying or generating the one or more metrics to indicate that the third-party server device is unreliable in providing the accurate status data, at least in response to the third-party server device causing the other client device to exhibit a modified state relative to any previous state of the other client device; 2. The method according to claim 1, wherein, Modifying or generating the one or more metrics to indicate that the third-party server device is unreliable in providing the accurate status data includes: modifying or generating the one or more metrics to indicate a basis for actively querying the third-party server device in response to determining that a subsequent command is received to cause the other client device to operate according to a specific state; 3. The method according to claim 1, wherein Modifying or bypassing a modification of one or more metrics to cause the third-party server device to be characterized as reliable in providing accurate status data includes: modifying or bypassing a modification of the one or more metrics to indicate a basis for bypassing an active query of the third-party server device in response to determining that a subsequent command is received to cause the other client device to operate according to a specific state; 4. The method according to claim 1, wherein, Causing the client device to render content representative of the first state of the other client device includes causing a graphical user interface of the client device to render one or more graphical elements at the graphical user interface; 5. The method according to claim 1, further comprising: Determine the device type characterizing the other client device, wherein modifying or generating the one or more metrics is based on the device type characterizing the other client device.

6. The method according to claim 1, further comprising: Determine the specific time at which the request is provided to the third-party server device, wherein modifying or generating the one or more metrics is based on the specific time at which the request is provided to the third-party server device.

7. The method according to claim 1, wherein Determining that the client device has received the command includes determining the type of command for causing the other client device to operate according to the second state, wherein modifying or generating the one or more metrics is based on the type of command.

8. The method according to claim 1, wherein, Determining the type of command includes determining whether the command is initialized via the automated assistant, a specific interface of the other client device, or a separate interface of a peripheral device associated with the other client device.

9. A method implemented by one or more processors, the method comprising: Cause a client device to store status data characterizing the status of another client device, wherein the client device and the other client device are connected to a common local area network, and the other client device is controlled using an automated assistant that is at least accessible via the client device; After storing the status data at the client device, provide a server request to a third-party server device to determine the current operating status of the other client device; After storing the status data at the client device, provide a client request to the other client device to determine the current operating status of the other client device; Based on sending the server request to the third-party server device, receive server status data from the third-party server device, wherein the server status data characterizes a specific status of the other client indicated by the third-party server device; Based on sending the client request to the other client device, receive client status data from the other client device, wherein the client status data characterizes another specific status of the other client device indicated by the other client device; and When the server status data and the client status data indicate a common operating status of the other client device: Cause the client device to store updated status data characterizing the common operating status of the other client device.

10. The method according to claim 9, further comprising: When the server status data and the client status data indicate the common operating status of the other client device: Modify or bypass modifying one or more metrics to cause the third-party server device to be characterized as reliable in providing accurate status data.

11. The method according to claim 9, further comprising: When the server status data and the client status data cannot indicate the common operating status of the other client device: Cause the client device to store other updated status data representing the other specific state of the other client device indicated by the other client device.

12. The method according to claim 9, further comprising: When the server status data and the client status data cannot indicate the common operating state of the other client device: Modify or generate one or more metrics to indicate that the third-party server device is unreliable in providing the accurate status data.

13. The method according to claim 12, further comprising: Determine the device type representing the other client device, wherein modifying or generating the one or more metrics is based on the device type representing the other client device.

14. The method according to claim 12, further comprising: Determine the specific time at which the server request is provided to the third-party server device, wherein modifying or generating the one or more metrics is based on the specific time at which the server request is provided to the third-party server device.

15. The method according to claim 9, further comprising: Determine the amount of time between a first timestamp corresponding to the state of the other client device and a second timestamp corresponding to receiving the server status data from the third-party server device.

16. A system, comprising: A memory storing instructions; One or more processors that execute the instructions for: Cause a client device to render content representing a first state of another client device, wherein the client device and the other client device are connected to a common local area network, and the other client device is controlled using an automated assistant that is at least accessible via the client device; After causing the client device to render the content, determine that the client device has received a command to cause the other client device to operate according to a second state; In response to determining that the client device has received a command to cause the other client device to operate according to the second state, provide a request to a third-party server device to cause the other client device to operate according to the second state; After sending the request to the third-party server device, receive status data from the third-party server device, wherein the status data represents the operating state of the other client device; When the status data indicates that the client device is operating according to the second state: At least in response to the third-party server device causing the other client device to exhibit any modified state relative to any previous state of the other client device, modify or bypass modifying one or more metrics to cause the third-party server device to be characterized as reliable in providing accurate status data, wherein the one or more metrics indicate the reliability of the third-party server device in providing accurate status data; When the status data indicates that the client device is not operating according to the second state: Modify or generate the one or more metrics to indicate that the third-party server device is unreliable in providing the accurate status data, at least in response to the third-party server device causing the other client device to present a status that is a modification of any previous state of the other client device.

17. A system comprising: A memory storing instructions; One or more processors operable to execute the instructions for: Causing a client device to render content representative of a first state of a smart device that is an addition to the client device, wherein the client device and the smart device are connected to a common local area network, and the smart device is controlled using an automated assistant that is accessible at least via the client device; After causing the client device to render the content, determining that the client device has received a command to cause the smart device to operate according to a second state; In response to determining that the client device has received a command to cause the smart device to operate according to the second state, providing a request to a third-party server device to cause the smart device to operate according to the second state; After sending the request to the third-party server device, receiving status data from the third-party server device, wherein the status data represents an operating state of the smart device; Based on the status data, determining whether the status data indicates that the smart device is operating according to the second state; and Modifying one or more metrics based on whether the status data indicates that the smart device is operating according to the second state, wherein the one or more metrics indicate the reliability of the third-party server device in providing accurate status data of the smart device.

18. The system according to claim 17, wherein When modifying the one or more metrics based on whether the status data indicates that the smart device is operating according to the second state, one or more of the processors are for: Modifying the one or more metrics to indicate a basis for actively querying the third-party server device in response to determining that a subsequent command to cause the smart device to operate according to a specific state is received.

19. The system according to claim 17, wherein, When modifying the one or more metrics based on whether the status data indicates that the smart device is operating according to the second state, one or more of the processors are for: Modifying the one or more metrics to indicate a basis for bypassing active querying of the third-party server device in response to determining that a subsequent command to cause the smart device to operate according to a specific state is received.

20. The system according to claim 17, wherein When causing the client device to render content representative of the first state of the smart device, one or more of the processors are for causing a graphical user interface of the client device to render one or more graphical elements at the graphical user interface, the one or more graphical elements representing the first state.

21. The system according to claim 20, wherein, One or more of the processors are further operable to execute the instructions to: Determine the type characterizing the smart device, wherein modifying or generating the one or more metrics is based on the type characterizing the smart device.

22. The system according to claim 20, wherein One or more of the processors are further operable to execute the instructions to: Determine a specific time at which the request is provided to the third-party server device, wherein modifying the one or more metrics is based on the specific time at which the request is provided to the third-party server device.

23. The system according to claim 20, wherein, The smart device is a smart light bulb or a smart security alarm.

24. The system according to claim 17, wherein One or more of the processors are further operable to execute the instructions to: Determine the type of the smart device, wherein modifying the one or more metrics is based on the type of the smart device.

25. The system according to claim 17, wherein One or more of the processors are further operable to execute the instructions to: Determine a specific time at which the request is provided to the third-party server device, wherein modifying the one or more metrics is based on the specific time at which the request is provided to the third-party server device.

26. The system according to claim 17, wherein, When it is determined that the client device has received the command, one or more of the processors are used to determine the type of the command for causing the smart device to operate according to the second state, wherein modifying the one or more metrics is based on the type of the command.

27. The system according to claim 26, wherein, When determining the type of the command, one or more of the processors are used to determine whether the command is initiated via the automated assistant, a specific interface of the smart device, or a separate interface of a peripheral device associated with the smart device.

28. A system, comprising: A memory storing instructions; One or more processors, the one or more processors being operable to execute the instructions for: Provide a server request for determining a current operating state of a smart device to a third-party server device, wherein the third-party server device at least selectively controls the smart device in response to an automated assistant request related to the smart device; Provide a client request for determining the current operating state of the smart device to the smart device; Based on sending the server request to the third-party server device, receive server status data from the third-party server device, wherein the server status data characterizes the server-indicated state of the smart device; Based on sending the client request to the smart device, receive smart device status data from the smart device, wherein the smart device status data characterizes the smart device-indicated state of the smart device; and Determine whether the server status data and the smart device status data indicate the same state of the smart device; and Modify one or more metrics based on whether the server status data and the smart device status data indicate the same state of the smart device.

29. The system according to claim 28, wherein, When modifying one or more metrics based on whether the server status data and the smart device status data indicate the same state of the smart device, one or more of the processors are for: In response to the server status data and the smart device status data indicating the same state: Modify the one or more metrics such that the third-party server device is characterized as being reliable in providing an accurate status.

30. The system according to claim 28, wherein, When modifying the one or more metrics based on whether the server status data and the smart device status data indicate the same status of the smart device, one or more of the processors are configured to: In response to the server status data and the smart device status data not indicating the same status: Modify the one or more metrics such that the third-party server device is characterized as being unreliable in providing the accurate status.

31. The system according to claim 28, wherein, The smart device is a smart light bulb.

32. The system according to claim 28, wherein, One or more of the processors are further operable to execute the instructions to: Determine the type of the smart device, wherein modifying the one or more metrics is based on the type of the smart device.

33. The system according to claim 28, wherein, One or more of the processors are further operable to execute the instructions to: Determine the specific time at which the server request is provided to the third-party server device, wherein modifying the one or more metrics is based on the specific time at which the server request is provided to the third-party server device.

Images