Self-learning home system and autonomous home operation framework

Abstract

The invention relates to a self-learning home system and an autonomous home operation framework. A computer-implemented method for automated operation of network devices within a home includes detecting actuator actions of a plurality of actuators within the home, each of the plurality of actuators for changing a state of at least one of the network devices. The detected actuator action is associated with one or more sensor values from a plurality of sensors within the home to generate configuration data. The configuration data includes a trigger graph having one or more trigger conditions and an action graph corresponding to the trigger graph. The action graph indicates one or more actuator actions associated with at least one of the plurality of actuators. Upon detecting the one or more trigger conditions, the at least one of the plurality of actuators is triggered to perform the one or more actions indicated by the action graph.

Description

[0001] Related Applications Cross-Application

[0002] This application is a divisional application, the application number of the original application is 201980076620.4, the application date is June 14, 2019, and the entire content of the original application is incorporated by reference in this application. This application claims priority and benefits of U.S. Provisional Application No. 62 / 770,070, entitled "Self-Learning Home System and Autonomous Home Operation Framework", filed on November 20, 2018, which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a smart home solution for controlling devices in a home communication network. Some aspects relate to a self-learning home system for controlling devices within a home and providing a framework for autonomous home operations. Background Art

[0004] Internet-of-Things (IoT) devices and solutions are becoming more and more widely used in home and office environments. Such connected devices can be used in a variety of ways, from increasing comfort to improving the feature set of a home to increasing the efficiency and security of other devices.

[0005] Existing home automation solutions integrate the control of home appliances into mobile device applications and allow users to create simplified rules that are triggered by user actions. However, the challenge with existing home automation solutions is that they require users to explicitly define rules for how appliances / devices should work together, which becomes more challenging even for technically inclined people as the number and complexity of triggerable appliances increases. Summary of the invention

[0006] Various examples are now described to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0007] According to a first aspect of the present disclosure, a computer-implemented method for automated operation of network devices within a home communication network is provided. The method includes detecting actuator actions of multiple actuators within the home communication network, each of the multiple actuators being used to change the state of at least one of the network devices. The detected actuator actions are associated with one or more sensor values ​​from multiple sensors within the home communication network to generate configuration data. The configuration data may include a trigger diagram having one or more trigger conditions. The configuration data also includes an action diagram corresponding to the trigger diagram. The action diagram indicates one or more actuator actions associated with at least one of the multiple actuators. When the one or more trigger conditions are detected, the at least one of the multiple actuators may be triggered to perform the one or more actions indicated by the action diagram.

[0008] In a first implementation form of the method according to the first aspect, the trigger diagram includes multiple trigger conditions, and the trigger conditions include one or more of the following: the one or more sensor values; the actuator action in the detected actuator action; and external information data received from an information source outside the home communication network.

[0009] In a second implementation form of the method according to the first aspect or any preceding implementation form of the first aspect, the multiple trigger conditions within the trigger diagram are connected via one or more logical connectors.

[0010] In a third implementation form of the method according to the first aspect or any preceding implementation form of the first aspect, when the multiple trigger conditions are detected, and further based on the one or more logical connectors, at least one of the multiple actuators is triggered to perform the one or more actions indicated by the action diagram.

[0011] In a fourth implementation form of the method according to the first aspect or any preceding implementation form of the first aspect, the action graph comprises a plurality of nodes coupled by edges. Each of the plurality of nodes corresponds to an actuator action of the one or more actuator actions associated with the at least one of the plurality of actuators.

[0012] In a fifth implementation form of the method according to the first aspect or any preceding implementation form of the first aspect, each of the edges coupling at least two of the plurality of nodes is associated with a time delay between the one or more actuator actions corresponding to the at least two nodes.

[0013] In a sixth implementation form of the method according to the first aspect or any of the preceding implementation forms of the first aspect, a training data set including a plurality of predetermined trigger conditions and a plurality of predetermined actuator actions corresponding to the plurality of predetermined trigger conditions is obtained. A machine learning (ML) program is trained using at least the training data set to generate a trained ML program. The trained ML program is applied to the detected actuator actions and the one or more sensor values ​​from the collected sensor data to generate the configuration data.

[0014] In a seventh implementation form of the method according to the first aspect or any preceding implementation form of the first aspect, a user-defined target for the automated operation of the network device within the home communication network is retrieved. A predefined sensor-actuator relationship that groups a subset of the plurality of actuators with a subset of the plurality of sensors based on common sensor and actuator locations is retrieved. The configuration data is further generated based on the user-defined target and the predefined sensor-actuator relationship.

[0015] In an eighth implementation form of the method according to the first aspect or any preceding implementation form of the first aspect, the configuration data comprises an automation rules table having a plurality of trigger diagrams and a corresponding plurality of action diagrams.

[0016] In a ninth implementation form of the method according to the first aspect or any preceding implementation form of the first aspect, a second automation rule table for a second home communication network is retrieved. Based on the second automation rule table, the trigger diagram and the action diagram in the configuration data are modified, or the trigger diagram or the action diagram in the configuration data is modified.

[0017] According to a second aspect of the present disclosure, there is provided a system comprising a plurality of actuators within a home communication network, each of the plurality of actuators being used to change the state of at least one of the network devices within the home communication network. The system further comprises a plurality of sensors within the home communication network, the plurality of sensors being used to collect sensor data. The system further comprises a memory storing instructions. The system further comprises one or more processors communicating with the memory, the plurality of actuators and the plurality of sensors. The one or more processors execute the instructions to detect actuator actions of the plurality of actuators and associate the detected actuator actions with one or more sensor values ​​from the collected sensor data to generate configuration data. The configuration data comprises a trigger diagram having one or more trigger conditions and an action diagram corresponding to the trigger diagram. The action diagram indicates one or more actuator actions associated with at least one of the plurality of actuators. When the one or more trigger conditions are detected, the at least one of the plurality of actuators can be triggered to perform the one or more actions indicated by the action diagram.

[0018] In a first implementation form of the device according to the second aspect, the trigger diagram comprises a plurality of trigger conditions connected via one or more logical connectors. The trigger conditions comprise one or more of: the one or more sensor values; an actuator action in the detected actuator actions; and external information data received from an information source external to the home communication network.

[0019] In a second implementation form of the device according to the second aspect or any preceding implementation form of the second aspect, the one or more processors execute the instructions to trigger at least one of the multiple actuators to perform the one or more actions indicated by the action diagram when the multiple trigger conditions are detected and further based on the one or more logical connectors.

[0020] In a third implementation form of the device according to the second aspect or any preceding implementation form of the second aspect, the action graph comprises a plurality of nodes coupled by edges. Each of the plurality of nodes corresponds to an actuator action of the one or more actuator actions associated with the at least one of the plurality of actuators. Each of the edges coupling at least two of the plurality of nodes is associated with a time delay between the one or more actuator actions corresponding to the at least two nodes.

[0021] In a fourth implementation form of the device according to the second aspect or any of the preceding implementation forms of the second aspect, the one or more processors execute the instructions to obtain a training data set including a plurality of predetermined trigger conditions and a plurality of predetermined actuator actions corresponding to the plurality of predetermined trigger conditions. A machine learning (ML) program is trained using at least the training data set to generate a trained ML program. The trained ML program is applied to the detected actuator actions and the one or more sensor values ​​from the collected sensor data to generate the configuration data.

[0022] In a fifth implementation form of the device according to the second aspect or any preceding implementation form of the second aspect, the one or more processors execute the instructions to retrieve a user-defined target for the automated operation of the network device within the home communication network. Retrieve a predefined sensor-actuator relationship that groups a subset of the plurality of actuators with a subset of the plurality of sensors based on common sensor and actuator locations. Generate the configuration data further based on the user-defined target and the predefined sensor-actuator relationship.

[0023] In a sixth implementation form of the device according to the second aspect or any preceding implementation form of the second aspect, the configuration data comprises an automation rules table having a plurality of trigger diagrams and a corresponding plurality of action diagrams.

[0024] In a seventh implementation form of the device according to the second aspect or any preceding implementation form of the second aspect, the one or more processors execute the instructions to retrieve a second automation rule table for a second home communication network, and based on the second automation rule table, modify the trigger diagram and the action diagram in the configuration data, or modify the trigger diagram or the action diagram in the configuration data.

[0025] According to a third aspect of the present disclosure, a non-transitory computer-readable medium is provided, which stores instructions for automated operation of network devices within a home communication network, and when executed by one or more processors, causes the one or more processors to perform the following operations. The operation includes detecting actuator actions of multiple actuators within the home communication network, each of the multiple actuators being used to change the state of at least one of the network devices. The operation also includes associating the detected actuator actions with one or more sensor values ​​from multiple sensors within the home communication network to generate configuration data. The configuration data includes a trigger diagram having one or more trigger conditions and an action diagram corresponding to the trigger diagram. The action diagram indicates one or more actuator actions associated with at least one of the multiple actuators. When the one or more trigger conditions are detected, the at least one of the multiple actuators is triggered to perform the one or more actions indicated by the action diagram.

[0026] In a first implementation form of the non-transitory computer-readable medium according to the third aspect, when executed, the instructions further cause the one or more processors to perform operations including obtaining a training data set including a plurality of predetermined trigger conditions and a plurality of predetermined actuator actions corresponding to the plurality of predetermined trigger conditions. The operations also include training a machine learning (ML) program using at least the training data set to generate a trained ML program, and applying the trained ML program to the detected actuator actions and the one or more sensor values ​​from the collected sensor data to generate the configuration data.

[0027] Any of the above examples may be combined with any one or more of the other examples described above to create new embodiments within the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views.The drawings generally illustrate, by way of example, and not by way of limitation, various embodiments discussed in this document.

[0029] Figure 1 is a high-level system overview diagram of a home communication network using automated operation of network devices according to some exemplary embodiments.

[0030] Figure 2 According to some exemplary embodiments Figure 1 A high-level architectural design diagram of a home communication network.

[0031] Figure 3 is a diagram showing a method that can be used with Figure 1 A block diagram of implementation details of a computing device used in conjunction with a home communications network.

[0032] Figure 4 is a diagram showing a method that can be used with Figure 1 Block diagram of the data collection process used in conjunction with a home communication network.

[0033] Figure 5 is a block diagram illustrating a process for creating an automation rules table that may be used for automated operations of a network device, according to some exemplary embodiments.

[0034] Fig. 6A is a block diagram illustrating the training and use of a machine learning (ML) program according to some exemplary embodiments.

[0035] Figure 6B is a diagram illustrating generation of a trained ML program using a neural network according to some exemplary embodiments.

[0036] Figure 7 is a flow chart illustrating correction / retraining of an automation rules table that may be used for automated operation of a network device according to some exemplary embodiments.

[0037] Figure 8 is a flow chart illustrating execution of an action graph from an automation rules table based on matching of trigger conditions according to some exemplary embodiments.

[0038] Fig.9A , Fig. 9B as well as Fig. 9C Various methods for federated computing related to automated operation of one or more network devices in a home are shown according to some exemplary embodiments.

[0039] Fig.10 A communication architecture with multiple homes and cloud-based ML training and automated rule table generation is shown according to some exemplary embodiments.

[0040] Fig.11A and Fig. 11B According to some exemplary embodiments, connections to Figure 1 Block diagram of a networked home communications network with third-party devices and third-party applications.

[0041] Fig. 12A A user interface representing an automation rules table is shown according to some example embodiments.

[0042] Fig. 12B A user interface for setting user goals related to automated operation of a network device is shown according to some example embodiments.

[0043] Fig. 12C A user interface for setting up sensor-actuator groupings associated with automated operations of a network device is shown according to some example embodiments.

[0044] Fig.12D A user interface for providing feedback to actuator actions performed based on an automation rules table is shown according to some example embodiments.

[0045] Fig.13 is a flow chart of a method suitable for automated operation of network devices within a home communication network, according to some exemplary embodiments.

[0046] Fig.14 is a block diagram illustrating a representative software architecture according to some example embodiments, which may be used in conjunction with the various device hardware described herein.

[0047] Fig.15 is a block diagram illustrating circuits of devices implementing algorithms and executing methods according to some exemplary embodiments. DETAILED DESCRIPTION

[0048] First, it should be understood that although the following provides an exemplary implementation of one or more embodiments, Figure 1-15 The disclosed systems and / or methods described may be implemented using any number of techniques, whether currently known or not yet in existence. The present invention should in no way be limited to the illustrative embodiments, drawings, and techniques illustrated below, including the exemplary designs and embodiments illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0049] The following is a detailed description in conjunction with the accompanying drawings, which are part of the description and show by way of diagrammatic illustration specific embodiments in which the present invention may be implemented. These embodiments are described in sufficient detail to enable those skilled in the art to practice the subject matter of the present invention, and it should be understood that other embodiments may be used and that structural, logical, and electrical changes may be made without departing from the scope of the present disclosure. Therefore, the exemplary embodiments described below are not intended to be limiting, and the scope of the present invention is defined by the appended claims.

[0050] As used herein, the term "actuator" may refer to a device or device component that is physically operated by a user (e.g., a light switch, a thermostat, a device knob). In some aspects, an "actuator" may also refer to a device that can be controlled or manipulated by voice, motion, or even by other means such as brain waves, emotions, or network signaling. As used herein, the term "sensor" may refer to a device for sensing environmental conditions (e.g., external temperature, humidity, wind direction, etc.) and non-environmental conditions (e.g., motion detection, sound detection, etc.), and providing the sensed data to one or more data collection modules or processors.

[0051] Techniques for autonomous home operation disclosed herein may include continuously monitoring connected sensors and actuators present in a communication network (e.g., a home communication network in a user's home), periodically collecting sensor values ​​and actuator activations, and performing learning computations that identify correlations between measured sensor values ​​and actuator activations. The goal of the learning process is to identify correlations between measured sensor values ​​and actuator activations and, through the correlations, identify (1) the activities performed by the user, (2) how the user performs different activities (i.e., a series of actuator activations), (3) user habits, and (4) use this information to generate a table of automation rules for performing automated actions using available actuator devices. In one embodiment, these automated actions can mimic the actuator activations that a user would typically perform during a given activity, but can also perform different (e.g., more efficient) executions of the activity. The overall goal of the home communication network is to logically combine the different sensors and actuators available and create intelligent sensing and automation device activations to create an automatically enhanced (in terms of comfort, feature set, safety, efficiency, etc.) home environment.

[0052] In some aspects, correlations can be identified using a machine learning (ML) program (also known as an artificial intelligence (AI) engine), which can be trained using, for example, a training data set. The ML program can be used to generate configuration data (e.g., an automation rule table) based on the identified correlations. The configuration data can include multiple trigger diagrams, each with a corresponding action diagram. The trigger diagram includes one or more trigger conditions, such as sensor values, actuator events, and external information values, which can all be connected by logical operators. When any trigger event in a given trigger diagram (modified by a logical operator) is detected to have been completed, the actions in the corresponding action diagram will be automatically executed. These actions can include executing one or more events related to the actuator.

[0053] Unlike existing home automation solutions where a user must explicitly provide rules for triggering actions of one or more devices within the system, the techniques disclosed herein can be used to automatically generate / update configuration data for home automation (e.g., an automation rule table with multiple trigger graphs and corresponding action graphs) and automatically execute the actions of the action graphs when a trigger condition is detected from one or more of the trigger graphs. In addition, machine learning techniques can be used to train ML programs within the home using local sensor and actuator data. In some aspects, learning can be combined so that it is performed in a remote environment (e.g., the ML program can be trained in the cloud), relying in part on data collected in the home, and combining other training methods and other training data (e.g., data collected at other locations such as other homes). The home communication network disclosed herein can also provide an interface for third-party device and application vendors to connect their devices and applications to the network and through this interface (1) use the available data, processing power and models / algorithms (e.g., trained ML programs) within the home network and (2) provide additional capabilities (e.g., additional computing power or sensing capabilities) for the automated home environment.

[0054] Figure 1 is a high-level system overview diagram of a home communication network 100 using automated operation of network devices according to some exemplary embodiments. Figure 1 , the home communication network 100 may include sensors 102 and actuators 104, which may be communicatively coupled to a computing device 108, such as a router or a cell phone. In some aspects, the sensors 102 may include temperature sensors, humidity sensors, motion sensors, heat sensors, smoke sensors, or any other type of sensor. The actuators 104 may include thermostats, switches, device knobs, or any other type of device that a user 112 may interact with and change its state through actions 114. As used herein, the term "actuator event" (or "actuator activation") represents an interaction of a user 112 with one or more of the actuators 104 (e.g., through actions 114) that changes the actuator state (e.g., a switch is turned on or off, a thermostat is adjusted, a shade is raised, etc.).

[0055] The computing device 108 can maintain a database 116 of sensor states associated with the sensors 102 and actuator events associated with the actuators 104. In some aspects, the sensor states and actuator events can be transmitted directly from the sensors 102 and actuators 104 to the computing device 108 for further processing and storage in the database 116. In some aspects, the sensor states and actuator events can be initially transmitted to the hub device 106 (e.g., a smart home hub device) and then transmitted to the computing device 108. The computing device 108 can also include an AI engine 118, which can be used to generate an automation rule table 120 based on the sensor data and actuator events, as further described below. In some aspects, the AI ​​engine 118 can also use external information, such as information from a cloud-based provider 124, through interaction 122. The cloud-based provider 124 can be used to perform, for example, the generation and management of automation rule tables 120 associated with reinforcement learning training and other joint computing (e.g., assisting multiple users at multiple locations).

[0056] The main components of the communication network 100 include processes for collecting and recording actuator activation values, sensor measurements, and performing calculations on sensor data, actuator data, and additional data (e.g., externally available information from a cloud-based provider 124) in order to create an automation rule table 120 for automated operations of home appliances or other devices connected within the home communication network 100 or trigger actions for automated operations (e.g., based on the table 120). The creation of the automation rule table 120 responsible for defining automated actions can be generated through statistical models, training processes of different machine learning methods performed locally (e.g., by the device 108) or remotely (e.g., by the cloud-based provider 124), or in a distributed hybrid manner (e.g., by the cloud-based provider 124 and one or more clients of the cloud-based provider 124).

[0057] In some aspects, the home communication network 100 provides additional functionality, such as personalized computing by means of a federated learning approach, the ability to share / distribute learned knowledge to service providers and / or other users in different locations, and the ability to seamlessly extend the system using third-party devices and applications that provide additional capabilities to the system. While some aspects of the home communication network 100 can operate autonomously and require no user interaction, the home communication network 100 can be used to provide information about its internals, such as the functionality it provides and the automated actions that have been learned and their triggers, which is provided to the user 112 via the user interface console 110. The user interface console 110 can also serve as an interface for the user to influence portions of the operations performed by the device 108 by providing feedback, setting goals for the AI ​​engine 118, setting sensor-actuator relationships and groupings, and the like.

[0058] Figure 2 According to some exemplary embodiments Figure 1 A high-level architectural design of a home communication network. Figure 2 , the AI ​​engine 118 may be operable to receive sensor data 204 as a result of sensor output sampling 202 of the sensor 102 and actuator event data 208 as a result of user actuator events 206 of the actuator 104. The data 204 and 208 may be time stamped.

[0059] In an exemplary embodiment, the AI ​​engine 118 may use one or more AI-related computing techniques 210 to generate the automation rules table 120. For example, the techniques 210 may include the use of statistical models, such as probabilistic relational models and pattern recognition techniques. The techniques 210 may also include machine learning (ML) models, such as reinforcement learning (RL), interactive reinforcement learning (IRL), continuous learning algorithms, neural networks (NN), recurrent neural networks (RNN), long short-term memory (LSTM), attention-based networks (ABN), and other techniques. The AI ​​engine 118 may also use data from the action trigger log 212 and interactions with the cloud-based provider 124. The action trigger log 212 may include a list of actions (e.g., actions performed by actuators) that have been automatically triggered based on one or more triggers specified in the automation rules table 120.

[0060] In an exemplary embodiment, when generating or updating the automation rules table 120, the AI ​​engine 118 may further consider user input (e.g., received via the UI console 110), such as goals 214 and predefined sensor-actuator relationships 216. The goals 214 may include user input that specifies desired comfort, energy efficiency or economy, safety, or any other user-defined goal. The predefined sensor-actuator relationships 216 may include one or more groupings of sensors and actuators, which may be based on common locations (e.g., sensors and actuators within a given room or other location in a user's home may be grouped together for increased efficiency).

[0061] Figure 3 is a diagram showing a method that can be used with Figure 1 A block diagram of implementation details of a computing device 300 for use in conjunction with a home communications network. Figure 3 , the computing device 300 may communicate with Figure 1 Device 300 may implement a connection layer 330 that may be used to communicate with sensors 338, actuators 340, and data hub 342. Sensors 338, actuators 340, and data hub 342 may have the same functionality as sensors 102, actuators 104, and hub 106, respectively.

[0062] Device 300 may be used to provide an operating environment 308 and a storage environment 328. Operating environment 308 may be used to implement AI engine 306 and suggestion system 304. Suggestion system 304 may be used to provide one or more suggestions for installing additional sensors and actuators or removing existing sensors and actuators to improve AI engine performance or to improve AI engine performance based on user-provided input (e.g., 214 and 216).

[0063] The storage environment 328 may be used to configure the algorithm / model storage 314 (for storing models used by the AI ​​engine 118, such as 210), the sensor log 316 (storing sensor data from the sensor 338), the actuator log 318 (storing actuator event data associated with the actuator 340), the automation rule table 320 generated by the AI ​​engine 306, the action trigger log 322 (similar to the log 212 that lists the automation actions performed based on the automation rule table), and the user-defined policies and goals 324 (such as 214 and 216). The device 300 may also perform a data compression function 312 associated with any data maintained by the storage environment 328. In addition, the data maintained by the storage environment 328 may be accessed through the access control function 310.

[0064] In some aspects, the cloud-based provider 124 can provide cloud-based training 332 for the AI ​​engine 306. In this regard, raw data associated with a user of the device 300 (e.g., non-privacy sensitive or cleansed data) can be exchanged with the cloud-based provider 124. The AI ​​engine 306 can also receive external information 334, which can be used to generate the automation rules table 320.

[0065] Device 300 may also include an application programming interface (API) 302. In this regard, third-party devices or applications 336 may use API 302 to interface with device 300 (e.g., to assist AI engine 306 with additional computing power, additional sensor and actuator data, etc.). Voice assistant 344 may also use API 302 and provide a user interface for voice-based interaction between device 300 and a user.

[0066] Figure 4 is a diagram showing a method that can be used with Figure 1 4 is a block diagram of a data collection process 400 for use in conjunction with a home communication network 100. More specifically, the process 400 can be used to obtain data for creating an automation rule table 120 for performing automated actions within the home communication network 100. Sensor values ​​can be continuously monitored and recorded for later processing. In addition, actuator activations triggered by a user or by some other means (i.e., automated operation of a third-party device) are also closely monitored and recorded. For example, the data collector module 402 can obtain sensor values ​​410 marked with time by a timer 408, and actuator values ​​414 associated with one or more user activation events 412. The data collector module 402 can also receive external information from an external source 418, which is marked with time by a timer 416. The external information can be pushed to the data collector module 402 by the external source 418, or the data collector module 402 can receive external data through a pull function 420.

[0067] Actuator activations may occur as a response to changes in the environment, or there may be correlations between different device activations. Correlations between different activations may be related to (i) the same environmental change, may be (ii) some form of temporal correlation (i.e., occur for temporally related reasons), or (iii) may be related to the same activity being performed. Changes in the environment may be sensed and recorded via sensor values, which may be analyzed at a later stage, and the relationship between environmental changes and device activations may be identified by the AI ​​engine 118. In addition to data collected locally from local sensors and actuators, the home communication network 100 may rely on and collect data from external sources 418 (e.g., the data collector module 402 may collect external data via the Internet from sources such as weather forecast sources, financial market services, or other types of services including information services from local governments, city governments, schools, and services that provide location updates for family members).

[0068] Once the data is received by the data collector module 402, the data may be forwarded for processing 422. The memory 406 may be used to store sensor data 424, actuator data 426, and external information data 428. The memory 406 may be local and remote, with implemented data privacy constraints. In addition, the memory 406 may include a database, a file system, an in-memory storage solution, or some alternative storage solution. Once the data is received by the data collector module 402, the data may be moved directly to the data storage 406, or pre-processed by data compression, filtering, cleaning, encoding, data cleaning 404 for more efficient or more secure storage. At the underlying data storage 406, due to different formatting requirements and usage of the AI ​​engine 118, the sensor data 424, actuator data 426, and data from external sources 428 may be stored in separate structures.

[0069] even though Figure 4 The data collector module 402 is shown as being separate from the data processing 422 and 404 functions, but in an exemplary embodiment, data collection and additional processing such as 422 and 404 may be performed by a single module (e.g., Fig.14 Data collection and management module 1464 or Fig.15 1570) in the implementation.

[0070] Figure 5 is a block diagram illustrating a process 500 for creating an automation rules table that may be used for automated operations of a network device according to some exemplary embodiments. Figure 5 , the data storage module 508 is used to store sensor data 502, actuator data 504, and external information data 506, which may be pre-processed (e.g., at 404). In some aspects, the data collection module 510 (which may be the same as the data collector module 402) may directly transmit the collected sensor and actuator data to the data storage module 508. In the calculation / training process 512, different mathematical, statistical, or machine learning models may be executed or trained by the AI ​​engine 118 to generate the automation rule table 514.

[0071] In an exemplary embodiment, the automation rules table 514 may include a trigger graph 516 and a corresponding action graph 524. The trigger graph 516 includes nodes that describe one or more trigger events, such as certain sensor values ​​518 (e.g., node S0), actuator events 520 (e.g., node A5), external information values ​​522 (e.g., node EI3), and any combination thereof. For example, the nodes within the trigger graph 516 may be connected by logical operations such as 530A and 530B. In this regard, when the sensor value S0 is detected together with the actuator event A5, or when the external information value EI3 is received, Figure 5The trigger diagram 516 shown in FIG. 5 can trigger the execution of actions specified by the action diagram 524. When the data storage module 508 stores the sensor values ​​518, the actuator events 520, and the external information values ​​522, the sensor values ​​518, the actuator events 520, and the external information values ​​522 can be indexed.

[0072] In an exemplary embodiment, the action diagram 524 may include multiple nodes 526A-526G coupled to one or more logical operators such as 528. Once a trigger event (e.g., S0 and A5, or just EI3) specified by the trigger diagram 516 is detected, the actions specified by the various nodes in the action diagram 524 can be automatically executed. In this regard, the action diagram 524 can be used to describe the actions to be performed when the trigger event specified by the trigger diagram is detected. Each of the nodes 526A-526G can specify an actuator event that must be executed. The action diagram 524 can further define the order of different actions to be performed, and can also specify timing constraints between actions or other conditional values ​​between different actions. For example, the link / edge between each two connected nodes can be marked with the time delay between the execution of the actuator event associated with each node. As an alternative, the length of each link / edge between a pair of connected nodes can be proportional to the delay. In an exemplary embodiment, the automation rule table 514 can include multiple trigger diagrams, each of which has a corresponding action diagram. Furthermore, the automation rules table 514 may be continuously updated based on the sensor data 502 , the actuator data 504 , the external information data 506 , user input (eg, 214 and 216 ), or other criteria.

[0073] Fig. 6A 6 is a block diagram 600A illustrating the training and use of a machine learning (ML) program 626 according to some exemplary embodiments. In some exemplary embodiments, a machine-learning program (MLP), also referred to as a machine learning algorithm or tool, is used to perform operations associated with associating data and with generating and maintaining automation rule tables, such as table 514, associated with autonomous home operations.

[0074] like Fig. 6AAs shown, machine learning program training 624 can be performed based on training data 628. For example, training data 628 can include one or more initial trigger graphs such as 516 and corresponding action graphs 524, 516 and 524 can be predefined (e.g., received from a cloud-based provider or generated by a user of the home communication network 100). During machine learning program training 624, features 602 can be evaluated based on training data 628 in order to further train the machine learning program. Machine learning program training 624 produces a trained machine learning program 626, which can include one or more classifiers 634, which can be used to provide an evaluation 632 based on new data 630. For example, after generating the trained ML program 626, new sensor data, actuator event data, and external information data can be evaluated by the trained ML program 626 to generate an evaluation 632. The evaluation 632 can include correlations between different sensor data and actuator event data, which can be used to generate an automation rule table, such as table 514. Machine learning program training 624 and trained ML program 626 may be part of the AI ​​engine 118 .

[0075] In some aspects, features 602 used during machine learning program training 624 may include one or more of the following: sensor state data 604, ..., 606 from multiple sensors 1, ..., N; actuator event data 610, ..., 612 from multiple actuators 1, ..., K; external information source data 614, ..., 616 from multiple external sources 1, ..., L; timer data 608 associated with sensor state data, actuator event data, or external information source data; user communication information 618; user data 620; and past user behavior data 622 (e.g., past user preferences, manual actuator adjustments, etc.).

[0076] Machine learning is a field of study that gives computers the ability to learn without being explicitly programmed. Machine learning explores the study and construction of algorithms, also referred to herein as machine learning tools, that can learn from existing data, correlate data, and make predictions about new data. Such machine learning tools operate by building models from exemplary training data (e.g., 628) in order to make data-driven predictions or decisions, represented as outputs or evaluations 632. Although exemplary embodiments are presented with respect to several machine learning tools, the principles presented herein can be applied to other machine learning tools.

[0077] In some exemplary embodiments, different machine learning tools may be used. For example, Logistic Regression (LR), Naive Bayes, Random Forest (RF), Neural Network (NN), Matrix Decomposition and Support Vector Machine (SVM) tools may be used to correlate sensor data, actuator data, external information data, and generate automation rule tables related to autonomous home operations.

[0078] Two common types of problems in machine learning are classification problems and regression problems. Classification problems, also known as categorization problems, aim to classify an item into one of several class values ​​(e.g., is this object an apple or an orange?). Regression algorithms aim to quantify some items (e.g., by providing a value of a real number). In some embodiments, an exemplary machine learning algorithm provides correlations between sensor data, actuator data, and external information data. The machine learning algorithm uses training data 628 to find correlations between identified features 602 that affect the results.

[0079] The machine learning algorithm analyzes new data 630 using features 602 to generate estimates 632 (e.g., data correlations of sensor data and actuator event data). Features 602 include a single measurable property of the phenomenon being observed and used to train the ML program. The concept of a feature is related to the concept of explanatory variables used in statistical techniques such as linear regression. Selecting informative, discriminative, and independent features is very important for the effective operation of MLPs in pattern recognition, classification, and regression. Features can be of different types, such as numeric features, strings, and graphics.

[0080] The machine learning algorithm utilizes the training data 628 to find correlations between the identified features 602 that influence the outcome or evaluation 632. In some exemplary embodiments, the training data 628 includes labeled data, which is known data of one or more identified features 602 and one or more outcomes, such as a known trigger graph that specifies sensor values, actuator events, and external information values, and a known action graph corresponding to the trigger graph that specifies actuator events that will be automatically executed when the conditions of the trigger graph are met.

[0081] The machine learning program is trained at operation 624 using the training data 628 and the identified features 602. The result of the training is a trained machine learning program 626. When the machine learning program 626 is used to perform an evaluation, the new data 630 is provided as an input to the trained machine learning program 626, and the machine learning program 626 generates an evaluation 632 as an output.

[0082] Figure 6B600B is a diagram illustrating the generation of a trained ML program 644 using a neural network 642 according to some exemplary embodiments. Figure 6B , source data 640 can be analyzed by neural network 642 (or another type of machine learning algorithm or technique) to generate trained machine learning program 644 (which can be the same as trained ML program 626). Source data 640 can include a training data set, such as 628, and data identified by one or more of features 602.

[0083] Machine learning techniques train models to accurately predict data fed into the model (e.g., what a user said in a given utterance; whether a noun is a person, place, or thing; what the weather will be like tomorrow). During the learning phase, the model is developed against an input training data set to optimize the model so that it correctly predicts the output given the input. In general, the learning phase can be supervised, semi-supervised, or unsupervised, indicating the descending level of providing the "correct" output corresponding to the training input. In the supervised learning phase, all outputs are provided to the model, guiding the model to develop general rules or algorithms that map inputs to outputs. In contrast, in the unsupervised learning phase, the desired outputs are not provided for the inputs so that the model can develop its own rules to discover relationships in the training data set. In the semi-supervised learning phase, an incompletely labeled training set is provided, where some outputs of the training data set are known and some are unknown.

[0084] The model can be run against the training data set for several stages (e.g., iterations), during which the training data set is repeatedly fed into the model to refine its results. For example, in the supervised learning stage, a model is developed to predict the output for a given set of inputs (e.g., data 640) and is evaluated in several stages to more reliably provide an output that is specified as a given input corresponding to the maximum number of inputs of the training data set. In another example, for the unsupervised learning stage, a model is developed to cluster the data set into n groups, and the model is evaluated in several stages for its consistency in placing a given input into a given group, as well as its reliability in producing the n desired clusters at each stage.

[0085] Once a stage has been run, the model is evaluated and the values ​​of its variables are adjusted to try to better refine the model in an iterative manner. In various aspects, the evaluation is biased towards false negatives, biased towards false positives, or uniformly biased towards the overall accuracy of the model. Depending on the machine learning technique used, the values ​​can be adjusted in a variety of ways. For example, in a genetic or evolutionary algorithm, the values ​​of the model that are most successful in predicting the desired output are used to develop the values ​​used by the model in subsequent stages, which may include random changes / mutations to provide additional data points. Those of ordinary skill in the art will be familiar with several other machine learning algorithms that can be applied to the present invention, including linear regression, random forests, decision tree learning, neural networks, deep neural networks, etc.

[0086] Each model develops rules or algorithms in several stages by changing the values ​​of one or more variables that affect the input to more closely map to the desired outcome, but perfect accuracy and precision may not be achieved because the training data set may vary, and preferably vary greatly. Therefore, some of the stages that make up the learning phase can be set to a given number of trials or a fixed amount of time / computation, or these stages can be terminated before reaching this number / amount when the accuracy of a given model is high enough or low enough or reaches an accuracy plateau. For example, if the training phase is designed to run n stages and generate a model with at least 95% accuracy, and such a model is generated before the nth stage, the learning phase can be terminated early and the generated model that meets the final target accuracy threshold is used. Similarly, if the inaccuracy of a given model is sufficient to meet the random chance threshold (for example, the model is only 55% accurate in determining the true / false output for a given input), the learning phase of this model can be terminated early, although other models in the learning phase can continue to be trained. Similarly, when a given model continues to provide similar accuracy or wavers in results over multiple stages - having reached a performance plateau - the learning stage for the given model can be terminated before reaching the stage number / computation amount.

[0087] Once the learning phase is complete, the model is finalized. In some exemplary embodiments, the finalized model is evaluated against a test criterion. In a first example, a test data set including known outputs of its inputs is fed into the finalized model to determine the accuracy of the model when processing data that has not been trained. In a second example, false positives or false negatives can be used to evaluate the finalized model. In a third example, the descriptions between the data clusters in each model are used to select the model that produces the clearest boundaries for its data clusters.

[0088] In some exemplary embodiments, the machine learning program 644 is trained by a neural network 642 (e.g., a deep learning, deep convolutional, or recursive neural network) that includes a series of "neurons," such as Long Short Term Memory (LSTM) nodes arranged in the network. Neurons are architectural elements used for data processing and artificial intelligence, particularly machine learning, and include memories that can determine when to "remember" and when to "forget" the values ​​they hold based on the weights of the inputs provided to a given neuron. Each of the neurons used here is used to accept a predetermined number of inputs from other neurons in the network, thereby providing relational and sub-relational outputs for the content of the analyzed frame. Individual neurons can be linked together and / or organized into tree structures in various configurations of neural networks to provide interactive and relational learning modeling to determine how each of the frames in an utterance relates to each other.

[0089] For example, an LSTM as a neuron includes several gates for processing input vectors (e.g., phonemes from an utterance), storage cells, and output vectors (e.g., context representations). The input gate and output gate control the flow of information into and out of the storage cell, respectively, while the forget gate optionally removes information from the storage cell based on the input of earlier linked units in the neural network. The weights and bias vectors of the various gates are adjusted during the training phase, and once the training phase is complete, these weights and biases are finalized for normal operation. Those skilled in the art will appreciate that neurons and neural networks can be constructed programmatically (e.g., through software instructions) or by dedicated hardware that links each neuron to form a neural network.

[0090] Neural networks use features to analyze data to generate estimates (e.g., identify speech units). A feature is a single measurable property of an observed phenomenon. The concept of a feature is related to the concept of explanatory variables used in statistical techniques such as linear regression. In addition, deep features represent the outputs of nodes in the hidden layers of a deep neural network.

[0091] Neural Networks

[0092] A neural network (e.g., 642), sometimes called an artificial neural network, is a computing system based on considerations of biological neural networks in animal brains. These systems gradually improve in performance, i.e., learn, to perform tasks, typically without the need for task-specific programming. For example, in image recognition, a neural network can be taught to recognize images containing objects by analyzing example images that have been labeled with the names of the objects, and after learning the objects and names, the results of the analysis can be used to recognize objects in unlabeled images. A neural network is based on a collection of connected units called neurons, where each connection between neurons, called a synapse, can transmit a unidirectional signal whose activation strength varies with the strength of the connection. A receiving neuron can activate and propagate a signal to downstream neurons connected to it, typically based on whether the combined input signal from potentially many transmitting neurons has sufficient strength, where strength is a parameter.

[0093] A deep neural network (DNN) is a stacked neural network consisting of multiple layers. These layers consist of nodes, which are locations where computations occur, loosely patterned on neurons in the human brain, and which fire when they encounter enough stimulation. Nodes combine inputs from the data with a set of coefficients or weights that amplify or suppress the inputs, which assign importance to the input for the task the algorithm is trying to learn. These input-weight products are summed, and the resulting sum is passed through the node's activation function to determine whether and to what extent the signal travels further through the network to affect the final result. DNNs use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output of the previous layer as input. Higher-level features are derived from lower-level features to form a hierarchical representation. The layer after the input layer can be a convolutional layer, which produces feature maps, which are the filtered results of the input and are used by the next convolutional layer.

[0094] In the training of DNN architecture, regression is constructed as a set of statistical processes for estimating the relationship between variables, which can include the minimization of a cost function. The cost function can be implemented as a function to return a number that represents how well the neural network performs in mapping training examples to correct the output. During training, if the cost function value is not within a predetermined range, based on known training images, backpropagation is used, where backpropagation is a common method for training artificial neural networks, which are used with optimization methods such as stochastic gradient descent (SGD) method.

[0095] The use of backpropagation can include both propagation and weight updates. When an input is presented to a neural network, it propagates forward through the neural network layer by layer until it reaches the output layer. The output of the neural network is then compared to the desired output using a cost function, and an error value is calculated for each of the nodes in the output layer. The error values ​​are propagated backward starting from the output until each node has an associated error value that roughly represents its contribution to the original output. Backpropagation can use these error values ​​to calculate the gradient of the cost function with respect to the weights in the neural network. The calculated gradients are fed to the selected optimization method to update the weights in an attempt to minimize the cost function.

[0096] As a result of the above-described training ML program 626 and generation of the automation rules table 514, the AI ​​engine 118 and devices 108 may gradually take over the operation of some portions of the home communication network 100 after a certain period of time required to reach a certain level of accuracy of the trained ML program 626. However, the automated actions performed based on the automation rules table 514 may not be accurate and may not conform to the user's preferences or the user's requirements. In addition, user routines may change over time, so a user may explicitly activate some actuators even though the system recently activated the same actuators. If these device activations are in response to automated actions based on the automation rules table 514, the AI ​​engine 118 may view the device activations as "corrective actions" to the automated actions. In this regard, the device activations may be used by the trained ML program 626 to retrain or recalibrate itself, such as Figure 7 shown, for more accurate automated equipment operation in the future.

[0097] Figure 7 is a flow chart illustrating a correction / retraining process 700 of an automation rule table that may be used for automated operation of a network device according to some exemplary embodiments. Figure 7 , when a user-triggered actuator activation event may be detected, the correction / retraining process 700 may be performed by the AI ​​engine 118 and may begin at operation 702. In some aspects, the actuator activation event may be triggered by the user at a specific time after a different actuator event is automatically triggered based on an automation rule table.

[0098] Once the user triggers the actuator activation within a specific time frame after the automation action occurs, it can be determined whether the user-triggered actuator activation event is in response to an automation action, such as an actuator event automatically triggered based on the automation rules table, at operation 704. At operation 708, if it is determined that the user-triggered actuator activation event is in response to an automation action defined by the automation rules table, the AI ​​engine 118 can call a function and run an algorithm that can be executed during a retraining process, which will refine the rules of the automation rules table 514 to more accurately perform future operations, and will update the automation rules table with the adjusted rules at operation 710.

[0099] Figure 7 The right side of the diagram shows in more detail the operations 708 for retraining the ML program 626 and refining the automation rules table 514. Once the user-triggered actuator activation 702 is detected as a response to the automated action performed by the system, the retraining process of the ML program 626 begins. The training engine 712, which can be part of the AI ​​engine 118, can use the actuator activation performed by the user, and based on the amount of change required to complete the user-initiated actuator activation, the training engine 712 can create rewards (starting with negative reward values) to cause the ML program 714 to change its behavior to get closer to the desired action. In response, the ML program 714 will output an adjusted action that it will take next time under the same triggering situation. The training engine 712 evaluates the adjusted action and outputs a new reward function and declares the simulated environment resulting from the adjusted action. This process can continue until the adjusted action created by the ML program 714 is within a certain threshold of the desired action determined by the training engine 712 based on the user's activation 702.

[0100] Figure 8 is a flow chart illustrating a process 800 of reasoning and executing an action graph from an automation rules table based on matching trigger conditions according to some exemplary embodiments. Figure 8 The functions shown in can be generated and managed by the automation rule table module (e.g., Fig.14 1462 or Fig.15 1565) or executed by another module in device 108.

[0101] See also Figure 8, the sensor data 802, the actuator event data 804, and the external information data 806 may be continuously monitored for changes or updates at operation 808. Upon detection of such changes or updates, at operation 810, the monitored data may be matched to one or more trigger conditions specified within a trigger diagram in the automation rules table 514. At operation 812, a determination may be made as to whether a rule with a matching trigger condition is found within the automation rules table 514. If such a rule has a matching trigger condition within a trigger diagram within the automation rules table, then at operation 816, an action diagram corresponding to the located trigger diagram may be executed. If such a rule is not located within the automation rules table, then at operation 814, processing may end without any actuator activation.

[0102] In some aspects, the techniques disclosed herein can be used to support learning activities and user habits in a federated manner across multiple home locations and cloud architectures. More specifically, rules or models, such as an automation rule table created by one home, can be passed to another home to speed up or even eliminate the learning or relearning process and improve the accuracy of the rules and ML models used to generate the table. Fig.9A , Fig. 9B as well as Fig. 9C Various methods for federated computing related to automated operation of one or more network devices in a home are shown according to some exemplary embodiments. Fig.9A An embodiment 900A is shown in which a service provider (e.g., cloud architecture 906) transmits a new model 908 to a home 902. As used herein, the term "model" refers to a trained machine learning program that can be used to generate an automation rule table. In the home 902, the new model 90 can be retrained / corrected at operation 910 based on actuator activations triggered by occupants of the home 902. In aspects where the service provider 906 does not provide an initial model, a new model can be generated by the home 902. At operation 912, the home 902 can transmit its newly built model or the difference between the initial model provided by the provider 906 and the results of its retrained model to the service provider 906. The provider 906 can perform further calculations at operation 914, such as adjusting its initial model based on the difference provided by the home 902, cleaning or deleting some parts of the model, and finally transmitting the adjusted model 916 to the home 904 for adoption.

[0103] Fig. 9B A different embodiment 900B is shown in which a household can provide updates to other households in a more direct manner, without having to provide updates through a service provider. Fig. 9B, the new model 920 may be transmitted by the service provider 906 to the home 902 and the home 904. The retraining / correction may be performed at operation 922 within the home 902. At operation 924, the home 902 may transmit the newly constructed model or the difference between the initial model and the retrained model to the home 904. At operation 926, the computing resources at the home 904 may perform local computation of the model for automating devices within the home communication network based on the initial model 920 received from the service provider 906 and the difference between the model transmitted by the home 902 at operation 924.

[0104] Fig. 9C Embodiment 900C is shown in which household 902 generates a new model 930 at its location and shares the new model 930 with service provider 906 at operation 932. Service provider 906 may perform additional processing 934 (e.g., validation, cleansing, fine-tuning, and personalization) related to the received model to generate an updated model 936. Service provider 906 may then share updated model 936 with a different household, such as household 904.

[0105] Fig.10 A communication architecture 1000 with multiple homes and cloud-based ML training and automated rule table generation is shown according to some exemplary embodiments. Fig.10 , data 1018 from multiple homes 1002, 1004, ... 1006 are transmitted to a service provider 1008 (e.g., a cloud-based architecture) so that training and ML program creation are performed by the service provider 1008. The data 1018 may include sensor data, user activation event data, external information data, etc., which can be used by a single home to generate an automation rule table by the trained ML program. In some aspects, the service provider 1008 can pre-process the data, such as cleaning it before the training engine 1010 uses it. The training engine 1010 can rely entirely on the data 1018 from the home, or can also use additional data 1016 provided by the service provider as synthetic data for training the ML program 1012. During the training process, the service provider 1008 can verify the accuracy or confidence level of the trained model (i.e., the ML program 1012), and can only send the trained model to one or more of the homes, so as to predict and generate automation rule tables at various home locations once the trained model has reached a confidence level 1014 above a threshold.

[0106] Fig.11A and Fig. 11B According to some exemplary embodiments, connections to Figure 1 A block diagram of a home communications network with third-party devices and third-party applications. Fig.11A, diagram 1100 illustrates a third-party device 1102 and a third-party application 1104 connected to a home communication network, such as network 100. For example, the third-party device 1102 and the third-party application 1104 can connect to the home communication network 100 via the API 302. In this regard, the third-party device 1102 and the third-party application 1104 can access network resources 1106, data 1108, machine learning models 1110, AI engines 1112, and application registries 1114 that can be maintained in the home communication network in connection with configuring home automation functionality as discussed herein.

[0107] In some aspects, the third-party devices 1102 may include additional sensors or actuators that collect and send data within the home communication network. The third-party applications 1104 can leverage the data and knowledge available within the home communication network to provide value-added services on top of it (e.g., by providing enhanced AI capabilities to some parts or the entire AI engine). In this case, the home communication network can be used to provide some of the available services and resources it has, such as network connectivity to other components of the network (i.e., sensors, actuators, data storage, computing engines, etc.), access to some parts of the data, access to trained AI models, access to computing resources, and access to bookkeeping components such as application registry 1114.

[0108] Fig. 11B A state diagram associated with an example embodiment of a third-party device and third-party application installation and registration process is shown. For the third-party device 1102, the first step 1120 includes powering on the device by connecting the device to an external power source or activating its own power source. At operation 1122, the device can be connected to the home communication network to be able to communicate with the rest of the network. At operation 1124, the device 1102 can publish information about itself and can register for services provided within the home communication network 100. The information being published about the device can include a description of the device and the data it provides during operation, such as the device type, functionality, service type, data type, and structure. At operation 1126, the third-party device 1102 can begin operating within the home communication network 100.

[0109] When the application is loaded onto the appropriate component within the home communication network, such as the application execution engine, the registration process of the third-party application 1104 can begin at operation 1130, and then the application is connected to the network (at operation 1132) through an API such as API 302. At operation 1134, the third-party application 1104 registers some information with the home communication network 100, such as the function type, function set, data requirements, etc. of one or more functions that can be performed by the application. At operation 1136, the home communication network can enable and set various permissions associated with the application 1104.

[0110] Even though the goal of the home communication network 100 and the AI ​​engine 118 is to operate completely autonomously in providing home automation without requiring any human intervention (other than installing and starting up the necessary system components during initial system installation of the AI ​​engine and training set), the device 108 can be used to provide an overview and / or explain the knowledge and / or rules it has learned during its operation. This information can be provided to the user 112 through one or more user interfaces (e.g., a home network device display, a mobile phone application, a web service, etc.). The information provided to the user can include the sensor values ​​and actuator events that triggered the rules, as well as a description of the rules in the automation rule table, such as which automated actions are being taken when a particular rule is triggered. In addition, the system may provide statistical information about when the rules are triggered and with what probability (e.g., based on previous automatic executions of the actuator events in the automation rule table). Fig. 12A A user interface 1200A representing an automation rule table is shown according to some exemplary embodiments. For example, when the light intensity sensor reading of the exterior light sensor is "4" and the time of day is within the indicated range, an actuator event for turning on the dining light can be activated. Another trigger diagram can generate an actuator for event execution (e.g., adjusting the thermostat and indoor temperature to 72°F) from a corresponding action diagram based on the exterior temperature being below 60°F and the time of day being within the specified range.

[0111] In addition to providing information about what the AI ​​engine 118 has learned, the device 108 may also provide an interface for the user to set goals about how the AI ​​engine 118 should adjust its decisions. These goals may be related to safety, comfort, energy conservation, etc. Fig. 12B A user interface 1200B for setting user goals (such as goal 214) related to automated operation of a network device according to some exemplary embodiments is shown. In one embodiment, a user may specify a goal with the highest standard using only interface 1202B, while in another embodiment, and in conjunction with user interface 1204B, a user may set an importance level for each goal.

[0112] In addition to specifying the goals for the AI ​​engine 118, the user may also provide information about the relationships between the sensors and actuators (e.g., information 216). The training subsystem may use this grouping information to create a more accurate model or complete training in a shorter time. The information provided may include logical or physical connections between different sensors and actuators. Logical connections may include that they are located in the same physical location (i.e., room) or are related to the same service (i.e., home function). Physical connections may mean that they are physically connected so as to communicate directly or cooperate to collectively perform some function or service. Fig. 12C A user interface for setting up a sensor-actuator group 1200C associated with automated operations of network devices is shown according to some exemplary embodiments. For example, a home security system 1212C may be configured in the same group as a time sensor 1202C, a day of the week sensor 1204C, a light sensor 1206C, a motion detector 1208C, and an external camera 1210C.

[0113] Another way to get input from the user to improve the accuracy and speed of training can be to accept "like" or "dislike" type feedback. Such feedback can be collected through a traditional user interface or voice control. The feedback can be used to guide the training for more accurate predictions, thereby achieving better prediction capabilities in less time. In addition to requesting / accepting "like" and "dislike" type feedback, users can also specify details about what they like or dislike and how the system may correct the action (e.g., raise the temperature higher than during the previous automated operation). Fig.12D User interfaces 1202D and 1204D are shown for providing feedback to actuator actions performed based on an automation rules table, according to some example embodiments.

[0114] Fig.13 1300 is a flow chart of a method for automating operation of network devices within a home communication network according to some exemplary embodiments. The method 1300 includes operations 1302, 1304, and 1306. By way of example and not limitation, the method 1300 is described as being performed by the device 108 using Fig.14 Modules 1460-1464 (or Fig.151560, 1565, and 1570) are executed. In operation 1302, actuator actions of multiple actuators (e.g., 104) within a home communication network (e.g., 100) are detected. Each of the multiple actuators can be used to change the state of at least one network device within the home communication network. In operation 1304, the detected actuator actions can be associated with one or more sensor values ​​from multiple sensors within the home communication network to generate configuration data. For example, the trained ML program 626 can associate actuator event data 504 with sensor data 502 from multiple sensors 102 to generate configuration data (e.g., automation rule table 514). The configuration data (e.g., table 514) may include a trigger diagram (e.g., 516) having one or more trigger conditions (e.g., 518-522). The configuration data may also include an action diagram (e.g., 524) corresponding to the trigger diagram (e.g., 516). The action diagram may indicate one or more actuator actions (e.g., 526A-526G) associated with at least one of the plurality of actuators (e.g., 104). At operation 1306, when one or more trigger conditions in the trigger diagram are detected, at least one of the plurality of actuators may be triggered to perform the one or more actions indicated by the action diagram.

[0115] Fig.14 is a block diagram illustrating a representative software architecture 1400 that may be used in conjunction with the various device hardware described herein, according to some exemplary embodiments. Fig.14 This is merely a non-limiting example of a software architecture 1402, and it will be appreciated that many other architectures may be implemented to achieve the functionality described herein. The software architecture 1402 may be executed on hardware, such as Fig.15 Device 1500 includes a processor 1505, a memory 1510, storage 1515 and 1520, and I / O components 1525 and 1530, etc. A representative hardware layer 1404 is shown, which may represent, for example Fig.15 The representative hardware layer 1404 includes one or more processing units 1406 with associated executable instructions 1408. The executable instructions 1408 represent executable instructions of the software architecture 1402, including Figure 1-13 The hardware layer 1404 also includes a memory and / or storage module 1410, which also has executable instructions 1408. The hardware layer 1404 may also include other hardware 1412, which represents any other hardware of the hardware layer 1404, such as other hardware shown as part of the device 1500.

[0116] exist Fig.14In the exemplary architecture of FIG. 1402, the software architecture 1402 can be conceptualized as a stack of layers, where each layer provides specific functionality. For example, the software architecture 1402 can include layers such as an operating system 1414, a library 1416, a framework / middleware 1418, an application 1420, and a presentation layer 1444. In operation, the application 1420 and / or other components within the layer can call an application programming interface (API) call 1424 through the software stack and receive a response, return value, etc., as shown by a message 1426, in response to the API call 1424. Fig.14 The layers shown in are representative in nature, and not all software architectures 1402 have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 1418, while other operating systems may provide such layers. Other software architectures may include additional or different layers.

[0117] The operating system 1414 can manage hardware resources and provide public services. The operating system 1414 may include, for example, a kernel 1428, a service 1430, a driver 1432, an AI engine management module 1460, a table of automated rules (TAR) generation and management module 1462, and a data collection and management module 1464. The kernel 1428 can act as an abstraction layer between the hardware and other software layers. For example, the kernel 1428 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The service 1430 can provide other public services to other software layers. The driver 1432 may be responsible for controlling or connecting to the underlying hardware. For example, the driver 1432 may include a display driver, a camera driver, a Bluetooth drivers, flash drives, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), drivers, audio drivers, power management drivers, etc., depending on the hardware configuration.

[0118] In some aspects, the AI ​​engine management module 1460 may include suitable circuitry, logic, interfaces and / or code and may be operable to perform the functions described herein in relation to the AI ​​engine 118, such as training an ML program, generating a trained ML program, retraining a trained ML program (e.g., Figure 7 ), and manage joint functions associated with trained ML programs (e.g., as combined Figures 9A-9C). The TAR generation and management module 1462 may include suitable circuitry, logic, interfaces and / or code and may be operable to perform functions associated with a TAR, such as generating a TAR (e.g., 514) using a trained ML program (e.g., 626), monitoring sensor values ​​to detect matches with trigger graphs within the TAR, and executing one or more actuator events specified by a corresponding action graph within the TAR (e.g., as combined with Figure 8 406 ). The data collection and management module 1464 may include appropriate circuitry, logic, interfaces and / or code and may be operable to perform data-related functions such as data collection (e.g., as performed by modules 402 and 510), data processing (e.g., as performed at operations 404 and 422), and data storage (e.g., as performed by module 508 at operation 406). In some aspects, one or more of the modules 1460, 1462, and 1464 may be combined into a single module.

[0119] The library 1416 may provide a common infrastructure that may be used by the application 1420 and / or other components and / or layers. The library 1416 generally provides a way to allow other software modules to perform tasks more easily than directly connecting to the underlying operating system 1414 functions (e.g., kernel 1428, service 1430, driver 1432 and / or module 1460-1464). The library 1416 may include a system library 1434 (e.g., C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1416 may include an API library 1436, such as a media library (e.g., a library that supports the presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG), a graphics library (e.g., an OpenGL framework that can be used to render 2D and 3D in graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a network library (e.g., WebKit that can provide network browsing functions), etc. The library 1416 may also include a variety of other libraries 1438 to provide many other APIs to the application 1420 and other software components / modules.

[0120] Framework / middleware 1418 (also sometimes referred to as middleware) may provide a higher level common infrastructure that may be utilized by applications 1420 and / or other software components / modules. For example, framework / middleware 1418 may provide various graphical user interface (GUI) functions, high level resource management, high level location services, etc. Framework / middleware 1418 may provide a wide range of other APIs that may be used by applications 1420 and / or other software components / modules, some of which may be specific to a particular operating system 1414 or platform.

[0121] Applications 1420 include built-in applications 1440 and / or third-party applications 1442. Examples of representative built-in applications 1440 may include, but are not limited to, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, and / or a game application. Third-party applications 1442 may include any of built-in applications 1440 as well as a wide range of other applications. In a particular example, third-party applications 1442 (e.g., applications created by an entity other than the vendor of a particular platform using Android TM or iOS TM Software development kit (SDK) applications can be developed on platforms such as iOS TM 、Android TM , Mobile software running on the mobile operating system of the phone or other mobile operating systems. In this example, third-party application 942 can call API calls 1424 provided by a mobile operating system such as operating system 1414 to implement the functions described herein.

[0122] Applications 1420 may utilize built-in operating system functionality (e.g., kernel 1428, services 1430, drivers 1432, and / or modules 1460-1464), libraries (e.g., system libraries 1434, API libraries 1436, and other libraries 1438), and framework / middleware 1418 to create a user interface to interact with a user of the system. Alternatively, or in addition, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 1444. In these systems, the application / module "logic" may be separated from the aspects of the application / module that interact with the user.

[0123] Some software architectures use virtual machines. Fig.14 In the example of , this is illustrated by virtual machine 1448. A virtual machine creates a software environment in which applications / modules can run as if they were on a hardware machine (e.g., Fig.15The virtual machine 1448 is executed by the host operating system ( Fig.14 The virtual machine 1448 is hosted by an operating system 1414 in a virtual machine 1448 and typically, though not always, has a virtual machine monitor 1446 that manages the operation of the virtual machine 1448 and the interface with the host operating system (i.e., operating system 1414). A software architecture 1402 such as an operating system 1450, libraries 1452, frameworks / middleware 1454, applications 1456, and / or a presentation layer 1458 executes within the virtual machine 1448. These software architecture layers that execute within the virtual machine 1448 may be the same as the corresponding layers previously described, or may be different.

[0124] Fig.15 is a block diagram illustrating circuitry of a device implementing an algorithm and performing a method according to some exemplary embodiments. Not all components need to be used in various embodiments. For example, a client, a server, and a cloud-based network device may each use a different set of components, or a larger storage device may be used, such as a server.

[0125] An exemplary computing device in the form of a computer 1500 (also referred to as computing device 1500, computer system 1500, or computer 1500) may include a processor 1505, memory storage 1510, removable storage 1515, non-removable storage 1520, input interface 1525, output interface 1530, and communication interface 1535, all connected by a bus 1540. Although the exemplary computing device is illustrated and described as a computer 1500, the computing device may take different forms in different embodiments.

[0126] Memory storage 1510 may include volatile memory 1545 and nonvolatile memory 1550, and may store programs 1555. Computer 1500 may include - or have access to a computing environment that includes - various computer-readable media, such as volatile memory 1545, nonvolatile memory 1550, removable memory 1515, and non-removable memory 1520. Computer memory includes random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

[0127] Computer readable instructions stored on a computer readable medium (e.g., a program 1555 stored in memory 1510) can be executed by the processor 1505 of the computer 1500. Hard disks, CD-ROMs, and RAM are some examples of items that include non-transitory computer readable media such as storage devices. The terms "computer readable medium" and "storage device" do not include carrier waves, because carrier waves are considered too transient. "Computer readable non-transitory medium" includes all types of computer readable media, including magnetic storage media, optical storage media, flash memory media, and solid-state storage media. It should be understood that the software can be installed in a computer and sold with the computer. Alternatively, the software can be obtained and loaded into the computer, including obtaining the software through a physical medium or distribution system, including, for example, obtaining the software from a server owned by the software creator or from a server that the software creator does not own but uses. For example, the software can be stored on a server for distribution over the Internet. As used herein, the terms "computer readable medium" and "machine readable medium" are interchangeable.

[0128] Program 1555 may use a customer preference structure that uses the modules discussed herein, such as AI management module 1560, TAR generation and management module 1565, and data collection and management module 1570. AI management module 1560, TAR generation and management module 1565, and data collection and management module 1570 may be the same as AI management module 1460, TAR generation and management module 1462, and data collection and management module 1464, respectively, as described in conjunction with Fig.14 discussed.

[0129] In an exemplary embodiment, the computer 1500 includes a detector module that detects actuator actions of a plurality of actuators within a home communication network, each of the plurality of actuators being used to change a state of at least one of the network devices, an association module that associates the detected actuator actions with one or more sensor values ​​from a plurality of sensors within the home communication network to generate configuration data, the configuration data including a trigger diagram having one or more trigger conditions and an action diagram corresponding to the trigger diagram, the action diagram indicating one or more actuator actions associated with at least one of the plurality of actuators, and a trigger module that triggers at least one of the plurality of actuators to perform the one or more actuator actions indicated by the action diagram when the one or more trigger conditions are detected. In some embodiments, the computer 1500 may include other or additional modules for performing any one or combination of the steps described in the embodiments. Furthermore, any additional or alternative embodiments or aspects of the method as shown in any of the figures or described in any of the claims are also contemplated to include similar modules.

[0130] Any one or more of the modules described herein may be implemented using hardware (e.g., a processor of a machine, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any suitable combination thereof). In addition, any two or more of these modules may be combined into a single module, and the functionality of a single module described herein may be subdivided between multiple modules. In addition, according to various exemplary embodiments, the modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0131] In some aspects, one or more of the modules 1560 - 1570 may be integrated into a single module, performing the respective functions of the integrated module.

[0132] Although some embodiments have been described in detail above, other modifications are possible. For example, the logic flow shown in the figure does not require the specific order or sequence shown to achieve the desired results. Other steps can be provided or steps can be deleted from the shown flow, and other components can be added to or deleted from the shown system. Other embodiments may be within the scope of the following claims.

[0133] It should also be understood that software including one or more computer executable instructions that facilitate the processes and operations described above with reference to any or all of the steps of the present disclosure can be installed in one or more computing devices consistent with the present disclosure and sold with it. Alternatively, the software can be acquired and loaded into one or more computing devices, including acquiring the software through physical media or distribution systems, including, for example, acquiring the software from a server owned by the creator of the software or from a server that the creator of the software does not own but uses. For example, the software can be stored on a server for distribution over the Internet.

[0134] In addition, it will be understood by those skilled in the art that the present disclosure is not limited in its application to the details of the structure and arrangement of the components described in the description or shown in the drawings. The embodiments herein are capable of other embodiments and can be practiced or executed in various ways. In addition, it is understood that the words and terms used herein are for descriptive purposes and should not be considered as limiting. The use of "including", "comprising" or "having" and their variations herein is intended to include the items listed thereafter and their equivalents as well as additional items. Unless otherwise limited, the terms "connect", "couple", "install" and their variations are widely used herein and include direct and indirect connections, couplings and installations. In addition, the terms "connect" and "couple" and their variations are not limited to physical or mechanical connections or couplings. In addition, terms such as upper, lower, bottom and top are relative and are used to help illustrate, but are not limiting.

[0135] The components of the illustrative apparatus, systems, and methods employed in accordance with the illustrated embodiments may be implemented at least in part in digital electronic circuitry, analog electronic circuitry, or computer hardware, firmware, software, or a combination thereof. These components may be implemented, for example, as a computer program product, such as a computer program, program code, or computer instructions tangibly embodied in an information carrier, or embodied in a machine-readable storage device, for execution or control of the operation thereof by a data processing apparatus such as a programmable processor, a computer, or multiple computers.

[0136] The computer program can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. The computer program can be deployed on a single computer or multiple computers at a site for execution, or it can be distributed on multiple sites and interconnected through a communication network. In addition, the functional programs, codes, and code segments used to implement the technology described herein can be easily understood by programmers in the art as being within the scope of the claims, and the technology described herein belongs to the art. The method steps associated with the illustrative embodiments can be performed by one or more programmable processors that execute computer programs, codes, or instructions to perform functions (e.g., by operating input data and / or generating output). The method steps can also be performed by a dedicated logic circuit, and the device for performing the method can be implemented as a dedicated logic circuit, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0137] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented using a general purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general purpose processor may be a microprocessor, and optionally, the general purpose processor may also be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration.

[0138] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The required elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, the computer will also include, or be operably coupled to receive data from or transfer data to or receive data from one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, such as semiconductor memory devices, such as electrically programmable read-only memory or ROM (electrically programmable read-only memory, EPROM), electrically erasable programmable ROM (electrically erasable programmable ROM, EEPROM), flash memory devices, and data storage disks (such as magnetic disks, internal hard disks or removable disks, magneto-optical disks, CD-ROMs, and DVD-ROMs). The processor and memory can be supplemented by or combined in a dedicated logic circuit.

[0139] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and methods. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described above may be represented by voltage, current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0140] As used herein, "machine-readable medium" (or "computer-readable medium") refers to a device capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of memory (e.g., Erasable Programmable Read-Only Memory (EEPROM)), and / or any appropriate combination thereof. The term "machine-readable medium" should be deemed to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store processor instructions. The term "machine-readable medium" should also be deemed to include any medium or combination of multiple media that can store instructions executed by one or more processors 1505, so that when the instructions are executed by one or more processors 1505, the one or more processors 1505 perform any one or more of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network that includes multiple storage devices or devices. The term "machine-readable medium" as used herein does not include the signal itself.

[0141] In addition, without departing from the scope of the present invention, the techniques, systems, subsystems and methods described and illustrated as discrete or separate in the various embodiments may be combined or integrated with other systems, modules, techniques or methods. Other items shown or discussed as coupled or directly coupled or communicating with each other may also be coupled or communicated indirectly via an interface, device or intermediate component, either electrically, mechanically or otherwise. Other variations, substitutions and altered examples may be determined by those skilled in the art without departing from the spirit and scope of the disclosure herein.

[0142] Although the present invention has been described with reference to specific features and embodiments of the present invention, it is apparent that various modifications and combinations of the present invention may be made without departing from the present invention. For example, other components may be added to the described system, or removed from the described system. The specification and drawings are to be considered merely as an illustration of the present invention as defined by the appended claims and any and all modifications, variations, combinations or equivalents falling within the scope of the present invention are contemplated. Other aspects may be within the scope of the following claims.

Claims

1. A computer-implemented method for automating the operation of network devices within a home, characterized in that include: at least one processor of a computing device decoding first information received from a plurality of actuators within the home, the first information indicating a plurality of actuator actions performed by the plurality of actuators within the home based on user interactions, each of the plurality of actuators being used to change a state of at least one of the network devices; the at least one processor of the computing device decoding second information received from a plurality of sensors within the home, the second information indicating one or more sensor values ​​generated by the plurality of sensors; The at least one processor obtains a machine learning (ML) model from a storage device of the computing device, wherein the ML model is trained by a training data set of the household to associate the first information with the second information; the at least one processor generating an association of the plurality of actuator actions and the one or more sensor values ​​based on application of the ML model to the first information and the second information; The at least one processor generates configuration data based on the association, the configuration data comprising: a trigger diagram having one or more trigger conditions specifying at least one of the plurality of actuator actions; and an action diagram corresponding to the trigger diagram, the action diagram indicating one or more automated actuator actions associated with at least one of the plurality of actuators; The at least one processor detects a trigger condition from among the one or more trigger conditions specified by the trigger diagram, and The at least one processor causes at least one of the plurality of actuators to perform the one or more automation actuator actions indicated by the action diagram based on detecting the one or more automation actuator actions performed without the user interaction.

2. The computer-implemented method of claim 1, wherein: The one or more trigger conditions include one or more of the following: the one or more sensor values; an actuator action among the plurality of actuator actions; External data is received from information sources external to the home.

3. The computer-implemented method of claim 2, wherein: The one or more trigger conditions within the trigger diagram are connected via one or more logical connectors.

4. The computer-implemented method of claim 3, wherein: Also includes: Based on the detected trigger condition of the one or more trigger conditions and further based on the one or more logical connectors, at least one of the plurality of actuators is caused to perform the one or more automated actuator actions.

5. The computer-implemented method according to any one of claims 1 to 3, characterized in that: The action diagram includes: A plurality of nodes are coupled via edges, wherein each of the plurality of nodes corresponds to an automated actuator action of the one or more automated actuator actions associated with at least one of the plurality of actuators.

6. The computer-implemented method of claim 5, wherein: Each of the edges coupling at least two nodes of the plurality of nodes is associated with a time delay between the one or more automated actuator actions corresponding to the at least two nodes.

7. The computer-implemented method according to claim 1 or 2, characterized in that: Also includes: Retrieving the training data set from the storage device of the computing device, the training data set comprising a plurality of predetermined trigger conditions and a plurality of predetermined actuator actions, the plurality of predetermined actuator actions corresponding to the plurality of predetermined trigger conditions; Prior to generating the association, the ML model is trained, the training using at least the training dataset.

8. The computer-implemented method of claim 7, wherein: Also includes: retrieving user-defined goals for said automated operation of said network devices within said home; retrieving a predefined sensor-actuator relationship that groups a subset of the plurality of actuators with a subset of the plurality of sensors based on common sensor and actuator locations; The configuration data is further generated based on the user-defined goal and the predefined sensor-actuator relationship.

9. The computer-implemented method according to any one of claims 1 to 3, characterized in that: The configuration data comprises an automation rules table comprising a plurality of trigger diagrams and a corresponding plurality of action diagrams, wherein the plurality of trigger diagrams comprises the trigger diagram, and wherein the plurality of action diagrams comprises the action diagram.

10. The computer-implemented method of claim 9, wherein: Also includes: retrieving a second automation rules table for a second home; Based on the second automation rule table, the plurality of trigger diagrams and the corresponding plurality of action diagrams in the configuration data are modified, or the plurality of trigger diagrams or the corresponding plurality of action diagrams in the configuration data are modified.

11. A system for home automation operation, characterized in that: include: a plurality of actuators within a home, each of the plurality of actuators being used to change a state of at least one of the network devices within the home; a plurality of sensors within the home, the plurality of sensors configured to generate sensor data including one or more sensor values; a memory for storing instructions; as well as at least one processor in communication with the memory, the plurality of actuators, and the plurality of sensors, wherein the at least one processor is configured to perform the following steps when executing the instructions: decoding first information received from the plurality of actuators, the first information indicating a plurality of actuator actions performed by the plurality of actuators within the home based on user interactions; decoding second information received from the plurality of sensors, the second information indicating the one or more sensor values; Obtaining a machine learning (ML) model from the memory, wherein the ML model is trained by a training data set of the family to associate the first information with the second information; generating an association of the plurality of actuator actions and the one or more sensor values ​​based on application of the ML model to the first information and the second information; Generate configuration data based on the association, the configuration data comprising: a trigger diagram having one or more trigger conditions specifying at least one of the plurality of actuator actions; and an action diagram corresponding to the trigger diagram, the action diagram indicating one or more automated actuator actions associated with at least one of the plurality of actuators; detecting a trigger condition from among the one or more trigger conditions specified by the trigger diagram, and At least one of the plurality of actuators is caused to perform the one or more automated actuator actions indicated by the action diagram based on the detected one or more automated actuator actions performed without the user interaction.

12. The system according to claim 11, characterized in that The trigger diagram includes the one or more trigger conditions connected by one or more logical connectors, and the one or more trigger conditions include one or more of the following: the one or more sensor values; an actuator action of the plurality of actuator actions; or External data is received from information sources external to the home.

13. The system according to claim 12, characterized in that The steps also include: Based on the detected trigger condition of the one or more trigger conditions and further based on the one or more logical connectors, at least one of the plurality of actuators is caused to perform the one or more automated actuator actions.

14. The system according to claim 11 or 12, characterized in that: The action graph includes a plurality of nodes coupled by edges, wherein each of the plurality of nodes corresponds to an automated actuator action of the one or more actuator actions associated with at least one of the plurality of actuators; Each of the edges coupling at least two nodes of the plurality of nodes is associated with a time delay between the one or more automated actuator actions corresponding to the at least two nodes.

15. The system according to claim 11, characterized in that The steps also include: Retrieving the training data set from the memory, the training data set comprising a plurality of predetermined trigger conditions and a plurality of predetermined actuator actions, the plurality of predetermined actuator actions corresponding to the plurality of predetermined trigger conditions; Prior to generating the association, the ML model is trained, the training using at least the training dataset.

16. The system according to claim 15, characterized in that The steps also include: retrieving user-defined goals for said automated operations within said home; retrieving a predefined sensor-actuator relationship that groups a subset of the plurality of actuators with a subset of the plurality of sensors based on common sensor and actuator locations; and The configuration data is further generated based on the user-defined goal and the predefined sensor-actuator relationship.

17. The system according to claim 11, characterized in that The configuration data includes an automation rules table having a plurality of trigger diagrams and a corresponding plurality of action diagrams, wherein the plurality of trigger diagrams includes the trigger diagram, and wherein the plurality of action diagrams includes the action diagram.

18. The system according to claim 17, characterized in that The steps also include: retrieving a second automation rules table for a second home; Based on the second automation rule table, the trigger diagram and the action diagram in the configuration data are modified, or the trigger diagram or the action diagram in the configuration data is modified.

19. A non-transitory computer-readable medium storing instructions for automating operations of a home network device, characterized in that: The at least one processor is configured to perform the following steps when executing the instructions: decoding first information received from a plurality of actuators within the home, the first information indicating a plurality of actuator actions performed by the plurality of actuators within the home based on user interactions, each of the plurality of actuators being used to change a state of at least one of the network devices; decoding second information received from a plurality of sensors within the home, the second information indicating one or more sensor values ​​generated by the plurality of sensors; Obtaining a machine learning (ML) model from a storage device of the computing device, wherein the ML model is trained by a training data set of the household to associate the first information with the second information; generating an association of the plurality of actuator actions and the one or more sensor values ​​based on application of the ML model to the first information and the second information; Generate configuration data based on the association, the configuration data comprising: a trigger diagram having one or more trigger conditions specifying at least one of the plurality of actuator actions; and an action diagram corresponding to the trigger diagram, the action diagram indicating one or more automated actuator actions associated with at least one of the plurality of actuators; detecting a trigger condition from among the one or more trigger conditions specified by the trigger diagram, and At least one of the plurality of actuators is caused to perform the one or more automated actuator actions indicated by the action diagram based on the detected one or more automated actuator actions performed without the user interaction.

20. The non-transitory computer readable medium of claim 19, wherein: When executed, the instructions further cause the at least one processor to: Retrieving the training data set from the storage device of the computing device, the training data set comprising a plurality of predetermined trigger conditions and a plurality of predetermined actuator actions, the plurality of predetermined actuator actions corresponding to the plurality of predetermined trigger conditions; Prior to generating the association, the ML model is trained, the training using at least the training data.

Images