Generating automations associated with controlling smart-home devices

Abstract

An automation engine generates an automation in response to a natural language request. The automation engine may generate automation scripts, automation instances, automation templates, automation code snippet(s), and / or automation summaries. As an example, a user may request an automation script to be generated to lock the doors and turn off the lights when the user leaves home. In response, the automation engine uses one or more machine learning models to automatically generate the script. Automations can be used to control a variety of smart devices, such as lights, audio devices, video devices, thermostats, locks, security systems, and the like. The automation engine outputs an automation script using a programming language, such as YAML Ain't Markup Language (YAML), Python, JavaScript, or some other programming language.

Description

GENERATING AUTOMATIONS ASSOCIATED WITH CONTROLLING SMART-HOME DEVICESTECHNICAL FIELD

[0001] This patent specification relates generally to programmatic generation of automations, such as automation scripts, associated with controlling smart-home devices. More specifically, this disclosure describes using natural language and machine learning techniques to automatically generate automations.BACKGROUND

[0002] Smart-home devices are rapidly becoming part of the modem home experience. These devices may include thermostats, keypads, touch screens, and / or other control devices for controlling environmental systems, such as HVAC systems or lighting systems. The smart-home environment may also include smart appliances, such as washing machines, dishwashers, refrigerators, garbage cans, and so forth, that interface with control and / or monitoring devices to increase the level of functionality and control provided to an occupant. Security systems, including cameras, keypads, sensors, motion detectors, glass-break sensors, microphones, and so forth, may also be installed as part of the smart-home architecture. Other smart-home devices may include doorbells, monitoring systems, hazard detectors, smart lightbulbs, and virtually any other electronic device that can be controlled via a wired / wireless network.

[0003] To control these smart-home devices, a user may create a home automation script that instructs smart-home devices to perform actions. For example, a home automation script may be created that adjusts the temperature of a thermostat, turns on lights within a house, and possibly perform other actions when a user arrives home. Learning how to create these scripts, and how to instruct each of the different smart-home devices to perform different actions, however, can be challenging.BRIEF SUMMARY

[0004] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardw are, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by one or processors, cause the system to perform the actions. One general aspect includes a method. In some embodiments, a method may include receiving, via one or more processors, a request to generate an automation, wherein the requestdescribes an automation that is associated with controlling an operation of one or more smart devices; and generating, via the one or more processors, the automation using a machine learning model trained to perform generate automations associated with controlling operations of at least one of the one or more smart devices, wherein the automation is one or more of an automation script, an automation instance, an automation template, an automation code snippet, or an automation summary. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0005] Implementations may include one or more of the following features. In some embodiments, the method may include determining, via the one or more processors, that the request is associated with a developer; and wherein generating the automation comprises generating at least one of the automation template, or the automation code snippet. In some embodiments, the method may include determining, via the one or more processors, that the request is associated with a user; and wherein generating the automation comprises generating at least one of the automation script, or the automation instance. In some embodiments, generating the at least one of the automation script comprises generating code in a human-readable data- serialization language. In some embodiments, the method may include determining, via the one or more processors, that the request is a request to generate an automation summary'; and wherein generating the automation comprises generating the automation summary. In some embodiments, the method may include determining, via the one or processors, one or more devices within an environment associated with the request; and wherein generating the automation is based, at least in part, on the one or more devices within the environment. In some embodiments, receiving the request to generate the automation comprises receiving a natural language description of the automation. In some embodiments, the method may include receiving feedback data indicating an accuracy of the automation; and updating the machine learning model based on the feedback data. In some embodiments, the method may include providing, via the one or more processors, the automation to a computing device; causing at least a portion of the automation to be presented on a display associated with the computing device; and receiving, from the computing device, one or more edits to the automation. In some embodiments, the method may include causing the machine learning model to be updated based at least in part on the one or more edits. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0006] In some embodiments, a system includes one or more processors. The system also includes one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving arequest to generate an automation, wherein the request describes an automation that is associated with controlling an operation of one or more smart devices; and generating the automation using a machine learning model trained to perform generate automations associated with controlling operations of at least one of the one or more smart devices wherein the automation is one or more of an automation script, an automation instance, an automation template, an automation code snippet, or an automation summary.. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0007] Implementations may include one or more of the following features. In some embodiments, the system the operations may include determining that the request is associated with a developer; and wherein generating the automation comprises generating at least one of the automation template, or the automation code snippet. In some embodiments, the operations may include determining that the request is associated with a user; and wherein generating the automation comprises generating at least one of the automation script, or the automation instance. The operations may also include receiving feedback data indicating an accuracy of the automation; and updating the machine learning model based on the feedback data. In some embodiments, the operations include causing at least a portion of the automation to be presented on a display associated with the computing device; receiving, from the computing device, one or more edits to the automation; and causing the machine learning model to be updated based at least in part on the one or more edits. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0008] In some embodiments, a non-transitory computer-readable medium may include instructions that, when executed by one or more processors, cause the one or more processors to perform operations including receiving a request to generate an automation, wherein the request describes an automation that is associated with controlling an operation of one or more smart devices; generating the automation using a machine learning model trained to perform generate automations associated with controlling operations of at least one of the one or more smart devices, wherein the automation is one or more of an automation script, an automation instance, an automation template, an automation code snippet, or an automation summary. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0009] Implementations may include one or more of the following features. In some embodiments, the non-transitory computer-readable medium may include additional instructions that, when executed by the one or more processors, cause the one or more processors to perform additional operations that may include: determining, via the one or more processors, that the request is associated with a developer; and wherein generating the automation comprises generating at least one of the automation template, or the automation code snippet.

[0010] In some embodiments, the non-transitory computer-readable medium may include additional instructions that, when executed by the one or more processors, cause the one or more processors to perform additional operations that include: determining, via the one or more processors, that the request is associated with a user; and wherein generating the automation comprises generating at least one of the automation script, or the automation instance. In some embodiments, the non-transitory computer-readable medium may include additional instructions that, when executed by the one or more processors, cause the one or more processors to perform additional operations that include: receiving feedback data indicating an accuracy of the automation; and updating the machine learning model based on the feedback data. In some embodiments, the non-transitory computer-readable medium may include additional instructions that, when executed by the one or more processors, cause the one or more processors to perform additional operations that include: providing the automation to a computing device; causing at least a portion of the automation to be presented on a display associated with the computing device; receiving, from the computing device, one or more edits to the automation; and causing the machine learning model to be updated based at least in part on the one or more edits. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0011] A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings. Also note that other embodiments may be described in the following disclosure and claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG. 1 is an example of a smart-home environment within which one or more of the devices, methods, systems, services, and / or computer program products described further herein will be applicable, according to some embodiments.

[0013] FIG. 2 illustrates a simplified block diagram of a representative network architecture that includes a smart-home network in accordance, according to some embodiments.

[0014] FIG 3 illustrates a simplified operating environment in which a server system interacts with client devices and smart devices and provides processing for generating automations, according to some embodiments.

[0015] FIG. 4 illustrates a simplified block diagram of a representative smart device, according to some embodiments.

[0016] FIG. 5 illustrates an example of training data, according to some embodiments.

[0017] FIG. 6 illustrates example automation outputs of a machine learning model trained to generate automations, according to some embodiments.

[0018] FIG. 7 illustrates example inputs, and an output example automation summary of a machine learning model trained to generate automations, according to some embodiments.

[0019] FIG. 8 illustrates a flowchart of a method for automatically generating automations using a trained machine learning model, according to some embodiments.

[0020] FIG. 9 illustrates a flowchart of a method for determining a type of user associated with a request, according to some embodiments.

[0021] FIG. 10 illustrates a flow chart of a method for performing pre-processing, according to some embodiments.

[0022] FIG. 11 illustrates a flow chart of a method for receiving feedback data and updating a trained machine learning model, according to some embodiments.

[0023] FIG. 12A illustrates a graphical user interface that includes user interface elements to edit an automation and provide feedback, according to some embodiments.

[0024] FIG. 12B illustrates example graphical user interfaces that includes user interface elements for discovering automations, according to some embodiments.

[0025] FIG. 12C illustrates example graphical user interfaces that includes user interface elements for discovering automations by performing a search, according to some embodiments.

[0026] FIG. 12D illustrates example graphical user interfaces that includes user interface elements for configuring an automation, according to some embodiments.

[0027] FIG. 12E illustrates an example graphical user interface that includes user interface elements for configuring an automation, according to some embodiments.DETAILED DESCRIPTION

[0028] In the following detailed description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the various embodiments of the present invention. Those of ordinary skill in the art will realize that these various embodiments of the present invention are illustrative only and are not intended to be limiting in any way. Other embodiments of the present invention will readily suggest themselves to such skilled persons having the benefit of this disclosure. It will be apparent to one skilled in the art that the present invention may be practiced without some or all of these specific details. In other instances, well known details have not been described in detail in order not to unnecessarily obscure the present invention.

[0029] In addition, for clarity purposes, not all of the routine features of the embodiments described herein are shown or described. One of ordinary skill in the art would readily appreciate that in the development of any such actual embodiment, numerous embodiment-specific decisions may be required to achieve specific design objectives. These design objectives will vary from one embodiment to another and from one developer to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming but would nevertheless be a routine engineering undertaking for those of ordinary skill in the art having the benefit of this disclosure.

[0030] FIG. 1 illustrates an example smart-home environment 100, according to some embodiments. The smart-home environment 100 includes a structure 150 (e.g., ahouse, office building, garage, or mobile home) with various integrated devices. It will be appreciated that devices may also be integrated into a smart-home environment 100 that does not include an entire structure 150, such as an apartment, condominium, or office space. Further, the smart-home environment 100 may control and / or be coupled to devices outside of the actual structure 150. Indeed, several devices in the smart-home environment 100 need not be physically within the structure 150. For example, a device controlling a pool heater 1 14 or irrigation system 116 may be located outside of the structure 150.

[0031] The term “smart-home environment” may refer to smart environments for homes such as a single-family house, but the scope of the present teachings is not so limited. The present teachings are also applicable, without limitation, to duplexes, townhomes, multi-unit apartment buildings, hotels, retail stores, office buildings, industnal buildings, and more generally any living space or workspace. Similarly, while the terms user, customer, installer, homeowner, occupant, guest, tenant, landlord, repair person, etc., may be used to refer to a person or persons acting in thecontext of some situations described herein, these references do not limit the scope of the present teachings with respect to the person or persons who are performing such actions. Thus, for example, the terms user, customer, purchaser, installer, subscriber, and homeowner may often refer to the same person in the case of a single-family residential dwelling, because the head of the household is often the person who makes the purchasing decision, buys the unit, and installs and configures the unit, as well as being one of the users of the unit. However, in other scenarios, such as a landlord-tenant environment, the customer may be the landlord with respect to purchasing the unit, the installer may be a local apartment supervisor, a first user may be the tenant, and a second user may again be the landlord with respect to remote control functionality. While the identity of the person performing the action may be germane to a particular advantage provided by one or more of the implementations, such an identity should not be construed in the descriptions that follow as necessarily limiting the scope of the present teachings to those individuals having those identities.

[0032] The depicted structure 150 includes a plurality of rooms 152, separated at least partly from each other via walls 154. The walls 154 may include interior walls or exterior walls. Each room may further include a floor 156 and a ceiling 158. Devices may be mounted on, integrated with and / or supported by a wall 154, floor 156, or ceiling 158.

[0033] In some implementations, the integrated devices of the smart-home environment 100 include intelligent, multi-sensing, network-connected devices that integrate seamlessly with each other in a smart-home network and / or with a central server or a cloud-computing system to provide a variety of useful smart-home functions. Any network-connected or network-connectable device having processing capabilities, and optionally also sensing capabilities, and that is deployed or is suitable for being deployed in a smart-home environment, such as smart-home environment 100, so that the device may be controlled via a wired and / or wireless network and may optionally interact with one or more other such devices may be referred to as a ‘"smart-home device” or “smart device”. The smart-home environment 100 may include one or more intelligent, multisensing, network-connected thermostats 102 (hereinafter referred to as ’‘smart thermostats 102”), one or more intelligent, network-connected, multi-sensing hazard detection units 104 (hereinafter referred to as “smart hazard detectors 104”), one or more intelligent, multi -sensing, network- connected entry way interface devices 106 and 120 (hereinafter referred to as “smart doorbells 106” and “smart door locks 120”), one or more intelligent, multi-sensing, network-connected alarm systems 122 (hereinafter referred to as “smart alarm systems 122”), and one or more other intelligent, network-connected devices. The smart-home environment 100 may also include other smart-home devices / controls, such as but not limited to monitoring systems (e.g., baby monitoringsystems, elderly monitoring systems, handicapped monitoring systems, ... ), home entertainment controls, energy conservation devices / controls, home control devices / controls, remote home management and monitoring devices / controls, safety.

[0034] In some implementations, the one or more smart thermostats 102 detect ambient climate characteristics (e.g., temperature and / or humidity) and control a HVAC system 103 accordingly. For example, a respective smart thermostat 102 includes an ambient temperature sensor.

[0035] The one or more smart hazard detectors 104 may include thermal radiation sensors directed at respective heat sources (e.g., a stove, oven, other appliances, a fireplace, etc.). For example, a smart hazard detector 104 in a kitchen 153 may include a thermal radiation sensor directed at a stove / oven 112. A thermal radiation sensor may determine the temperature of the respective heat source (or a portion thereol) at which it is directed and may provide corresponding blackbody radiation data as output.

[0036] The smart doorbell 106 and / or the smart door lock 120 may detect a person's approach to or departure from a location (e.g., an outer door), control doorbell / door locking functionality (e g., receive user inputs from a portable electronic device 166-1 to actuate bolt of the smart door lock 120), announce a person’s approach or departure via audio or visual devices, and / or control settings on a security system (e.g.. to activate or deactivate the security system when occupants go and come). In some implementations, the smart doorbell 106 may include some or all of the components and features of the camera 118. In some implementations, the smart doorbell 106 includes a camera 118.

[0037] The smart alarm system 122 may detect the presence of an individual within close proximity’ (e.g., using built-in IR sensors), sound an alarm (e.g.. through a built-in speaker, or by sending commands to one or more external speakers), and send notifications to entities or users within / outside of the smart-home network 100. In some implementations, the smart alarm system 122 also includes one or more input devices or sensors (e.g., keypad, biometric scanner, NFC transceiver, microphone) for verilying the identity of a user, and one or more output devices (e.g., display, speaker) for providing notifications. In some implementations, the smart alarm system 122 may also be set to an “armed” mode, such that detection of a trigger condition or event causes the alarm to be sounded unless a disarming action is performed.

[0038] In some implementations, the smart-home environment 100 may include one or more intelligent, multi-sensing, network-connected wall switches 108 (hereinafter referred to as “smart wall switches 108”). along with one or more intelligent, multi-sensing, network-connected wall plug interfaces 1 10 (hereinafter referred to as “smart wall plugs 1 10”). The smart wall switches108 may detect ambient lighting conditions, detect room-occupancy states, and control a power and / or dim state of one or more lights. In some instances, smart wall switches 108 may also control a power state or speed of a fan, such as a ceiling fan. The smart wall plugs 110 may detect occupancy of a room or enclosure and control supply of power to one or more wall plugs (e.g., such that power is not supplied to the plug if nobody is at home).

[0039] In some implementations, the smart-home environment 100 of FIG. 1 may include a plurality of intelligent, multi-sensing, network-connected appliances 112 (hereinafter referred to as ■‘smart appliances 112”), such as refrigerators, stoves, ovens, televisions, washers, dryers, lights, stereos, intercom systems, garage-door openers, floor fans, ceiling fans, wall air conditioners, pool heaters, irrigation systems, security systems, space heaters, window AC units, motorized duct vents, and so forth. In some implementations, when plugged in. an appliance may announce itself to the smart home network, such as by indicating what type of appliance it is, and it may automatically integrate with the controls of the smart home.

[0040] Such communication by the appliance to the smart home may be facilitated by either a wired or wireless communication protocol. The smart home may also include a variety of noncommunicating legacy appliances 140, such as older-model conventional washers / dryers, refrigerators, and / or the like, which may be controlled by smart wall plugs 110. The smart-home environment 100 may further include a variety of partially communicating legacy appliances 142, such as infrared (“IR”) controlled wall air conditioners or other IR-controlled devices, which may be controlled by IR signals provided by the smart hazard detectors 104, hand-held remote controls, key FOBs, or the smart wall switches 108.

[0041] In some implementations, the smart-home environment 100 may include one or more network-connected cameras 118 that are configured to provide video monitoring and security in the smart-home environment 100. The cameras 118 may be used to determine the occupancy of the structure 150 and / or particular rooms 152 in the structure 150, and thus may act as occupancy sensors. For example, video captured by the cameras 118 may be processed to identify the presence of an occupant in the structure 150 (e.g., in a particular room 152). Specific individuals may be identified based, for example, on their appearance (e.g., height, face) and / or movement (e.g., their walk / gait). Cameras 118 may additionally include one or more sensors (e.g., IR sensors, motion detectors), input devices (e.g., microphone for capturing audio), and output devices (e.g., speaker for outputting audio). In some implementations, the cameras 118 may each be configured to operate in a day mode and in a low-light mode (e.g., a night mode). In some implementations, the cameras 118 each include one or more IR illuminators for providing illumination while thecamera is operating in the low-light mode. In some implementations, the cameras 118 include one or more outdoor cameras. In some implementations, the outdoor cameras include additional features and / or components such as weatherproofing and / or solar ray compensation.

[0042] The smart-home environment 100 may additionally or alternatively include one or more other occupancy sensors (e.g., the smart doorbell 106, smart door locks 120, touch screens, IR sensors, microphones, ambient light sensors, motion detectors, smart nightlights 180, etc.). In some implementations, the smart-home environment 100 may include radio-frequency identification (RFID) readers (e.g., in each room 152 or a portion thereof) that determine occupancy based on RFID tags located on or embedded in occupants. For example, RFID readers may be integrated into the smart hazard detectors 104, and RFID tags may attached to clothing, and / or integrated in hand-held devices such as a smart phone.

[0043] The smart-home environment 100 may also include communication with devices outside of the physical home but within a proximate geographical range of the home. For example, the smart-home environment 100 may include a pool heater monitor 114 that communicates a current pool temperature to other devices within the smart-home environment 100 and / or receives commands for controlling the pool temperature. Similarly, the smart-home environment 100 may include an irrigation monitor 116 that communicates information regarding irrigation systems within the smart-home environment 100 and / or receives control information for controlling such irrigation systems.

[0044] By virtue of network connectivity, one or more of the smart-home devices of FIG. 1 may further allow a user to interact with the device even if the user is not proximate to the device. For example, a user may communicate with a device using a computer (e.g., a desktop computer, laptop computer, or tablet) or some other portable electronic device 166 (e.g., a mobile phone, such as a smart phone). A webpage or application may be configured to receive communications from the user and control the device based on the communications and / or to present information about the device’s operation to the user. For example, the user may view a cunent set point temperature for a device (e.g., a stove) and adjust it using a computer. The user may be in the structure during this remote communication or outside the structure.

[0045] As discussed above, users may control smart devices in the smart-home environment 100 using a network-connected computer or portable electronic device 166. In some examples, some or all of the occupants (e.g., individuals who live in the home) may register their device 166 with the smart-home environment 100. Such registration may be made at a central server to authenticate the occupant and / or the device as being associated with the home and to give permission to theoccupant to use the device to control the smart devices in the home. An occupant may use their registered device 166 to remotely control the smart devices of the home, such as when the occupant is at work or on vacation. The occupant may also use their registered device to control the smart devices when the occupant is actually located inside the home, such as when the occupant is sitting on a couch inside the home. It should be appreciated that instead of or in addition to registering devices 166, the smart-home environment 100 may make inferences about (1) which individuals live in the home and are therefore occupants, and (2) which devices 166 are associated with those individuals. As such, the smart-home environment may “learn” who is an occupant and permit the devices 166 associated with those individuals to control the smart devices of the home.

[0046] In some implementations, in addition to containing processing and sensing capabilities, devices 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, and / or 122 (collectively referred to as “the smart devices” or “the smart-home devices”) are capable of data communications and information sharing with other smart devices, a central server or cloud-computing system, and / or other devices that are network-connected. Data communications may be carried out using any of a variety of custom or standard wireless protocols (e.g.. IEEE 802. 15.4, Wi-Fi. ZigBee, 6L0WPAN. Thread, Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, etc.) and / or any of a variety of custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0047] In some implementations, the smart devices may serve as wireless or wired repeaters. In some implementations, a first one of the smart devices communicates with a second one of the smart devices via a wireless router. The smart devices may further communicate with each other via a connection (e.g., network interface 160) to a network, such as the Internet 162. Through the Internet 162, the smart devices may communicate with a server system 164 (also called a central server system and / or a cloud-computing system herein). The server system 164 may be associated with a manufacturer, support entity, or service provider associated with the smart device(s). In some implementations, a user is able to contact customer support using a smart device itself rather than needing to use other communication means, such as a telephone or Internet-connected computer. In some implementations, software updates are automatically sent from the server system 164 to smart devices (e.g., when available, when purchased, or at routine intervals).

[0048] In some implementations, the network interface 160 includes a conventional network device (e.g., a router), and the smart-home environment 100 of FIG. 1 includes a hub device 180that is communicatively coupled to the network(s) 162 directly or via the network interface 160. The hub device 180 may be further communicatively coupled to one or more of the above intelligent, multi-sensing, network-connected devices (e.g., smart devices of the smart-home environment 100). Each of these smart devices optionally communicates with the hub device 180 using one or more radio communication networks available at least in the smart-home environment 100 (e.g.. ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi and other radio communication networks). In some implementations, the hub device 180 and devices coupled with / to the hub device can be controlled and / or interacted with via an application running on a smart phone, household controller, laptop, tablet computer, game console or similar electronic device. In some implementations, a user of such controller application can view status of the hub device or coupled smart devices, configure the hub device to interoperate with smart devices newly introduced to the home network, commission new smart devices, and adjust or view settings of connected smart devices, etc. In some implementations the hub device extends the capabilities of low-capability smart devices to match the capabilities of the highly capable smart devices of the same type, integrates functionality of multiple different device types - even across different communication protocols, and is configured to streamline adding of new devices and commissioning of the hub device. In some implementations, hub device 180 further comprises a local storage device for storing data related to, or output by, smart devices of smart-home environment 100. In some implementations, the data includes one or more of: video data output by a camera device, metadata output by a smart device, settings information for a smart device, usage logs for a smart device, and the like.

[0049] In some implementations, smart-home environment 100 includes a local storage device 190 for storing data related to, or output by, smart devices of smart-home environment 100. In some implementations, the data includes one or more of: video data output by a camera device (e.g.. camera 118). metadata output by a smart device, settings information for a smart device, usage logs for a smart device, and the like. In some implementations, local storage device 190 is communicatively coupled to one or more smart devices via a smart home network. In some implementations, local storage device 190 is selectively coupled to one or more smart devices via a wired and / or wireless communication network. In some implementations, local storage device 190 is used to store video data when external network conditions are poor. For example, local storage device 190 is used when an encoding bitrate of camera 118 exceeds the available bandwidth of the external network (e.g., network(s) 162). In some implementations, local storage device 190 temporarily stores video data from one or more cameras (e.g.. camera 118) prior to transferring the video data to a server system (e.g., server system 164). In some implementations,the smart-home environment 100 includes sendee robots 182 that are configured to carry out, in an autonomous manner, any of a variety of household tasks.

[0050] According to some examples, all / some of the smart devices can be controlled using an automation 174 generated by an automation engine 168 using one or more machine learning models 170. In some implementations, the automation engine 168 generates an automation script 174A (which may be referred to herein as a “script’'). Generally, an automation script is a set of instructions that when executed (e.g., by one or more smart devices), or some other device / component), automate tasks in a smart home. As an example, a senpt may be used to lock the doors and turn off the lights when you leave home, turn on lights when its dark outside and you are home, dim the lights when you watch a movie, and the like. Scripts can be used to control a variety of smart devices, such as lights, audio devices, video devices, thermostats, locks, security’ systems, and the like. In another example, an automation script may include instructions to automate other tasks, such as but not limited to providing notifications, generating / s ending email messages, text messages, and the like. In some configurations, the automation engine 168 outputs an automation script 174A using a programming language, such as YAML Ain’t Markup Language (YAML), Python. JavaScript, or some other programming language.

[0051] Once a script is created, it can be triggered for execution by a variety of events, such as the time of day, the opening of a door, or the arrival of a person, and the like. As briefly discussed above, creating automations using a scripting language can be a complex and high-friction process. For example, a user may not be familiar with the scripting language used to control smart devices, the user may not know the specifics of how to perform the actions, the user may not know what actions the smart devices can perform, and the like. Even if a user is familiar with a particular scripting language, the user may not be familiar with a different version of the scripting language.

[0052] Using techniques described herein, a user may provide a natural language prompt 172 to describe an automation 174 that they would like to be generated as an automation 174. The term “automation” as used herein refers to the output of the automation engine 168. In some examples, the output of the automation engine 168 includes an automation script 174A, an automation instance 174B, an automation template 174C, an automation code snippet(s) 174D, and / or an automation summary 174E (see FIG. 3).

[0053] In response to receiving the natural language prompt 172, an artificial intelligence (Al) mechanism, such as automation engine 168. generates one or more automations 174. such as an automation script 174A, an automation instance 174B, an automation template 174C, and / or an automation code snippet(s) 174D that when executed perform the requested smart homeautomation. Instead of a user having to manually create the script for the automation 174, the automation engine 168 may use one or more machine learning models 170 to programmatically generate the script. In some examples, the automation engine 168 can generate templates and / or code snippets (e.g., for developers), and / or instances and / or code snippets (e.g., for a user to control devices in their home environment).

[0054] According to some examples, the automation engine 168 determines whether to supply an instance and / or a template based on a type of user interacting with the automation engine 168. For instance, in some examples, when a developer requests an automation 174 that turns on the lights at sunset, the automation engine 168 using the machine learning model 170 returns an automation template 174C, and when a user (e.g., a consumer) requests an automation 174 that turns off the lights at bedtime, the automation engine 168 returns an automation script 174A and / or an automation instance 174B that can be executed (e.g., without modification) within the environment of the user or associated with the user. An automation template 174C includes instructions to control an operation of one or more smart devices and interact with smart-home devices but is not specific to a particular environment. For instance, automation template 174C may not be adapted to one or more smart devices that are present in the particular environment, such as a smart-home environment (e.g., the type, location, capabilities, etc. of such devices). Thus, automation template 174C may be a framework that may need to be adapted to a particular environment for controlling an operation of one or more smart devices in the particular environment. The process of adapting an automation template 174C to a particular environment may be referred to as instantiating the automation template 174C, thus creating an automation instance 174B. According to some configurations, the automation engine 1 8 provides one or more automation code snippets 174D to the user that when executed perform some / all of the requested automation. A code snippet 174 is a small portion of re-usable code that does not include all of the code associated with an automation instance. For example, the automation engine 168 may output a code snippet 174 associated with a portion of the operations associated with an automation.

[0055] In some instances, a user may enter a natural language prompt into a text box on a client application (See FIGs. 12A-12F) and / or the user may use some other type of input, such as voice input. As will be described in more detail below, in some examples, the automation engine 168 may receive the prompt 172, pre-process the prompt, provide the prompt to one or more machine learning models 170 (e.g., trained to generate automations for smart devices), perform postprocessing, and provide the automation 174 to the user. See FIG. 6 for an example automation template 174C output and an example automation instance 174B output.

[0056] After an automation 174, such as an automation script 174A, is generated and provided to the user, the user may revise / edit the automation script 174A, and / or provide feedback related to the generated script (e.g., thumbs up / down / detailed feedback). In some examples, the feedback can be used to improve the performance of the machine learning model 170. For example, the training of the machine learning model 170 can be updated based on changes made to the automation script 174 and / or other feedback provided to the automation engine 168 from the user.

[0057] The machine learning model 170 can be one or more machine learning models and can be any suitable trained machine learning model configured for generating an automation 174. For example, the model 170 can be a trained neural network, such as a convolutional neural network (CNN). In some implementations, a pre-trained machine learning model can be obtained and finetuned to form the machine learning model used to generate the automations 174. The pre-trained machine learning model can be fine-tuned with training data that includes example inputs and example outputs. In some implementations, the training data can include a plurality of training examples (e.g., natural language prompts) with each training example including an example output. By fine-tuning the pre-trained machine learning model with training data, the parameters of the pre-trained machine learning model can be configured to generate the automations more precisely. More details are provided below with regard to FIG. 3.

[0058] According to some examples, a user 1 6 may also provide an automation 174, such as an automation script 174A, automation template 174C, automation instance 174B, automation code snippet(s) 174D, to the automation engine 168 to receive an automation summaiy 174E from the automation engine 168. For instance, a user may provide an automation script 174A to the automation engine 168 and receive an automation summary 174E that provides a summary of the automation. As an example, the user 166 may receive a summary7of an automation script that indicates that the lights in the living room are dimmed when a movie is started on a television within the living room. See FIG. 7 for example inputs and automation summary output.

[0059] In some cases, a graphical user interface (GUI) (See FIGs. 12A-12F) may be used to interact with the automation engine 168. The GUIs may include UI elements that display automation examples that may be used as a starting point for a user. For example, a user may select a ty pe of automation (e.g., nighttime security7, intruder alarm, lighting, ... ) that they would like to generate. Based on the selection of the type of automation, the automation engine 168 may request the automation script to be generated. According to some examples, the automation 174 can be customized by the user and / or by the automation engine 168 (’‘e.g., based on automationsthe user has scripted in the past”). In some configurations, the user can select an automation and then provide a natural language prompt 172 that modifies the selected automation.

[0060] FIG. 2 illustrates a simplified block diagram of a representative network architecture 200 that includes a smart home network 202 in accordance with some implementations. In some implementations, the smart devices 204 in the smart-home environment 100 (e.g., devices 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, and / or 122) combine with the hub device 180 to create a mesh network in smart home network 202. In some implementations, one or more smart devices 204 in the smart home network 202 operate as a smart home controller. Additionally, and / or alternatively, hub device 180 operates as the smart home controller. In some implementations, a smart home controller has more computing power than other smart devices. In some implementations, a smart home controller processes inputs (e g., from smart devices 204, electronic device 166, and / or server system 164) and sends commands (e.g., to smart devices 204 in the smart home network 202) to control operation of the smart-home environment 100. In some implementations, some of the smart devices 204 in the smart home network 202 (e.g., in the mesh network) are “spokesman” nodes (e.g., 204-1) and others are “low-powered” nodes (e.g., 204-9). Some of the smart devices in the smart-home environment 100 are battery powered, while others have a regular and reliable power source, such as by connecting to wiring (e.g., to 120V line voltage wires) behind the walls 154 of the smart-home environment. The smart devices that have a regular and reliable power source are referred to as “spokesman” nodes. These nodes are ty pically equipped with the capability of using a wireless protocol to facilitate bidirectional communication with a variety of other devices in the smart-home environment 100, as well as with the server system 164. In some implementations, one or more “spokesman” nodes operate as a smart home controller. On the other hand, the devices that are battery7powered are the “low-power” nodes. These nodes tend to be smaller than spokesman nodes and typically only communicate using wireless protocols that require very little power, such as Zigbee, ZWave, 6L0WPAN, Thread, Bluetooth, etc.

[0061] In some implementations, some low-power nodes may be incapable of bidirectional communication. These low-power nodes may send messages, but they are unable to “listen.” Thus, other devices in the smart-home environment 100, such as the spokesman nodes, need not send information to these low-power nodes. In some implementations, some low-power nodes are capable of only a limited bidirectional communication. For example, other devices are able to communicate with the low-power nodes only during a certain time period.

[0062] In some implementations, the smart devices may serve as low-power and spokesman nodes to create a mesh network in the smart-home environment 100. In some implementations, individual low-power nodes in the smart-home environment may regularly send out messages regarding what they are sensing, and the other low-powered nodes in the smart-home environment - in addition to sending out their own messages - may forward these messages, thereby causing the messages to travel from node to node (i.e.. device to device) throughout the smart home network 202. In some implementations, the spokesman nodes in the smart home network 202, which are able to communicate using a relatively high-power communication protocol, such as IEEE 802. 11, are able to switch to a relatively low-power communication protocol, such as IEEE 802.15.4, to receive these messages, translate the messages to other communication protocols, and send the translated messages to other spokesman nodes and / or the server system 164 (using, e.g., the relatively high-power communication protocol). Thus, the low-powered nodes using low- power communication protocols can send and / or receive messages across the entire smart home network 202, as well as over the Internet 162 to the server system 164. In some implementations, the mesh network enables the server system 164 to regularly receive data from most or all of the smart devices in the home, make inferences based on the data, facilitate state synchronization across devices within and outside of the smart home network 202, and send commands to one or more of the smart devices to perform tasks in the smart-home environment.

[0063] The spokesman nodes and some of the low-powered nodes are capable of “listening."’ Accordingly, users, other devices, and / or the server system 164 may communicate control commands to the low-powered nodes. For example, a user may use the electronic device 166 (e.g., a smart phone) to send commands over the Internet to the server system 164, which then relays the commands to one or more spokesman nodes in the smart home network 202. The spokesman nodes may use a low-power protocol to communicate the commands to the low-power nodes throughout the smart home network 202, as well as to other spokesman nodes that did not receive the commands directly from the sen- er system 164.

[0064] In some implementations, a smart nightlight 180, which is an example of a smart device 204, is a low-power node. In addition to housing a light source, the smart nightlight 180 houses an occupancy sensor, such as an ultrasonic or passive IR sensor, and an ambient light sensor, such as a photo resistor or a single-pixel sensor that measures light in the room. In some implementations, the smart nightlight 180 is configured to activate the light source when its ambient light sensor detects that the room is dark and when its occupancy sensor detects that someone is in the room. In other implementations, the smart nightlight 180 is simply configured to activate the light source when its ambient light sensor detects that the room is dark. Further, in some implementations, thesmart nightlight 180 includes a low-power wireless communication chip (e.g., a ZigBee chip) that regularly sends out messages regarding the occupancy of the room and the amount of light in the room, including instantaneous messages coincident with the occupancy sensor detecting the presence of a person in the room. As described above, these messages may be sent wirelessly (e.g., using the mesh network) from node to node (i.e., smart device to smart device) within the smart home network 202 as well as over the Internet 162 to the server system 164.

[0065] Other examples of low-power nodes include battery-operated versions of the smart hazard detectors 104. These smart hazard detectors 104 are often located in an area without access to constant and reliable powder and may include any number and type of sensors, such as smoke / fire / heat sensors (e.g., thermal radiation sensors), carbon monoxide / dioxide sensors, occupancy / motion sensors, ambient light sensors, ambient temperature sensors, humidity sensors, and the like. Furthermore, smart hazard detectors 104 may send messages that correspond to each of the respective sensors to the other devices and / or the server system 164, such as by using the mesh network as described above.

[0066] Examples of spokesman nodes include smart doorbells 106, smart thermostats 102, smart wall switches 108, and smart wall plugs 110. These devices are often located near and connected to a reliable power source, and therefore may include more power-consuming components, such as one or more communication chips capable of bidirectional communication in a variety of protocols.

[0067] As explained above with reference to FIG. 1, in some implementations, the smart-home environment 100 of Figure 1 includes a hub device 180 that is communicatively coupled to the network(s) 162 directly or via the network interface 160. The hub device 180 is further communicatively coupled to one or more of the smart devices using a radio communication network that is available at least in the smart-home environment 100. Communication protocols used by the radio communication netw ork include, but are not limited to, ZigBee, Z-Wave, Insteon, EuOcean, Thread. OSIAN, Bluetooth Low Energy and the like. In some implementations, the hub device 180 not only converts the data received from each smart device to meet the data format requirements of the netw ork interface 160 or the netw ork(s) 162, but also converts information received from the network interface 160 or the network(s) 162 to meet the data format requirements of the respective communication protocol associated with a targeted smart device. In some implementations, in addition to data format conversion, the hub device 180 further processes the data received from the smart devices or information received from the network interface 160 or the network(s) 162 preliminary. For example, the hub device 180 can integrate inputs frommultiple sensors / connected devices (including sensors / devices of the same and / or different types), perform higher level processing on those inputs - e.g., to assess the overall environment and coordinate operation among the different sensors / devices - and / or provide instructions to the different devices based on the collection of inputs and programmed processing. It is also noted that in some implementations, the network interface 160 and the hub device 180 are integrated to one network device. Functionality described herein is representative of particular implementations of smart devices, control application(s) running on representative electronic device(s) (such as a smart phone), hub device(s) 180, and server(s) coupled to hub device(s) via the Internet or other Wide Area Network (WAN). All or a portion of this functionality and associated operations can be performed by any elements of the described system - for example, all or a portion of the functionality described herein as being performed by an implementation of the hub device can be performed, in different system implementations, in whole or in part on the server, one or more connected smart devices and / or the control application, or different combinations thereof.

[0068] FIG. 3 illustrates a representative operating environment in which a server system 164 interacts with client devices and smart devices and provides processing for generating automations. As shown in FIG. 3, the server system 164 receives input from client devices 320. For example, the portable electronic devices 166A, and 166B illustrated in FIG. 1 are examples of a client device 320. In some implementations, the server system 164 is one or more servers that provides automation script services to client devices 320.

[0069] In accordance with some implementations, each of the client devices 320 includes a client-side module. The client-side module communicates with a server-side module executed on the server system 164 through the one or more networks 162. The client-side module provides client-side functionality for communications with the server-side module. The server-side module provides server-side functionality7for communication with client-side modules and generating automation scripts, providing summaries of automation scripts, and / or performing other functionality.

[0070] In some implementations, the server system 164 includes one or more processors 302, an automation engine 168, a speech-to-text engine 304, a pre-processor 306, a validator 308, a data store 330, a model trainer, and an I / O interface 312. The I / O interface 312 facilitates the clientfacing input and output processing. The data store 330 stores different information, such as smart device data 332 associated with smart devices in different environments, script data 334, training data 336, test data 338, other data 340, machine learning model(s) 170, and the like.

[0071] Examples of a representative client device 320 include a handheld computer, a wearable computing device, a personal digital assistant (PDA), a tablet computer, a laptop computer, a desktop computer, a cellular telephone, a smart phone, an enhanced general packet radio service (EGPRS) mobile phone, a media player, a navigation device, a game console, a television, a remote control, a point-of-sale (POS) terminal, a vehicle-mounted computer, an eBook reader, or a combination of any two or more of these data processing devices or other data processing devices.

[0072] Examples of the one or more networks 162 include local area networks (LAN) and wide area networks (WAN) such as the Internet. The one or more networks 162 are implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FIREWIRE, Long Term Evolution (LTE), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0073] In some implementations, the server system 164 may be implemented on one or more standalone data processing apparatuses or a distributed network of computers. In some implementations, the server system 164 also employs various virtual devices and / or services of third-party service providers (e.g., third-party cloud service providers) to provide the underlying computing resources and / or infrastructure resources of the server system 164. In some implementations, the server system 164 includes, but is not limited to, a server computer, a handheld computer, a tablet computer, a laptop computer, a desktop computer, or a combination of any two or more of these data processing devices or other data processing devices.

[0074] The server-client environment shown in FIG. 3 includes both a client-side portion (e.g., the client-side module) and a server-side portion (e.g., the server-side module). The division of functionality between the client and server portions of operating environment can vary7in different implementations. For example, in some implementations, the client-side module is a thin-client that provides only user-facing input and output processing functions, and delegates all other data processing functionality to a backend server (e.g., the server system 164). Although many aspects of the present technology are described from the perspective of the server system 164, the corresponding actions performed by a client device 320 would be apparent to one of skill in the art. Similarly, some aspects of the present technology may be described from the perspective of a client device, and the corresponding actions performed by the server would be apparent to one of skill in the art. Furthermore, some aspects of the present technology may be performed by the sen' er system 164, and a client device 320.

[0075] FIG. 3 also illustrates training and using one or more machine learning models 170 for generating automations 174, according to certain embodiments. As briefly discussed above, one or more automations 174 can be generated using a machine learning model, such as one or more machine learning models 174, that receives an input, such as a natural language request, and outputs a generated automation 174.

[0076] According to some examples, a model trainer 310 may train one or more machine learning models 170. The model trainer 310 uses at least a portion of training data 336 to train a particular model 170. For example, the training data 336 may include a portion of training data to train a first machine learning model 170 to generate automation scripts 174A, automation instances 174B, automation templates 174C, and automation code snippets 174D, and another portion of training data 336 to train a second machine learning model 170 to generate automation summaries 174E.

[0077] In some examples, the training data 336 includes data such as a prompt (and variations of the prompt), and example code that when executed causes the automation to be performed.Referring to FIG. 5, an instance of the training data 336 is illustrated. As shown in FIG. 5, a prompt 502 is included, along with variations of the prompt 504, and a YAML instance. Other data can also be used to train the one or more machine learning models 170.

[0078] Generally, the model trainer 310 determines parameters for the machine learning models 170 that optimize performance for generating an automation 174. In some examples, the model trainer 310 uses the instances of training data 310 to determine the parameters (e.g., weights and / or biases) that minimize an objective function. An objective function may include using a solution of an instance of the training data 310 into a machine learning model 170 and evaluating it against the training data. The parameters can be tuned to control the behavior of a machine learning model 170. In some examples, parameters may be defined and optimized to reduce memory', reduce processing, and / or adapt a model to a specific scenario.

[0079] According to some examples, during training, the model trainer 310 calculates, using an objective function, the difference between the actual output of a machine learning model 170 and the predicted output to determine the cost / error. The error is a function of the parameters of the model (e.g., the weights and bias). To minimize the error, the model parameters can be incrementally updated by minimizing the objective function over the instances of the training instances obtained from training data 336. After the parameters are specified by the model trainer 310, the machine learning model 170 may be validated using validation data, such as in other data 340. The machine learning model 170 may then be deployed for use by the automation engine 168.As should be understood, other training / validation mechanisms are contemplated and may be used. For example, the machine learning models 170 may be new models or existing models.

[0080] As briefly discussed above, a user 166 requests an automation 174 to be generated by the automation engine 168. In some examples, the user 166 submits a natural language prompt / query that describes the automation 174 that they would like to have generated. Instead of having to manually code the automation 174, the user can use natural language to provide the prompt. For example, a user may enter (e.g., via a text input box or speech) a prompt "‘Turn on living room lights at sunset, dim at 10 PM, and turn off at midnight’’, or some variation of the prompt (e.g., “Illuminate the living room at sunset, soften the light at 10 PM, and extinguish the lights at midnight.”) to have an automation 174 generated. In some examples, the speech-to-text engine 304 may convert speech to text that can be provided to the automation engine 168.

[0081] According to some examples, a request may not be specific to perform certain actions (e.g., at night turn on my living room lights), but instead could be a generic request such as “generate a cool new automation for my home.” In response to the generic request, the automation engine 168 can identify the smart devices within the home, and then determine different automations 174 that could be generated. In some examples, the automation engine 168 could access information that determines what automations are currently popular and / or trending. The automation engine could then present the user with different options using a graphical user interface and then create the automation.

[0082] In some examples, automation engine 168 may perform or cause to perform preprocessing before using one or more machine learning models 170 to generate the automations 174. According to some configurations, the pre-processing can include cleaning up the request / prompt that the user issues. The pre-processing can include but is not limited to appending data to the request, adjusting portions of the request, and the like. In some examples, the preprocessing can include determining what smart devices are associated with the user (e.g., within one or more environments), determining what functionality is associated with the smart devices, and determining what type of user generated the request (e.g., a developer or an end user).

[0083] In some configurations, the type of automation 174 returned to a computing device associated with a user is based on the type of user making the request (e.g., a developer or an end user). Automatically determining what type of automation to generate provides many benefits over existing systems. For example, providing a developer with an automation template 174C, and / or automation code snippets 174D, saves time for a developer as they are provided with a framework that is a modifiable starting point. This also reduces the use of computing resources (e.g.,processing / memoiy usage) since the automations can be easily modified for different environments.

[0084] The automatic generation of the automations 174 also reduces time to develop an automation and significantly reduces the time a user / developer spends searching documentation to find the smart-home device to perform desired functionality, how to interact with a specific device type, and / or the correct syntax to interact with the smart device. For instance, a developer may not know the nuances of a customized syntax that may be used by a company to interact with its smart devices. As an example, even if a user is a YAML expert, that user may still need to understand how to code in a customized variant of YAML. Further, even simple automation scripts 174A may use many lines of code. For instance, an automation script 174A generated for the natural language prompts 172 “Send me a notification if an unfamiliar face is detected in the backyard while I am on vacation”. “When TV turns on Dim lights, if the lights are previously on and at a higher brightness”, “When my kid comes back home from school in the evening, send me a notification”, and the like may all use more than twenty lines of code to perform the desired actions.

[0085] Using techniques described herein, a user 166 can cause complex automations to be automatically generated that use a diverse set of smart devices having different capabilities without spending a large amount of time and / or use of computing resources. For example, computer use is reduced since the automation engine 168 can automatically generate the different automations 174 that can be executed and / or modified to execute with minimal intervention. Further, the automations 174 generated can be more efficient in the use of computer resources since the automation engine 168 can factor this into the generation of the automations 174.

[0086] As discussed above, manually creating automations 174 using scripting language can be a complex, time-consuming, and high-friction process for some users. In some examples, artificial intelligence (Al) techniques can be used to generate automations. For instance, Al techniques can be used to generate automations, such as automation scripts 174A, instances 174B, templates 174C, code snippets 174D, and summaries 174E. According to some embodiments, the automation 174 that is returned depends on the type of user (e.g., developer, user) making the request.

[0087] When a developer requests an automation 174, a template 174C and / or code snippet 174D may be returned by the automation engine 168. When an end user requests an automation 174, an instance 174B and / or a script 174A (e g., specific to the user's environment) may be returned. As a more concrete example, in some configurations, when a developer asks for an automation 174 that turns on the lights at sunset, a template 174A is generated and provided to the developer and when an end user asks for the same automation 174, an instance 174B is returned.According to some examples, the automation scripts 174A, templates 174C, and / or automation code snippets 174D are generated and output as YAML.

[0088] Once preprocessed, the request is provided by the automation engine 168 to one or more machine learning models 170. The one or more machine learning models 170 generates an automation 174 that can be provided to the user 166. In some examples, the generated automation 174 can be validated. For instance, after receiving the script / template, a validator 308 can be used to validate the YAML. and possibly make corrections to the generated code.

[0089] According to some examples, the automation engine 168 is configured to suggest and / or generate automations 174 based on data, such as smart device data 332, usage information, and the like. For instance, if a user 1 6 takes a set of actions such as turning on a set of lights and closing my living room blinds at 10 min past sunset, then the automation engine 168 may recommend and / or generate a smart home automation for that user.

[0090] As briefly discussed above, in some examples, an automation summary 174E (e.g.. a textual description of an automation or a voice description of an automation, in particular a natural language textual or voice description) can be generated from an automation script. For instance, an automation script can be provided by a user to the automation engine 168. The automation engine 168 generates an automation summary 174E that describes what an automation (e.g.. an automation script written in YAML) does. Instead of a user having to know how to interpret an automation script, the automation engine 168 uses one or more machine learning models 170 to programmatically generate the automation summary' 174E. See FIG. 7 for example inputs and an example output for an automation summary 174E.

[0091] According to some examples, the user may provide feedback (e g., thumbs up / thumbs down feedback, open ended feedback, . .. ). The user, such as a developer / tester, may also be provided with a rewrite option that allows the user to make changes to the script and then, based on the changes, the script can be updated / verified by the automation engine 168. For example, the user may refine specific sections of the script, insert new sections, delete sections, move sections, and the like.

[0092] FIG. 4 is a block diagram illustrating a representative smart device 204 in accordance with some implementations. In some implementations, the smart device 204 (e.g., any devices of a smart-home environment 100, Figure 1) includes one or more processing units (e.g., CPUs, ASICs, FPGAs, microprocessors, and the like) 402, one or more communication interfaces 404, memory 406, communications module 442 with radios 440, and one or more communication buses 408 for interconnecting these components (sometimes called a chipset). In some implementations, the userinterface 410 includes one or more output devices 412 that enable presentation of media content, including one or more speakers and / or one or more visual displays. In some implementations, the user interface 410 also includes one or more input devices 414, including user interface components that facilitate user input such as a keyboard, a mouse, a voice-command input unit or microphone, a touch screen display, a touch-sensitive input pad, a gesture capturing camera, or other input buttons or controls. Furthermore, some smart devices 204 use a microphone and voice recognition or a camera and gesture recognition to supplement or replace the keyboard. In some implementations, the smart device 204 includes one or more image / video capture devices 418 (e.g., cameras, video cameras, scanners, photo sensor units). The built-in sensors 4490 may include, for example, one or more thermal radiation sensors, ambient temperature sensors, humidity sensors, IR sensors, occupancy sensors (e.g., using RFID sensors), ambient light sensors, motion detectors, accelerometers, and / or gyroscopes.

[0093] The radios 440 enable one or more radio communication networks in the smart-home environments, and allow a smart device 204 to communicate with other devices. In some implementations, the radios 440 are capable of data communications using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4. Wi-Fi, ZigBee, 6L0WPAN, Thread. Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, etc.) custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), and / or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0094] The communication interfaces 404 include, for example, hardware capable of data communications using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6L0WPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, etc.) and / or any of a variety of custom or standard wired protocols (e.g., Ethernet, HomePlug, etc ), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0095] The memory 406 includes high-speed random-access memory, such as DRAM. SRAM, DDR RAM, or other random-access solid-state memory devices; and, optionally, includes nonvolatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid state storage devices. The memory 406. or alternatively the non-volatile memory- within the memory- 406, includes a non-transitory computer readable storage medium. In some implementations, the memory 406, or the non-transitory computer readable storage medium of the memory 406, stores the following programs, modules, and data structures, or a subset or superset thereof: operatinglogic 420 including procedures for handling various basic system sen-ices and for performing hardware dependent tasks; a device communication module 422 for connecting to and communicating with other network devices (e.g.. network interface 160. such as a router that provides Internet connectivity, networked storage devices, network routing devices, server system 164, etc.) connected to one or more networks 162 via one or more communication interfaces 404 (wired or wireless); an input processing module 426 for detecting one or more user inputs or interactions from the one or more input devices 414 and interpreting the detected inputs or interactions; a user interface module 428 for providing and displaying a user interface in which settings, captured data, and / or other data for one or more devices (e.g., the smart device 204, and / or other devices in smart-home environment 100) can be configured and / or viewed; one or more applications 430 for execution by the smart device (e.g., games, social network applications, smart home applications, and / or other web or non-web based applications) for controlling devices (e.g., executing commands, sending commands, and / or configuring settings of the smart device 204 and / or other client / electronic devices), and for reviewing data captured by devices (e.g., device status and settings, captured data, or other information regarding the smart device 204 and / or other client / electronic devices); a device-side module 432, which provides device-side functionalities for device control, data processing and data review, including but not limited to: a command receiving module 4320 for receiving, forwarding, and / or executing instructions and control commands (e.g., from a client device 320, from a server system 164, from user inputs detected on the user interface 410. etc.) for operating the smart device 204; a data processing module 4322 for processing data captured or received by one or more inputs (e.g., input devices 414, image / video capture devices 418, location detection device 416), sensors (e.g., built-in sensors 490), interfaces (e.g., communication interfaces 404, radios 440), and / or other components of the smart device 204, and for preparing and sending processed data to a device for review (e.g., client devices 220 for review by a user); device data 434 storing data associated with devices (e.g., the smart device 204), including, but is not limited to: account data 4340 storing information related to user accounts loaded on the smart device 204, wherein such information includes cached login credentials, smart device identifiers (e.g., MAC addresses and UUIDs). user interface settings, display preferences, authentication tokens and tags, password keys, etc.; local data storage database 4342 for selectively storing raw or processed data associated with the smart device 204 (e.g., video surveillance footage captured by a camera 118); a bypass module 436 for detecting whether radio(s) 440 are transmitting signals via respective antennas coupled to the radio(s) 440 and to accordingly couple radio(s) 440 to their respective antennas either via a bypass line or an amplifier (e.g., a low noise amplifier); and a transmission access module 438 for granting ordenying transmission access to one or more radio(s) 440 (e.g., based on detected control signals and transmission requests).

[0096] Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or otherwise rearranged in various implementations. In some implementations, the memory 406, optionally, stores a subset of the modules and data structures identified above. Furthermore, the memory 406, optionally, stores additional modules and data structures not described above.

[0097] Turning now to FIG. 8, a flowchart 800 of a method is illustrated for automatically generating automations using a trained machine learning model, according to some embodiments.

[0098] At 802. a query is received that requests an automation 174. As discussed above, the automation engine 168 receives a query provided by a user, such as user 166. Instead of having to manually code the automation 174, the user 166 can use natural language to provide the query for the automation 174 that they would like to have generated. For example, a user may enter (e g., via a text input box or speech) a prompt “Turn on living room lights at sunset, dim at 10 PM, and turn off at midnight’’. In another example, a user 166 may request the automation engine 166 to “Provide a summary of the living room lights automation”, and / or provide the automation 174 to the automation engine 166 to have an automation summary generated. In other examples, an automation can be generated to control operations of one or more applications (e g., generate text messages, add calendar events, ... ).

[0099] At 804, a type of user making the request for an automation is determined. As discussed above, the automation engine 168 may determine a type of automation 174 to return based on the ty pe of user (e.g., a developer or an end user) making the request. For instance, in some examples, when a developer requests an automation 174, the automation engine 168 using the machine learning model 170 returns an automation template 174C. or code snippets 174D, and when the user is an end user the automation engine 168 returns an automation script 174A and / or an automation instance 174B that can be executed (e.g., without modification) within the environment of the user.

[0100] At 806, pre-processing can be performed before generating the automation. As discussed above, the automation engine 168 may process the request by altering the request, determine whatsmart devices are available, the functionality of the smart devices, and the like. See FIG. 10 for further details.

[0101] At 808. the automation is generated using one or more machine learning models 170. As discussed above, the automation engine 168 may use one or more machine learning models 170 to obtain an automation(s) 174 based on the request. In some examples, the automation engine 168 may return more than one automation 174 based on the request. For example, the request may ask for specific automations, and / or the automation engine 168 may identify that this user typically asks for certain automations 174.

[0102] At 810, post-processing can be performed. As discussed above, the automation engine 1 8 may programmatically validate the generated automation to identify whether it is correct. In some examples, the automation engine 168 may programmatically correct one or more of the errors detected.

[0103] At 812. the automation 174 is provided. As discussed above, the automation engine 168 causes the automation(s) 174 to be provided to a computing device associated with the user. In some examples, the automation 174 may be provided as a text file that includes human-readable text. In other examples, the automation 174 may be an executable instance that can be executed by one or more devices associated with the user.

[0104] At 814. the machine learning model 170 is updated when determined. As discussed above, users may provide feedback for the automations 174 generated by the automation engine 168. As will be discussed in more detail in FIG. 11, the automation engine 168 and / or some other device or component may use the feedback data to refine one or more of the machine learning models 170.

[0105] FIG. 9 illustrates a flowchart 900 of a method for determining a type of user associated with a request, according to some embodiments.

[0106] At 902, an interface used to provide the query is determined. As discussed above, the automation engine 168 may determine what type of user (e.g., developer or end-user) is making a request. In some configurations, the type of user can be determined by the automation engine 168 based on the interface the user uses to interact with the automation engine 168. For example, when the interaction is received through a developer interface, the user providing the query is identified as a developer. When the interaction is received through an end-user interface, the user providing the query is identified as an end-user. In other examples, the automation engine 168 may identify the type of user based on some other information. For example, the automation engine 168 mayidentify a type of user based on content of the request (e.g., the request asks for an automation for their home) and / or based on account settings, or some other information available to the automation engine 168.

[0107] At 904, a determination is made as to whether the user is an end user. As discussed above, the automation engine 168 can identify' a type of user using different information such as but not limited to the fype of interface used to access the automation engine 168, account information associated with the user, and the like. When the user is an end user, the process flows to 906. When the user is not an end user, the process flows to 908.

[0108] At 906, a determination is made to return an automation that is not specific to an environment. As discussed above, the automation engine 168 generates an automation 174 that can be easily changed to accommodate different users instead of being personalized for a particular user.

[0109] At 908. a determination is made to return an automation that is specific to an environment. As discussed above, the automation engine 168 generates an automation 174 that is personalized for a particular user based on available smart devices within an environment.

[0110] FIG. 10 illustrates a flowchart 1000 of a method for performing pre-processing, according to some embodiments.[OHl] At 1002, smart devices are determined. As discussed above, the smart devices within an environment may be determined using different techniques. For example, the automation engine 168 may access a registry to identify' the smart devices within one or more environments. In other examples, the automation engine 168 may perform operations to identify' smart devices such as determining what devices are communicating on a network within an environment. In yet other examples, at least a portion of the smart devices may be queried by the automation engine 168 to determine smart devices 204 that are available within an environment.

[0112] At 1004, functionality of the smart devices is determined. As discussed above, the smart devices may have a wide range of functionality. In some examples, the server 164 may receive data from most or all the smart devices in the home, make inferences based on the data, facilitate state synchronization across devices within and outside of the smart home network 202, and send commands to one or more of the smart devices to perform tasks in the smart-home environment. In some examples, a hub device may provide information to the server 164. In other examples, the automation engine 168 may access stored data that includes information about the functionality of the smart devices within an environment.

[0113] At 1006, information about the smart device is provided to the automation engine 168. As discussed above, the automation engine 168 may use the smart device information to personalize the automation 174 provided to the user.

[0114] FIG. 11 illustrates a flowchart 1100 of a method for receiving feedback data and updating a trained machine learning model, according to some embodiments.

[0115] At 1102, feedback is received. As discussed above, the user may revise / edit the automation script 174A, automation template 174B. code snippets 174D and / or provide feedback related to the generated automation (e.g., thumbs up / down feedback or more detailed feedback). In some examples, the feedback can be used to improve the performance of the machine learning model 170. For example, the training of the machine learning model 170 can be updated based on changes made to the automation script 174 and / or other feedback provided to the automation engine 168 from the user.

[0116] At 1104, one or more machine learning models is updated based on the feedback. As discussed above, one or more of the machine learning models can be fine-tuned using the feedback data. In some examples, the training of the one or more machine learning models may be updated using labeled and / or unlabeled training data generated using the feedback. By fine-tuning the machine learning model, the parameters of the machine learning model can be configured to generate the automations more precisely.

[0117] At 1106, the updated machine learning model is deployed. As discussed above, the automation engine 168 may use the updated machine learning model 170 to generate the automations.

[0118] FIG. 12A illustrates a graphical user interface that includes user interface elements to edit an automation and provide feedback.

[0119] As illustrated, an example automations GUI 1202 shows information about an automation 174, ’‘Sunset Lighting Automation”, generated for a user. In the current example, the automation 174 is an automation script 174A that is scheduled to run between sunset every day and end at midnight.

[0120] For purposes of explanation, assume that the user provided a natural language prompt such as “Turn on lights in living room at sunset, dim them at 10 PM, and turn them off at midnight” to the automation engine 174. As briefly discussed above, the user could have also provided variations of the prompt that could be used to achieve the desired automation, such as but not limited to “turn on living room lights at sunset, dim at 10PM, and turn off at midnight”,“illuminate the living room at sunset, soften the light at 10 PM, and extinguish the lights at midnight", “turn on the living room lights when the sun goes down, turn them down a notch at 10 PM, and turn them off when you go to bed”, “control the living room lights so they are on at sunset, dimmed at 10 PM, and off at midnight”, “illuminate the living room at sunset, reduce the brightness at 10 PM, and extinguish the lights at midnight”, “turn on the living room lights when it gets dark, turn them down a bit at 10 PM, and turn them off when you go to bed”, “please turn on the living room lights at sunset, dim them at 10 PM, and turn them off at midnight”.

[0121] According to some examples, these prompt variations may be used to train one or more ML models used to generate the automation script. The ML models may also be automatically updated by feeding other variations of the prompt received from users.

[0122] In response to the user selecting the edit automation UI element 1206, the script editor 1208 UI element is displayed. The script editor GUI 1208 allows the user to view and make changes to the automation 174. As discussed above, in some examples, changes made to the automation 174 can be used to update one or more machine learning models that was used to generate the automation 174.

[0123] In some examples, in response to the user selecting the provide feedback UI element 1204, a feedback GUI element 1210 is displayed. As discussed above, in some examples, the user may provide feedback for the automation 174 such as, but not limited to thumbs up / down feedback, or more detailed feedback. In the current example, the user can select the thumbs up UI element 1212 to provide positive feedback, or the thumbs down UI element 1214 to provide negative feedback. The user could also use feedback text UI element 1216 to provide more detailed feedback. In some examples, this feedback can be used to update one or more machine learning models that w as used to generate the automation 174.

[0124] FIG. 12B illustrates example graphical user interfaces that includes user interface elements for discovering automations.

[0125] As illustrated, a discover automations GUI 1220 shows UI elements related to discovering automations 174. In the current example, the discover automations GUI 1220 provides different UI elements that allow a user to search automations 174, see recommended automations that works in the home environment of the user (e.g., morning lighting automation and arrive home camera automation), and a top chart area that displays top rated / use automations 174 from which the user can select an automation 174. In the current example, the user has selected an area of the discover automations GUI 1220, as indicated by element 1226. that causes the automations generator 168 to display top chart GUI 1228 that displays top automations according to differentcriteria, such as but not limited to, a specified time period (e.g., daily, weekly, monthly, all time, ... ), categories (e.g., lighting, safety, utilities, ... ). and availability (e.g., works with the smart devices in the home of the user).

[0126] FIG. 12C illustrates example graphical user interfaces that includes user interface elements for discovering automations by performing a search.

[0127] As illustrated, the discover GUI 1220 shows UI elements related to discovering automations 174. In the current example, the user has started entering text into the search UI element 1230 to discover automations. According to some configurations, the user may search for automations 174 using different input methods, such as using a keyboard 1232 or providing audio (not shown). In some examples, when the user starts to enter a search term, the discover automations GUI 1220 shows recent search terms (e.g., nighttime, light sync, doorbell, . . . ). In the current example, the user may either enter the search term, a portion of the search term, or select the nighttime recent search in order to see the top chart GUI 1234 that presents the results of the search.

[0128] The top chart GUI 1234 provides different UI element, such as UI elements 1236A, that allow a user to view the results of the search and to select one of the automations 174 returned as a result of the search. For example, the user may select one of UI elements 1236A. 1236B. 1236C, 1236D, or 1236E to see additional details relating to the selected automation 174. In some examples, additional details may be shown for a returned result. For instance, the night time energy savings automation shows that the automation engine 174 has identified that the environment of the user is missing two items (e.g., associated with a missing smart device, functionality of a smart device, .. . ) for that automation to be performed within the environment of the user.

[0129] FIG. 12D illustrates example graphical user interfaces that includes user interface elements for configuring an automation.

[0130] As illustrated, the selected automation GUI 1240 shows UI elements related to viewing and specifying options for a selected automation. In the current example, the user has selected the nighttime security UI element 1236D from top chart GUI 1234 illustrated in FIG. 12C to view and specify options associated with the night time security automation.

[0131] According to some configurations, the selected automation GUI 1240 includes UI elements that allow a user to view and configure options relating to the selected automation. In the current example, the selected automation GUI 1240 shows a schedule UI element 1242 that allowsa user to change the schedule, and a device UI element 1244 that allows a user to select the devices to use to perform the automation. In the current example, in response to selectin the device UI element 1244, the device GUI 1244 is displayed.

[0132] The device GUI 1244 shows different devices that can be used with the selected automation. As illustrated, the user has selected camera 1 in the living room, camera 4 in the living room, and the front door doorbell at the front door as smart devices to use with the selected automation. In some examples, the automation engine 168 may provide an indication that a smart device is not supported for a selected automation, such as camera 6 that is located in the garage.

[0133] FIG. 12E illustrates an example graphical user interface that includes user interface elements for configuring an automation.

[0134] As illustrated, the selected automation GUI 1240 shows UI elements related to viewing and specifying options for a selected automation. In the current example, the user has caused menu UI element 1250 to be displayed (e.g.. by right clicking within the display of the selected automation GUI 1240.

[0135] According to some configurations, the menu UI element 1250 includes UI elements that allow a user to share the automation with another user, edit the script for the selected automation, get help for the selected automation, provide feedback for the selected automation, and report an issue with the selected automation.

[0136] In the foregoing description, for the purposes of explanation, numerous specific details were set forth in order to provide a thorough understanding of various embodiments of the present invention. It will be apparent, however, to one skilled in the art that embodiments of the present invention may be practiced without some of these specific details. In other instances, welUknown structures and devices are shown in block diagram form.

[0137] The foregoing description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the foregoing description of the exemplary' embodiments will provide those skilled in the art with an enabling description for implementing an exemplary embodiment. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the invention as set forth in the appended claims.

[0138] Specific details are given in the foregoing description to provide a thorough understanding of the embodiments. However, it will be understood by one of ordinary' skill in the art that the embodiments may be practiced without these specific details. For example, circuits,systems, networks, processes, and other components may have been shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may have been shown wdthout unnecessary detail in order to avoid obscuring the embodiments.

[0139] Also, it is noted that individual embodiments may have been described as a process which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may have described the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed, but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to a return of the function to the calling function or the mam function.

[0140] The term “computer-readable medium” includes, but is not limited to portable or fixed storage devices, optical storage devices, wireless channels and various other mediums capable of storing, containing, or carrying instruction(s) and / or data. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, for arded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0141] Furthermore, embodiments may be implemented by hardw are, software, firmware, middleware, microcode, hardw are description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine readable medium. A processor(s) may perform the necessary tasks.

[0142] In the foregoing specification, aspects of the invention are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the invention is not limited thereto. Various features and aspects of the above-described invention may be used individually or jointly. Further, embodiments can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope ofthe specification. The specification and drawings are, accordingly , to be regarded as illustrative rather than restrictive.

[0143] Additionally, for the purposes of illustration, methods were described in a particular order. It should be appreciated that in alternate embodiments, the methods may be performed in a different order than that described. It should also be appreciated that the methods described above may be performed by hardware components or may be embodied in sequences of machineexecutable instructions, which may be used to cause a machine, such as a general-purpose or special-purpose processor or logic circuits programmed w ith the instructions to perform the methods. These machine-executable instructions may be stored on one or more machine readable mediums, such as CD-ROMs or other ty pe of optical disks, floppy diskettes, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memory, or other types of machine- readable mediums suitable for storing electronic instructions. Alternatively, the methods may be performed by a combination of hardware and software.

Claims

WHAT IS CLAIMED IS:

1. A method, comprising: receiving, via one or more processors, a request to generate an automation, wherein the request describes an automation that is associated with controlling an operation of one or more smart devices; and generating, via the one or more processors, the automation using a machine learning model trained to perform generate automations associated with controlling operations of at least one of the one or more smart devices, wherein the automation is one or more of an automation script, an automation instance, an automation template, an automation code snippet, or an automation summary.

2. The method of claim 1, further comprising: determining, via the one or more processors, that the request is associated with a developer; and wherein generating the automation comprises generating at least one of the automation template, or the automation code snippet.

3. The method of claim 1. further comprising: determining, via the one or more processors, that the request is associated with a user; and wherein generating the automation comprises generating at least one of the automation script, or the automation instance.

4. The method of claim 3, wherein generating the at least one of the automation script comprises generating code in a human-readable data-serialization language.

5. The method of claim 1, further comprising: determining, via the one or more processors, that the request is a request to generate an automation summary7; and wherein generating the automation comprises generating the automation summary.

6. The method of any of the preceding claims, further comprising determining, via the one or processors, one or more devices within an environment associated with the request; and wherein generating the automation is based, at least in part, on the one or more devices within the environment.

7. The method of any of the preceding claims, wherein receiving the request to generate the automation comprises receiving a natural language description of the automation.

8. The method of any of the preceding claims, further comprising: receiving feedback data indicating an accuracy of the automation; and updating the machine learning model based on the feedback data.

9. The method of any of the preceding claims, further comprising: providing, via the one or more processors, the automation to a computing device; causing at least a portion of the automation to be presented on a display associated with the computing device; and receiving, from the computing device, one or more edits to the automation.

10. The method of claim 9, further comprising causing the machine learning model to be updated based at least in part on the one or more edits.

11. A system comprising: one or more processors; and one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving a request to generate an automation, wherein the request describes an automation that is associated with controlling an operation of one or more smart devices; and generating the automation using a machine learning model trained to perform generate automations associated with controlling operations of at least one of the one or more smart devices wherein the automation is one or more of an automation script, an automation instance, an automation template, an automation code snippet, or an automation summary.

12. The system of claim 11, the operations further comprising: determining that the request is associated with a developer; and wherein generating the automation comprises generating at least one of the automation template, or the automation code snippet.

13. The system of claim 11, the operations further comprising: determining that the request is associated with a user; andwherein generating the automation comprises generating at least one of the automation script, or the automation instance.

14. The system of any of claims 11 to 13, the operations further comprising: receiving feedback data indicating an accuracy of the automation; and updating the machine learning model based on the feedback data.

15. The system of any of claims 11 to 14, the operations further comprising: causing at least a portion of the automation to be presented on a display associated with a computing device; receiving, from the computing device, one or more edits to the automation; and causing the machine learning model to be updated based at least in part on the one or more edits.

16. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving a request to generate an automation, wherein the request describes an automation that is associated with controlling an operation of one or more smart devices; and generating the automation using a machine learning model trained to perform generate automations associated with controlling operations of at least one of the one or more smart devices, wherein the automation is one or more of an automation script, an automation instance, an automation template, an automation code snippet, or an automation summary.

17. The non-transitory computer-readable medium of claim 16, comprising additional instructions that, when executed by the one or more processors, cause the one or more processors to perform additional operations comprising: determining, via the one or more processors, that the request is associated with a developer; and wherein generating the automation comprises generating at least one of the automation template, or the automation code snippet.

18. The non-transitory computer-readable medium of claim 16, comprising additional instructions that, when executed by the one or more processors, cause the one or more processors to perform additional operations comprising:determining, via the one or more processors, that the request is associated with a user; and wherein generating the automation comprises generating at least one of the automation script, or the automation instance.

19. The non-transitory computer-readable medium of any of claims 16 to 18, comprising additional instructions that, when executed by the one or more processors, cause the one or more processors to perform additional operations comprising: receiving feedback data indicating an accuracy of the automation; and updating the machine learning model based on the feedback data.

20. The non-transitory computer-readable medium of claim 16, comprising additional instructions that, when executed by the one or more processors, cause the one or more processors to perform additional operations comprising: providing the automation to a computing device; causing at least a portion of the automation to be presented on a display associated with the computing device; receiving, from the computing device, one or more edits to the automation; and causing the machine learning model to be updated based at least in part on the one or more edits.

Images