Accessory management system using environmental model

Abstract

This application relates to accessory management systems using an environment model. A controller is disclosed that can be used to control the operation of various accessories. A group of accessories can be organized into an accessory network that facilitates coordinated control of multiple accessories. The accessory network can be organized according to an environment model that can include a hierarchical representation of the physical environment in which the accessories are located. The environment model can be synchronized across different controllers that have access to the accessories.

Description

[0001] This application is a divisional application of the invention patent application filed on May 29, 2015, with application number 201580028720.1 and invention title "Attachment Management System Using an Environment Model".

[0002] Cross-reference to related applications

[0003] This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 005,764, filed May 30, 2014, entitled “Networking, Communication and Security for an Accessory Management System”; U.S. Provisional Patent Application No. 62 / 094,391, filed December 19, 2014, entitled “Networking, Communication and Security for an Accessory Management System”; and U.S. Patent Application No. 14 / 725,912, filed May 29, 2015. The disclosures of these patent applications are incorporated herein by reference in their entirety.

[0004] This disclosure also relates to U.S. Patent Application No. 14 / 614,914, filed February 5, 2015, and U.S. Provisional Patent Application No. 61 / 935,967, filed February 5, 2014, the disclosures of which are incorporated herein by reference in their entirety.

[0005] This disclosure also relates to U.S. Application No. 14 / 725,891, filed May 29, 2015, the entire disclosure of which is incorporated herein by reference. Technical Field

[0006] This disclosure relates in general to an attachment management system, and more specifically to an attachment management system using an environment model. Background Technology

[0007] Electronic devices are becoming increasingly prevalent in a range of applications. Mobile phones, tablets, and home entertainment systems are just some of the electronic devices that users frequently interact with.

[0008] Another type of electronic device that is becoming more prevalent includes various devices that can be controlled electronically, such as thermostats, lighting equipment, and home appliances. Summary of the Invention

[0009] Currently, users may find it difficult to manage multiple electronically controllable devices or systems. For example, a user's home might contain a thermostat, an electronically controllable lighting system, a home security system, etc. Each of these systems can be manufactured by different manufacturers, and each manufacturer may offer dedicated controller devices (e.g., IR-based remote control devices) or controller applications (“or applications”) that users can install and run on general-purpose devices such as smartphones, tablets, or home computer systems. Each controller device or application is typically tailored to a specific manufacturer's system and may not interoperate with systems from other manufacturers or even other systems from the same manufacturer. This piecemeal approach is not easily scalable. Users attempting to create a “smart home” environment by leveraging a heterogeneous array of devices that can be centrally controlled or managed face the challenge of accumulating too many controller devices and / or controller applications.

[0010] Specific embodiments of the invention operate within the context of protocols for communication between a controller device (or “controller”) and any number of other electronic devices to be controlled (referred to herein as “accessory devices” or simply “accessories”). For example, the controller can be implemented on a general-purpose computing device such as a desktop computer, laptop computer, tablet computer, smartphone, other mobile phone, other handheld or wearable computing device by providing appropriate executable program code for the general-purpose computing device; alternatively, the controller can be a dedicated computing device. Accessories can include any device capable of being controlled by the controller. Examples of accessories include lighting fixtures, thermostats, door locks, automatic door openers (e.g., garage door openers), still cameras, or video cameras, etc. Accessories and the controller can communicate with each other via wired or wireless channels using standard transmission protocols such as Wi-Fi, Bluetooth, Bluetooth LE, etc. It should be understood that other communication protocols and transmissions may be used.

[0011] In some implementations, a "unified" attachment protocol can be provided, via which the controller can send command and control messages to attachments and receive responses from attachments in a unified format, regardless of the attachment's type or function. For example, an attachment can be defined as a collection of services, where each service is defined as a set of characteristics, each characteristic having a defined value at any given time. These characteristics can represent various aspects of the attachment's state. The protocol can define the message format via which the controller can (e.g., by reading) query and (e.g., by writing) update the characteristics of the attachment (individually or in groups), thereby allowing the controller to determine and / or change the attachment's state. Therefore, any type of attachment can be controlled in a consistent manner, regardless of its function.

[0012] In some implementations, the protocol may define security measures to prevent unauthorized controllers from manipulating attachments. For example, an attachment may be configured to accept requests only from controllers that have previously been paired with the attachment and are therefore recognized by it. The protocol may specify a pairing process to minimize the risk of pairing without the attachment's legitimate owner / operator's approval. Furthermore, the protocol may specify end-to-end message encryption, ensuring that only specific controllers and attachments can decrypt messages exchanged between them.

[0013] Some aspects of this invention relate to controller networks in which multiple controllers can be paired with or otherwise configured to communicate with the same attachment (or the same set of attachments, such as an attachment network). In some controller networks, one or more controllers can establish a level of privilege (e.g., "administrative" privilege) with an attachment that authorizes these controllers to determine whether other controllers should be authorized to transmit command and control messages to the attachment. For example, a first controller can be paired with an attachment. Establishing a pairing may involve providing the attachment with the first controller's long-term public key and, in exchange, receiving a first long-term public key for the attachment. Other operations (e.g., out-of-band PIN or password exchange) may also be involved in establishing a pairing. Independent of any communication with the attachment, the first controller can obtain a long-term public key for a second controller. The first controller can use the first long-term public key received during pairing to establish an authenticated session with the attachment. The authenticated session may have a session key and can be used to encrypt all communication within the authenticated session. Within the authenticated session, the first controller can perform a pairing add-on operation with the attachment to establish a pairing between the attachment and the second controller. The pairing process may include providing the attachment with a long-term public key for the second controller, and in exchange receiving a second long-term public key for the attachment (which may or may not be the same key received when the first controller established its pairing). The first controller may transmit the second long-term public key for the attachment to the second controller. This process establishes a pairing between the second controller and the attachment; subsequently, the second controller may establish its own authenticated session to send command and control messages to the attachment. The first controller may repeat the pairing process to establish pairings between the attachment and any number of controllers.

[0014] In some cases, the first controller can instruct the accessory to grant administrator (or "admin") privileges to the second controller. Granting this privilege allows the second controller to perform pairing add operations to add additional controllers as needed, depending on the specific implementation. The second controller may or may not be able to grant administrator privileges to the additional controllers. Administrator privileges can be automatically assigned to the first controller that pairs with a brand new accessory (or with an accessory that has not yet been paired). Using administrator privileges helps device owners manage which controllers have access to specific accessories.

[0015] In some controller networks, one or more controllers may act as proxies for communicating with an accessory. For example, an accessory may be configured to communicate only with controller devices that are physically present in the local environment (e.g., on a local area network, which may be wired or wireless as needed, or within range of a point-to-point wireless communication protocol such as Bluetooth). A controller device not physically present in the local environment may establish communication with another controller (a proxy device or agent) that is physically present in the accessory's local environment. This proxy can relay messages and responses between the geographically distant controller device and the accessory. The geographically distant controller and accessory may establish a pairing authentication session and encrypt their communication; the proxy does not need to be able to read messages and responses, only to relay them as if receiving them. In some implementations, it may be necessary for the controller acting as a proxy for another controller to establish its own pairing authentication session with the accessory before the accessory will accept any messages relayed by the proxy. In some implementations, the proxy and another controller may also establish a pairing authentication session between them, which can provide further protection against unauthorized access.

[0016] In some implementations, the controller may prefer direct communication when possible and use a proxy when the attachment cannot be directly contacted. For example, after pairing with the attachment, the controller may receive user input (or other input) requesting interaction with the attachment (e.g., to check or change its status). In response to this input, the controller may determine whether the attachment can be directly contacted, such as whether the controller and the attachment are on the same local area network (LAN). If so, the controller may communicate directly with the attachment to establish a pairing authentication session and exchange command and control messages. If the attachment cannot be directly contacted, the controller may attempt to identify a proxy that can be contacted, for example, via a LAN or WAN. The attachment may establish communication with the proxy, and then the controller can communicate with the attachment through the proxy.

[0017] In some implementations, the agent can be any controller that has been paired with the attachment and exists in the attachment's local environment. The agent can receive requests from the controller to communicate with the attachment. In response, the agent can establish its own pairing authentication session with the attachment. Messages received from the controller can be relayed to the attachment via the pairing authentication session, and messages received from the attachment via the pairing authentication session can be relayed to the controller. The agent may not know the content of the messages it relays; therefore, for example, the controller and the attachment can send encrypted messages using keys unknown to the agent (e.g., its own pairing authentication session key). The agent can continue relaying messages until one or the other (or both) of the controller and the attachment sends a message indicating that relaying can be terminated. At this point, the agent can end its authentication session and stop relaying messages.

[0018] Such agents can provide relay functionality that extends the physical range of a controller network without connecting attachments to a wide area network. Some agents, referred to as coordinators, offer additional functionality beyond simply relaying messages. For example, a coordinator can mediate access to attachments (or a group of attachments, such as an attachment network). The coordinator can pair with attachments and remain in the local environment with them. Other controllers can pair with the coordinator to supplement or replace pairing with attachments. During a pairing authentication session between a controller and the coordinator, the controller can send instructions to the coordinator to control the attachment. The coordinator can establish a pairing authentication session with the attachment and provide corresponding command and control messages to the attachment. The coordinator can receive responses from the attachment and send corresponding responses to the controller. In this configuration, the coordinator can read messages sent by the controller to the attachment and the attachment's responses to the controller. Access to attachments can be controlled by controlling access to the coordinator. For example, an attachment can be configured to pair only with the coordinator. Furthermore, in situations where multiple controllers may simultaneously attempt to control the same attachment, the coordinator can coordinate their actions, such as implementing priority logic to resolve conflicting instructions. In some implementations, the coordinator can also impose access restrictions on each controller or each attachment. A coordinator is not required, but when a coordinator is present, some implementations may require or prefer to communicate with the annexes through the coordinator.

[0019] Specific aspects of the invention may relate to an accessory network, wherein the accessory network can be a set of accessories jointly controlled by the same controller (or a group of controllers, such as a controller network). The accessory network may provide features and options to coordinate control of multiple accessories. For example, an environment model may be defined for the accessory network, wherein accessories are assigned to various locations within the environment model based on their physical location. In some embodiments, the environment model may be a hierarchical representation of a physical environment (e.g., a home), which may include lowest-level objects (e.g., rooms), wherein each accessory is assigned to a lowest-level object within the lowest-level objects (e.g., accessories may be assigned to rooms based on where they are installed or where they spend the majority of their time). Lowest-level objects may be grouped into higher-level objects (e.g., rooms may be grouped into areas within a home). Accessories in the network may be controlled individually or at any hierarchical level of the environment model (e.g., shutting down all accessories in a particular room or area). As a supplement to or alternative to physical-based grouping, the environment model may also include additional logical groupings of accessories, such as “service groups” of accessories that may be used together, and in some embodiments, accessories may be assigned to a physical group and any number (including zero) of logical groups. In some implementations, the environment model may also provide “action groups” where a single triggering event or condition (e.g., a user command or a detectable event, such as time of day) can cause the invocation of functions of several attachments in the network (e.g., turning off the lights or locking the door when the user goes to bed).

[0020] An attachment network may be linked to a controller network, which may be a set of controllers with permissions to access all or part of the attachment network. For example, the attachment network model may include an access list that identifies controllers with permissions to access the attachment network. Different controllers may have different levels of permissions. In some embodiments, the controller network may include a coordinator as described above, which can manage access to specific attachments by specific controllers according to a licensing scheme. Each controller linked to the attachment network may receive a copy of the attachment network model (e.g., through various synchronization and / or data sharing operations). Some controllers may have permissions to edit the attachment network model and / or add or remove other controllers from the access list.

[0021] The following detailed description, together with the accompanying drawings, will provide a better understanding of the nature and advantages of the invention. Attached Figure Description

[0022] Figure 1 A home environment according to an embodiment of the present invention is shown.

[0023] Figure 2 An example of a controller network configuration according to an embodiment of the present invention is shown.

[0024] Figure 3 An example of an environment model for an attachment network according to an embodiment of the present invention is shown.

[0025] Figure 4 A table illustrating examples of service groups that can be defined according to embodiments of the present invention is shown.

[0026] Figure 5 A table is shown listing the options for defining action groups and triggers according to an embodiment of the invention.

[0027] Figure 6 A table is shown listing examples of action groups that can be defined according to embodiments of the present invention.

[0028] Figure 7 This is a flowchart of a process for setting up an environment model according to an embodiment of the present invention.

[0029] Figure 8 This is a table showing an access list for an environment model according to an embodiment of the present invention.

[0030] Figure 9 A direct synchronization technique for transmitting updates to an environment model according to an embodiment of the present invention is shown.

[0031] Figure 10A cloud-based synchronization technique for transmitting updates to an environment model, according to an embodiment of the present invention, is illustrated.

[0032] Figures 11-38 A user interface screen according to various embodiments of the present invention is shown.

[0033] Figure 39 This is a simplified block diagram of a controller according to an embodiment of the present invention.

[0034] Figure 40 This is a simplified block diagram of an appendix according to an embodiment of the present invention. Detailed Implementation

[0035] Exemplary Environment

[0036] Figure 1 A home environment 100 according to an embodiment of the present invention is illustrated. The home environment 100 includes a controller 102 capable of communicating with various accessory devices (also referred to as accessories) located within the environment. The controller 102 may include, for example, a desktop computer, laptop computer, tablet computer, smartphone, wearable computing device, personal digital assistant, or any other computing device or group of devices capable of transmitting command and control messages to the accessories and providing a user interface to allow the user to instruct desired operations on the accessories (e.g., as described in U.S. Provisional Patent Application No. 61 / 935,967 and U.S. Patent Application No. 14 / 614,914 above). In some embodiments, the controller 102 may be implemented using multiple discrete devices. For example, a base station and one or more mobile remote control stations may be present, which communicate with the accessories and may be mounted at a fixed location within the environment 100. The mobile remote control station (e.g., a handheld or wearable device, such as a mobile phone, tablet computer, smartwatch, glasses, etc.) provides a user interface and communicates with the base station to achieve effective control of the accessories. In some embodiments, the base station may act as a coordinator or agent as described below.

[0037] It can control any type of accessory device. Examples of accessory devices include door lock 104, garage door system 106, light fixture 108, security camera 110, and thermostat 112. In some cases, controller 102 can communicate directly with the accessories; for example, controller 102 is shown communicating directly with door lock 104 and garage door system 106. In other cases, controller 102 can communicate via an intermediary. For example, controller 102 is shown communicating via wireless network access point 114 with accessories 108, 110, 112 located on a wireless network provided by access point 114. As described above, in some embodiments, controller 102 may include a base station, and base station functionality may be integrated into access point 114 or into one of the accessories to be controlled (e.g., thermostat 112). In some embodiments, the intermediary may act as a proxy or coordinator as described below.

[0038] It can use various communication transmissions and combinations of transmissions, and can use different transmissions with different devices. For example, some wireless transmissions are published by the Bluetooth SIG. Classic or Smart communication protocols (referred to herein as "Bluetooth" and "Bluetooth LE") support direct point-to-point communication between devices within a limited range. Other wireless transmissions, such as those conforming to... Networking standards and wireless networks, along with protocols published by the Wi-Fi Alliance (referred to herein as "Wi-Fi networks"), can define a wireless network together with a central access point that routes communication between different devices on the network. Furthermore, while wireless communication transmissions are shown, wired transmissions can also be provided for some or all of the accessories. For example, light bulb 108 can be connected to access point 114 via a wired connection, and controller 102 can communicate with light bulb 108 by wirelessly sending messages to access point 114, thus enabling message transmission to light bulb 108 via a wired connection. Other combinations of wired and wireless communication are also possible.

[0039] Furthermore, although one controller 102 is shown, a home environment may have multiple controller devices. For example, each person living in the home may have their own portable device (or apparatus) that can act as a controller for some or all of the accessories 104-112. Different controller devices may be configured to communicate with different subgroups of accessories; for example, a child's controller may be prevented from modifying settings on the thermostat 112, while a parent's controller device may be permitted to modify settings. Such permission or privilege may be configured and controlled, for example, using the techniques described below and in U.S. Provisional Patent Application No. 62 / 005,764, U.S. Provisional Patent Application No. 62 / 094,391, and U.S. Application No. 14 / 725,891 cited above.

[0040] In some implementations, a unified attachment protocol facilitates communication between controller 102 and one or more attachments 104-112. This protocol provides a simple and scalable framework that models attachments as a collection of services, where each service is defined as a set of characteristics, each characteristic having a defined value at any given time. Various characteristics can represent aspects of the attachment's state. For example, for thermostat 112, characteristics could include power (on or off), current temperature, and target temperature. In some implementations, the message format can be transmission-dependent while conforming to the same attachment model. Examples of service- and characteristic-based attachment models are described in U.S. Provisional Patent Application No. 61 / 935,967 and U.S. Patent Application No. 14 / 614,914.

[0041] The protocol may also define the message format for controller 102 to send command and control messages (requests) to attachment 112 (or other attachments) and for attachment 112 to send response messages to controller 102. Command and control messages may allow controller 102 to query the current state of attachment characteristics and, in some cases, allow modification of characteristics (e.g., modifying a power characteristic that can turn an attachment on or off). Therefore, any type of attachment can be controlled by sending appropriate messages, regardless of its functionality or manufacturer. The format may be the same across the various attachments. Examples of the message format are described in the aforementioned U.S. Provisional Patent Application No. 61 / 935,967 and U.S. Patent Application No. 14 / 614,914.

[0042] The agreement may also provide a notification mechanism that allows Annex 112 (or other annexes) to selectively notify controller 102 when the state changes. Multiple mechanisms may be implemented, and controller 102 may register for or subscribe to the most appropriate notification mechanism for a given purpose. Examples of notification mechanisms are described in the aforementioned U.S. Provisional Patent Application No. 61 / 935,967 and U.S. Patent Application No. 14 / 614,914.

[0043] In some implementations, communication with a given accessory may be limited to an authorized controller. The protocol may specify one or more mechanisms (including mechanisms referred to herein as “pairing establishment” and “pairing addition”) for establishing a “pair” between controller 102 and a given accessory (e.g., door lock accessory 104) in an environment that provides a high degree of confidence that controller 102 can control accessory 104. Pairing establishment may include out-of-band information exchange (e.g., a user may enter a numeric or alphanumeric PIN or password provided by accessory 104 into an interface provided by controller 102) to establish a shared secret. The shared secret may be used to support the secure exchange of a “long-term” public key between controller 102 and accessory 104, and each device may store a long-term public key received from the other, making the established pairing sustainable. After pairing is established, controller 102 is considered authorized, and controller 102 and accessory 104 may then communicate or not communicate as needed without losing the established pairing. When controller 102 attempts to communicate with or control annex 104, it may first perform a "pairing verification" procedure to verify the existence of an established pairing (e.g., this would be the case if controller 102 had previously completed a pairing establishment with annex 104). This pairing verification procedure may include each device proving that it possesses a long-term private key corresponding to the long-term public key exchanged during pairing establishment, and may further include establishing a new shared secret or session key to encrypt all communications during the "pairing verification" session (also referred to herein as a verification session). During the pairing verification session, a controller with appropriate privileges may perform a "pairing add" procedure to establish another pairing with the annex on behalf of another controller. Either device may terminate the pairing verification session at any time simply by destroying or invalidating a copy of its session key.

[0044] In some implementations, multiple controllers may pair with the same attachment (e.g., by performing a pairing establishment or by adding a pair with a controller that has previously performed a pairing establishment), and the attachment may accept and respond to communication from any of its paired controllers, while rejecting or ignoring communication from unpaired controllers. Examples of pairing establishment, pairing addition, and pairing verification processes, as well as other examples of security-related operations, are described in U.S. Provisional Patent Application No. 61 / 935,967 and U.S. Patent Application No. 14 / 614,914, cited above.

[0045] It should be understood that the home environment 100 is exemplary, and variations and modifications are possible. Embodiments of the invention can be implemented in any environment where a user expects to control one or more accessory devices using a controller device, including but not limited to homes, automobiles or other vehicles, office buildings, campuses with multiple buildings (e.g., university or corporate parks), etc. Any type of accessory device can be controlled, including but not limited to door locks, door openers, lighting fixtures or systems, switches, power outlets, cameras, environmental control systems (e.g., thermostats and HVAC systems), kitchen appliances (e.g., refrigerators, microwave ovens, dishwashers), other household appliances (e.g., washing machines, dryers, vacuum cleaners), entertainment systems (e.g., TVs, stereo systems), windows, blinds, security systems (e.g., alarms), sensor systems, etc. A single controller can be paired with any number of accessories and can selectively communicate with different accessories at different times. Similarly, a single accessory can be controlled by multiple controllers with which it has been paired. Any function of an accessory can be controlled by modeling its functionality as a service with one or more characteristics and allowing the controller to interact with the service and / or its characteristics (e.g., reading, modifying, receiving updates). Therefore, the protocols and communication processes used in the embodiments of the present invention can be uniformly applied to any context having one or more controllers and one or more attachments, regardless of the functionality of the attachments or the shape factor or specific interface of the controller.

[0046] Figure 2 A network configuration 200 according to an embodiment of the present invention is illustrated. Configuration 200 allows controller 202 to communicate via coordinator 210 with an accessory 204 located in a local environment 206 (e.g., a home environment). Each controller 202 may be an electronic device owned and / or operated by a user who frequently enters and exits environment 206 (e.g., a family member or a frequent visitor to the home). Controllers 202 may each be similar to... Figure 1 The controller 102, and the accessory 204 may be similar to Figure 1 The various attachments shown.

[0047] Annex 204 may each communicate with a coordinator device (or “coordinator”) 210, which may be located in the local environment 206. As used herein, a “coordinator” can be an electronic device capable of operating as a controller for Annex 204 and relaying messages to Annex 204 from other controllers (e.g., controller 202). In some embodiments, coordinator 210 may be a “smart” device capable of coordinating operations between multiple controllers and / or annexes and is not limited to passively relaying messages. Coordinator 210 may be any device capable of presenting itself as a controller to Annex 204 and capable of securely communicating with controller 202. In some embodiments, coordinator 210 may be a device that is expected to remain in the local environment 206 and is expected to be powered on and available for communication most or all of its time. (It should be understood that the coordinator 210 may be unavailable occasionally, for example, in conjunction with software or firmware upgrades, power outages, or other interruption events.) For example, the coordinator 210 may be implemented in a desktop computer, a Wi-Fi or access point unit, a dedicated accessory control base station, a set-top box for a television or other appliance (which may also perform base station functions in addition to interacting with the television or other appliance), or any other electronic device as required.

[0048] In some implementations, coordinator 210 and annex 204 may communicate using a local area network (LAN) such as a Wi-Fi network and / or a point-to-point communication medium such as Bluetooth LE. It should be understood that other communication protocols may be used. In some implementations, controller 202, annex 204, and coordinator 210 may support the unified annex protocol described above, which supports both Wi-Fi and Bluetooth LE as transmission methods.

[0049] exist Figure 2 In the example, controllers 202(1) and 202(4) are currently located in a local environment 206 with attachment 204 and coordinator 210. For example, controller 202(1) may be on the same LAN as attachment 204 and coordinator 210. Controllers 202(2) and 202(3) are simultaneously located outside the local environment 206 but connected to a communication network 208 (e.g., the Internet); such controllers are said to be “remote” from attachment 204 and coordinator 210. It should be understood that controller 202 may be a mobile device that is sometimes within and sometimes outside the local environment 206. Attachment 204 need not be mobile and need not be connected to communication network 208. In some embodiments, coordinator 210 may be connected to communication network 208 and may permit remote controllers 202(2) and 202(3) to remotely access attachment 204.

[0050] In the example shown, controller 202 can communicate with attachment 204 via coordinator 210, and coordinator 210 can be described as acting as a “proxy” for attachment 204. Coordinator 210 can communicate directly with attachments 204(1) and 204(2). With respect to attachment 204(3), coordinator 210 can communicate via “bridge” 212. Bridge 212 can function to relay commands between the controller and the attachment; in some embodiments, bridge 212 can also switch between different communication protocols used by coordinator 210 or controller 202 and attachment 204(3). Furthermore, in some embodiments, bridge 212 can be implemented as a “tunnel” that can provide secure end-to-end communication between coordinator 210 and attachment 204(3). Examples of agents, bridges, and tunnels are described in U.S. Provisional Patent Application No. 62 / 005,764, U.S. Provisional Patent Application No. 62 / 094,391, and U.S. Patent Application No. 14 / 725,891 cited above.

[0051] In network configuration 200, controller 202 can be configured to communicate with accessory 204 via coordinator 210 whenever possible. Thus, as shown, controller 202(1) in local environment 206 communicates with coordinator 210 instead of directly communicating with accessory 204, such as remotely located controllers 202(2) and 202(3). Direct communication between any controller 202 and accessory 204 can be limited, for example, to situations where coordinator 210 is unavailable. In other embodiments, controller 202 can communicate directly with accessory 204 as long as controller 202 and accessory 204 happen to be within each other's range (e.g., on the same Wi-Fi network or within Bluetooth range). For example, as shown, controller 202(4) can communicate directly with accessory 204(2).

[0052] In some implementations, a coordinator 210 may be used to coordinate access to multiple attachments 204 by multiple controllers 202. For example, instead of establishing a pairing between each controller 202 and each attachment 204, each controller 202 may be paired with the coordinator 210, and the coordinator 210 may be paired with each attachment 204. A controller-coordinator pairing may also be established using the same pairing establishment and / or pairing addition process used for establishing controller-attachment pairings, where the coordinator acts as an attachment. For coordinator-attachment pairing to proceed, the coordinator acts as a controller. Therefore, the coordinator 210 may present itself as an attachment when communicating with controllers (e.g., any of the controllers in controllers 202) and as a controller when communicating with attachments (e.g., attachment 204).

[0053] Coordinator 210 facilitates the operation of the ancillary network, including Annex 204. For example, coordinator 210 may maintain an environment model for the ancillary network and provide that model (or portions thereof) to each controller 202; an example of an environment model is described below. Controllers 202 can operate Annex 204 by interacting with coordinator 210.

[0054] In some implementations, coordinator 210 may manage permissions associated with the annex network or environment model to restrict access by a specific controller 202 to some or all annexes 204. In some implementations, controller 202 may prioritize routing all requests to annexes 204 via coordinator 210, and in some implementations, annexes 204 may be configured to communicate only directly with coordinator 210 and ignore requests originating directly from controller 202. This allows coordinator 210 to grant and other restrictions on access to annexes 204.

[0055] Centralizing communication with attachments via coordinator 210 simplifies the management of controller and / or attachment networks (e.g., controllers 202 and attachments 204 in local environment 206). For example, if a new attachment is acquired, it only needs to pair with coordinator 210 to allow all controllers 202 to access it. Similarly, if a new controller 202 is acquired, it only needs to pair with coordinator 210 to allow it to access all attachments 204. In environments with multiple controllers (e.g., a household where members each have multiple devices) and potentially dozens of attachments, the time savings can be considerable.

[0056] It should be noted that in configuration 200, one or more controllers (e.g., controller 202(1)) may be permitted to communicate indirectly (through coordinator 210) rather than directly with one or more accessories (e.g., accessory 204(1)), regardless of whether controller 202(1) is in the local environment 206. This may occur, for example, if controller 202(1) has been paired with coordinator 210 but not directly with accessory 204(1). In some cases, this can provide enhanced security; for example, an accessory paired with coordinator 210 may refuse to establish any other pairings. However, there may be situations where direct access is desired and direct pairings between a particular accessory, such as accessory 204(1), and one or more controllers 202 may be permitted. For example, suppose accessory 204(1) is a door lock and controller 202(1) is a mobile phone. If a direct pairing is established between Attachment 204(1) and Controller 202(1), a user can use Controller 202(1) to lock or unlock Attachment 204(1) via direct communication, thereby locking or unlocking the door. This may be useful, for example, when Coordinator 210 is temporarily unavailable. In some embodiments, Coordinator 210 may be used to instruct Attachment 204(1) which controller in Authorized Controller 202 is authorized for direct access, and Attachment 204(1) may be paired with the authorized controller 202. In some embodiments, Attachment 204(1) may be configured to accept direct communication from Authorized Controller 202 only when Coordinator 210 is unavailable. Thus, a general rule could be that all communication with Attachment 204 goes through Coordinator 210, with exceptions made on a per-attachment and per-controller basis.

[0057] Coordinator 210 may function as an intelligent agent to allow controllers to manipulate attachments, rather than simply relaying messages. For example, coordinator 210 may pair with each controller 202 and with each attachment 204. When controller 202(1) receives, for example, a user request to interact with a specific attachment, such as attachment 204(1), controller 202(1) may establish a first pairing verification session with coordinator 210 and provide its instructions for attachment 204 to coordinator 210 through the first pairing verification session. Coordinator 210 may receive the instructions, establish a second pairing verification session with attachment 204, and send appropriate control messages to attachment 204 through the second pairing verification session. In some embodiments, coordinator 210 may know the content of the instructions, and in some embodiments, the messages sent to attachment 204 do not need to correspond to the instructions provided by controller 202(1). For example, while communicating with controller 202(1), coordinator 210 may also communicate with another controller (e.g., controller 202(2)). Controllers 202(1) and 202(2) may each provide instructions to coordinator 210 regarding attachment 204. Coordinator 210 may analyze the received instructions, for example, to detect and resolve conflicts, in cases where controller 202(1) instructs coordinator 210 to open attachment 204 and controller 202(2) instructs coordinator 210 to close attachment 204. Priority rules or other rules for resolving conflicts may be used (e.g., coordinator 210 may be programmed with "open" taking precedence over "close"; instructions from controllers with administrator privileges taking precedence over instructions from controllers without administrator privileges; etc.). Coordinator 210 may apply priority rules to resolve any conflicts and may send instructions to attachment 204 based on the resolution. Upon receiving a response from attachment 204, coordinator 210 may determine whether to send a corresponding message (or a different message) to controllers 202(1) and / or controller 202(2). As another example, coordinator 210 may enforce permissions established for each controller 202 and / or attachment 204. For example, when one of the controllers in controller 202 sends a request, coordinator 210 may apply decision logic to determine whether the controller 202 sending the request has the appropriate permissions; if not, coordinator 210 may reject the request. The decision logic may be simple or complex as needed; for example, it may restrict which hours or how long of the day a controller belonging to a child can operate a specific accessory (e.g., TV), while a parent's controller may have unrestricted access, or it may restrict a controller associated with a guest (e.g., a babysitter) to operating a specific subset of accessories. Therefore, coordinator 210 is not limited to acting as a passive relay of messages between controllers and accessories, but may actively intervene to resolve conflicting instructions, enforce any restrictions that may exist on the privileges or permissions granted to a particular controller or user, and so on.

[0058] It should be understood that network configuration 200 is exemplary, and variations and modifications are possible. Any number of controllers and any number of attachments can be included in the network configuration. In some embodiments, a proxy can be used to replace coordinator 210, which relays messages between controllers and attachments without reading the message content. In some embodiments, coordinator 210 can be omitted entirely. Some or all attachments 204 may be accessible only within the local environment. Furthermore, as described below, different controllers 202 may have different levels of permission in accessing attachments 204; for example, remote access via network 208 may be permitted for some controllers 202, but not for others.

[0059] Exemplary annex network with coordination control

[0060] As described above, the coordinator 210 can be particularly useful in the context of an automated environment with several controllable attachments. Examples include homes, cars or other vehicles, office buildings, garden areas with multiple buildings, etc. For illustrative purposes, an example of a specific implementation of an attachment network for a home will be described; those skilled in the art who understand this disclosure will appreciate that similar attachment networks can be implemented in other automated environments.

[0061] In one example of an attachment network, each attachment is connected to one or more controllers and can be controlled by sending messages, for example, as described in U.S. Provisional Patent Application No. 61 / 935,967 and U.S. Patent Application No. 14 / 614,914 cited above. This works perfectly for small networks with only a few attachments. However, in some cases, especially as the number of attachments increases, it may be helpful to establish meaningful (to the user) attachment groups that can be managed in a coordinated manner. Therefore, certain embodiments of the present invention can provide an environment model that can be used to coordinate controls among multiple attachments in an attachment network.

[0062] Figure 3 An example of an environment model 300 for a “home” environment according to an embodiment of the present invention is shown. The home environment model 300 may be based on the physical layout of a user’s home. For example, a home typically includes several rooms, and each room may be represented in the home environment model 300 as “room” elements 302(1)-302(10). In this example, room element 302 may include elements that do not actually correspond to rooms in the house, such as “front yard” 302(7) and “back yard” 302(9). In addition, room element 302 may also include “anywhere” element 302(10) that may be used to represent a place in the home but whose location in the home may change or is not defined as an attachment to a room.

[0063] Depending on where the accessories are typically located in the home, each accessory 304(1)-304(24) can be assigned to one of the room elements 302. Thus, for example, the “master bedroom” room element 302(1) may include a fan 304(1), a light 304(2), a TV 304(3), and a music system 304(4). The “kitchen” room element 302(4) may include an oven 304(9), a refrigerator 304(10), a light 304(11), and a fan 304(12). Similarly, other room elements 302 may include other accessories that may be found in those locations, with the front yard room element 302(7) and the backyard room element 302(9) including accessories found externally, such as yard lights 304(19), 304(23) and sprinkler systems 304(20), 304(24). Anywhere component 302 (10) may include accessories that tend to move around the house, such as a vacuum cleaner 304 (25) or an accessory located in an area not defined as a room by the user responsible for maintaining the environmental model 300 (e.g., a small room in a hallway, etc.).

[0064] Furthermore, it may be useful to divide room element 302 into “zone” elements 306(1)-306(3) to reflect the arrangement of rooms in the home. For example, in a house where the bedrooms are upstairs and the common areas are downstairs, “upstairs” zone element 306(1) may include master bedroom element 302(1), Jill’s room element 302(2), and Jack’s room element 302(3), while downstairs zone element 306(2) may include kitchen room element 302(4), family room element 302(5), and foyer room element 302(6). Anywhere element 302(10) may be in its own zone or not assigned to any zone. It is not necessary for every room to be assigned to a zone.

[0065] The Home Environment Model 300 provides a flexible framework for organizing and grouping attachments according to user needs, and Figure 3The examples shown are not intended to limit in any respect. For example, a user may define more or fewer room elements 302 as needed and may assign any attachments 304 to any room element 302 without limiting the number or combination of attachments that can be assigned to a room. Any place element 302 (10) may be a predefined element that is always provided in the model, but the user is not required to assign any attachments 304 to it. Similarly, a user may define more or fewer area elements 306 as needed; each area may correspond to the physical arrangement of rooms or logical groups, and multiple groups of areas may coexist and overlap. For example, area elements 306 may be defined for a “sleep” area and an “activity” area as a complement to or alternative to “upstairs” and “downstairs” areas. In smaller homes (e.g., studios or one-bedroom apartments), users may find defining areas unnecessary, and the use of area elements may be optional. For larger homes or other large environments, it may be desirable to have a structure with more levels of hierarchy than rooms and areas (e.g., a “wing” may have upstairs and downstairs areas), and model 300 may be adjusted accordingly.

[0066] In some implementations, annex 304 in the home environment model 300 can be controlled individually or at the level of a room, area, or the entire house. For example, the user can instruct... Figure 2 The coordinator 210 turns on all the outside lights, thus affecting the lights 304(19), 304(21), and 304(23) “outside” the area element 306(3). Alternatively, the user can instruct the coordinator 210 to turn off all accessories in Jack’s room 302(3), thus affecting Jack’s light 304(18) and fan 304(7).

[0067] In some cases, users may want to coordinate control over annexes that may be assigned to different rooms or areas. Therefore, in some implementations, users can enhance the environment model 300 by grouping the various annexes into "service groups". Figure 4 Table 400 illustrates examples of definable service groups according to embodiments of the present invention. Service groups can coexist with environment model 300, and a single service group can include attachments in any combination of rooms or areas. For example, attachment 304 can be assigned to both a room (which may be located in one or more areas) and one or more service groups. Examples shown by way of illustration are the "Safety Lights" service group 402, the "Ventilation" service group 404, the "Cooking" service group 406, and the "Garage" service group 408. The safety lights group 402 may include a front yard light 304 (19), a backyard light 304 (21), a porch light 304 (18), and a kitchen light 304 (11). Figure 3As shown, these accessories are assigned to four different rooms in two different zones, but the user can issue a command to the safety light group 402 to turn on all the lights in the group. Similarly, the ventilation group 404 can include all the fans in the house, regardless of the room or zone.

[0068] The accessories in a service group can be dissimilar; for example, cooking group 406 may include various accessories that a user expects to turn on while cooking, including kitchen light 304(11), kitchen fan 304(12), and family room TV 304(13). Garage service group 408 may include front yard light 304(19), garage light 304(21), and garage door 304(22). It should also be noted that an accessory may be assigned to multiple service groups; for example, kitchen fan 304(12) may be in both ventilation service group 404 and cooking service group 406.

[0069] In some implementations, a user may instruct the controller to set the state of an entire service group, such as turning the group on or off. The controller may send a corresponding write message to each attachment having a characteristic corresponding to the state change. For example, a user may instruct the controller to turn the garage service group 408 on (or off). The controller may determine that the front yard light 304(19) and the garage light 304(21) have the "on" characteristic and can therefore be turned on (or off), but the garage door 304(22) does not have the "on" characteristic. Therefore, the controller may send write requests to the front yard light 304(19) and the garage light 304(21) to write to the "on" characteristic but not to the garage door 304(22).

[0070] Although not required, using service groups can provide another degree of flexibility when coordinating control over multiple attachments.

[0071] Another model for coordinated control of multiple attachments can be based on "action groups." An action group defines a set of actions to be taken when a "trigger" occurs. In some implementations, triggering an action group can be performed in two phases. In the first phase, a "trigger event" is detected at a controller (e.g., any of the controllers in controller 202 described above) or a coordinator (e.g., coordinator 210 described above). In response to the detection of a trigger event, the controller 202 (or coordinator 210) that detected the event can test whether a "trigger condition" is met. If met, one or more "result actions" can be executed. Therefore, a user can define an action group by specifying a trigger event, a trigger condition, and one or more result actions.

[0072] Figure 5Table 500 illustrates options for defining action groups and triggers according to embodiments of the invention. Column 502 lists examples of trigger events. As used herein, a trigger event can be any event that occurs and can be detected by a controller (e.g., any of the controllers 202 described above) or a coordinator (e.g., coordinator 210 described above). Column 502 lists categories of detectable trigger events. A “geofence” can be defined relative to the location of the local environment 206, such as within half a mile, 1000 feet, etc. A controller 202 (or other user equipment) capable of determining its location (e.g., using a Global Positioning Service (GPS) receiver, etc.) can detect when it enters or leaves the geofence. In some embodiments, controller 202 may test the trigger condition in response to detecting that it has entered or left the geofence; otherwise or alternatively, controller 202 may report the event to coordinator 210, which can test the trigger condition. In some embodiments, more granular geo-triggered events may be supported, such as detecting when a user enters or leaves a specific room in their home.

[0073] An attachment state change event can be detected when the controller or coordinator receives a state change notification from the attachment. The sending of state change notifications from the attachment to the controller can be implemented in a unified attachment protocol, as described in, for example, U.S. Provisional Patent Application No. 61 / 935,967 and U.S. Patent Application No. 14 / 614,914 cited above. (It should be understood that since the coordinator 210 can operate as a controller relative to attachment 204, the coordinator 210 can receive state change notifications from the attachment in the same manner as other controllers.)

[0074] Time / date events can be events defined by a reference clock and / or calendar. Examples of time / date events could include "10:00 PM every day", "8:00 AM Saturday", and so on. Any controller or coordinator that can access clock and / or calendar data to determine the current time and / or date can detect when a time / date event occurs.

[0075] Weather events can be tied to external conditions that may change daily. Examples of weather events include sunrise and sunset, which occur at different times depending on the date and location. Other examples include outdoor temperatures exceeding or falling below thresholds, the start or cessation of rain, changes in atmospheric pressure, wind speeds reaching thresholds, ambient light levels exceeding or falling below thresholds, seismic activity, etc. Controllers or coordinators can use various technologies to detect weather events. For example, a controller or coordinator can periodically poll online weather services to retrieve weather data. Alternatively, weather sensors (such as thermometers, anemometers, barometers, and light sensors) can be installed outside the home, and weather data can be retrieved from these sensors. Sunrise and sunset times can vary with location and date, but are much more predictable than most weather events and can be calculated based on home location and calendar date or obtained from online services.

[0076] A “notification” event can include any type of electronic message signal generated at the source and receivable by the coordinator or controller, regardless of the source. For example, a coordinator or controller may be able to receive emergency broadcast messages (e.g., tornado warnings, tsunami warnings, earthquake activity alerts). As another example, a coordinator may receive notifications from one of the controllers associated with a home (e.g., a notification that a controller user’s schedule has changed); in some implementations, such notifications may be generated by a controller temporarily not in the local environment. As yet another example, a coordinator may receive notifications from a coordinator in a different local environment (e.g., a neighbor’s home, assuming the neighbor has authorized notification sharing); such notifications may include, for example, weather information based on weather sensors installed by the neighbor, security information (e.g., detection of potential break-ins or suspicious activity), etc.

[0077] The presence or absence of a user can be determined using presence sensors in the home and / or by the presence or absence of specific user devices typically carried by each user. For example, coordinator 210 can detect which controllers 202 are in the local environment 206 and infer the presence or absence of the user of controller 202 accordingly.

[0078] In some implementations, there is no anticipated event that can be detected as a triggering event. For example, if a user normally arrives home at 6:00 PM, the user's continued absence at later times (e.g., 8:00 PM or 10:00 PM) is detected as a triggering event.

[0079] In some implementations, such as based on analysis of user behavior at different times, machine learning can be used to define triggering events. For example, such analysis can be performed by coordinator 210 and can be of arbitrary complexity. For instance, coordinator 210 can observe user interactions with various accessories and detect patterns such as "if it rains for more than 10 minutes, the user will turn off the automatic sprinkler." Based on this pattern, coordinator 210 can define "raining for more than 10 minutes" as a triggering event.

[0080] Manual user input can also be a triggering event. For example, a user can explicitly instruct the controller to initiate a specific group of actions.

[0081] When a triggering event is detected, the controller or coordinator that detected the event may test one or more triggering conditions associated with the event. In some implementations, the controller that detected the triggering event may report the event to a coordinator, which may test one or more triggering conditions. Column 504 lists the categories of triggering conditions that can be tested.

[0082] In some implementations, the "always" condition indicates that the resulting action should be executed whenever the triggering event is detected. For example, an action group defined using a manually triggered event could have a trigger condition defined as "always," meaning the user could manually invoke the action group at any time. Trigger conditions based on time / date, weather conditions, and the presence / absence of one or more users can be defined, similar to trigger events of the corresponding category. Thus, for example, detecting a weather event might lead to a test of the time / date condition to determine whether to execute the resulting action.

[0083] Attachment status triggering conditions can be based on the current state of the attachment at the time of the triggering event. For example, in response to detecting a triggering event, the controller or coordinator that detected the triggering event can send a read request to a specific attachment to determine its state. Thus, for example, the triggering event could be a state change of an attachment (e.g., the front door changing from an unlocked state to a locked state), and the triggering condition could be based on the state of another attachment (e.g., whether the garage door is locked).

[0084] Location conditions can be based on the location of the controller, which detects an event and / or the location of another user (or another user's device). For example, the triggering event could be sunset, and the triggering condition could be that the user is not home; the resulting action could be turning on the hallway lights.

[0085] Historical trends can be determined using machine learning. For example, if a user normally turns on specific lights when they arrive home, the trigger condition could be based on whether those lights are on when a trigger event occurs around the geofence where the user enters their home.

[0086] Column 506 lists examples of categories of “result actions” that can be implemented in response to the fulfillment of trigger conditions. One category involves changing the state of an accessory. For example, a controller or coordinator that detects a trigger event can test the trigger conditions associated with the trigger event. If the conditions are met, the controller or coordinator can issue one or more write requests to one or more accessories to change their state. Thus, an action group could be defined, for example: “When the front door state changes from unlocked to locked (trigger event), check the garage door state; if the garage door is unlocked (trigger condition), lock the garage door.” Detecting a change in the front door state can be based on a notification from a front door accessory. The garage door state can be checked by sending a read request to the garage door accessory, and the garage door can be locked by sending a write request to the garage door accessory.

[0087] Another category involves reading and reporting the status of attachments. For example, if a trigger condition is met, a controller or coordinator can issue one or more read requests to one or more attachments to determine their status and can notify the user of the status (e.g., via pop-up or push notifications on the user's device). Therefore, assuming a window in the home is being operated as an attachment, for example, the action group could be defined as: "When it starts raining (trigger event), determine if the user is home; if the user is not home (trigger condition), read the window's status and report the status to the user." The user would then be able to close the window remotely.

[0088] In some implementations, the resulting action may include actions other than interacting with the accessory. For example, as described above, the controller may be implemented on a smartphone or other device that supports functions other than communicating with the accessory. In some implementations, in response to detecting a trigger condition, the controller may invoke other functions of the device. For example, the controller may invoke the device's messaging functions (e.g., email, SMS) to send a message to someone. The controller may also launch an application on the device (e.g., a music player). Thus, for example, an action group may be defined as "when the user arrives home (trigger event), if the speaker system is not playing music (trigger condition), launch the music application and connect it to the speaker system (resulting action)."

[0089] As these examples illustrate, a general form of a trigger action group can be "When (trigger event) occurs, check (trigger condition); if (trigger condition) is true, execute (result action)." Users can define action groups by specifying the trigger event, trigger condition, and result action from the categories in Table 500 or other categories as needed. In various implementations, any level of complexity can be defined for the trigger event and / or trigger condition (e.g., "If it is after dark and it is not Tuesday and there are no users at home, then..."). The same trigger event and / or trigger condition can trigger any number of result actions.

[0090] As a further example, Figure 6 Table 600 illustrates examples of action groups that can be defined according to embodiments of the invention. Each action group may have a user-assigned (or system-assigned) name 602, a trigger event 604, a trigger condition 606, and one or more result actions 608 to be taken (e.g., by sending a control message to an attachment). Although three examples are shown, any number of action groups can be defined, and some action groups may have multiple trigger conditions (e.g., different combinations of trigger events and trigger conditions can trigger result actions for the same group). In some cases, some action groups may be predefined by the attachment management system provider, and users can customize predefined action groups or add other action groups as needed.

[0091] For example, the "Go Home" action group 610 can be triggered based on a geofence trigger event, such as in a controller (e.g., Figure 2 When the controller 202(2) enters a specific radius (e.g., half a mile or 1,000 feet) of the home. The location of the controller 202(2) can be determined using an onboard GPS system or the like, and when its location crosses within a defined geofence, the controller 202(2) can send a message to the coordinator 210 to notify the coordinator 210 that a trigger event has been met. The trigger condition can be a time after sunset. The “returning home” action may include, for example, turning on the light 304(17) in the foyer 302(6), setting the thermostat 304(15) to the desired temperature, and / or other desired actions. Upon receiving a message that the controller 202(2) has crossed the geofence, the coordinator 210 can determine whether the current time is after sunset, and if so, the coordinator 210 can send a control message to take action. In some embodiments, the controller 202(2) can send control messages to the attachments directly or through an agent or other intermediary as needed, and the use of the coordinator 210 is not required.

[0092] The "Watch Movie" action group 612 can be manually triggered, i.e., triggered by expressing a user command. For example, the user can select "Watch Movie" from the activity list presented by the controller, or say a specific phrase, such as "Start watching movie," to the controller's voice control interface. In this case, the trigger condition is "always," so that the action of watching movie is performed whenever the user enters an appropriate command. Actions to be performed in response to the trigger event may include turning off the light 304 (11) in the kitchen 302 (4), dimming the light 304 (14) in the family room 304 (5), turning on the TV 304 (13) in the family room 304 (5), and invoking the movie selection interface of the TV 304 (13). The movie selection interface may be, for example, a selection menu to select a movie stored locally or a movie available from a streaming media source connected to the TV 304 (13). In the latter case, invoking the movie selection interface may include instructing the TV 304 (13) to connect to a streaming media source and obtain information about available movies. (It should be understood that in this context, any type of video content can be treated as a "movie.")

[0093] The "Sleep" action group 614 can be triggered at a specific time each day, such as 10 p.m. or at some other time when family members are normally asleep. In this example, the trigger event is set to 10 p.m., and the trigger condition is set to "Sunday night" (which can be defined to represent the nights from Sunday to Thursday); therefore, the "Sleep" action group will not be automatically triggered at 10 p.m. on Friday or Sunday. Examples of sleep actions may include turning off all the lights in downstairs area 306(2), Jill's room 302(2), and Jack's room 302(4); turning off TV 304(13); dimming the lights in master bedroom 302(1); and starting a "bedtime" music playlist on music player 304(4). Other actions may also be included.

[0094] In some implementations, automatically triggered action groups can also be manually triggered, for example, by the user speaking a command into the controller's voice control interface to select the action group from the controller's graphical user interface, thereby executing a gesture-based gesture detected by the controller, and so on. In some implementations, some or all action groups may have defined "spoken" voice commands; for example, the user might be able to say something like "Goodnight, home" to invoke the sleep action group 614. When the user manually invokes the action group, there is no need to test the triggering event and triggering conditions.

[0095] Figure 6The examples in the text are merely illustrative. Other action groups and triggers are also possible. As mentioned above, in some cases, the triggering event may be based on a state change in some attachments. For example, suppose a household has a sprinkler system and a dishwasher, and expects the sprinkler to run after the dishwasher finishes. The action group used to start the sprinkler could be triggered by a state change of the dishwasher from "cycle in progress" to "cycle complete". A coordinator or other controller may monitor an attachment to detect state changes (other examples of notification processes and controllers for detecting attachment state changes are described in U.S. Provisional Patent Application No. 61 / 935,967 and U.S. Patent Application No. 14 / 614,914 cited above), and may automatically send a message to another attachment when a specific state change is detected.

[0096] It should be understood that in any situation where a user is controlling a coordinated attachment group (e.g., within a room, area, or service group, or using an action group), attachment control can be achieved by using a pairing authentication session between the controller and the attachment to send individual messages to each attachment in the group. Depending on the specific implementation, the controller can be a user-operated controller (if it communicates directly with the controlled attachment), or it can be such as Figure 2 The coordinator of coordinator 210. In the latter case, the user's controller can issue instructions to the coordinator at the granularity of the controlled group (e.g., "turn on the safety lights"), and the coordinator can use table 400 or a similar data structure to determine which attachments should receive control messages. Similarly, the user's controller can issue instructions to the coordinator such as "execute the movie-watching action group", and the coordinator can use table 600 or a similar data structure to determine which attachments should receive control messages.

[0097] In some implementations, users can define multiple action groups with various triggering events and / or triggering conditions. Where an attachment state change is a triggering event for an action group, it's possible that executing one action group can trigger the execution of another action group. This can lead to conflicting or "inconsistent" action groups. For example, a first action group might include a result action that triggers a second action group, which could undo the effect of the first action group, or executing the second action group could re-trigger the first action group (potentially leading to a loop of repeatedly executing action groups).

[0098] To avoid conflicts, some implementations provide a "consistency check" logic component (e.g., within the controller device) to determine whether a newly defined trigger action group conflicts with any previously defined trigger action groups. For example, as part of the environment model, the controller device may access a list of existing trigger action groups. When a user interacts with the controller device's interface to define a new trigger action group, the controller device may perform a consistency check. If the consistency check fails, the controller device may refuse to add the new trigger action group to the list of existing trigger action groups. The user can be notified of the specific reason for the rejection and can correct the problem, thereby allowing the addition of the new trigger action group.

[0099] Various consistency checks can be implemented. For example, the controller device can determine whether the triggering event or triggering condition for a new triggering action group corresponds to the result action of an existing triggering action group. If they do not correspond, the new action group is considered consistent and added to the list of existing triggering action groups. If the triggering event or triggering condition for a new triggering action group does not correspond to the result action of an existing triggering action group, the controller device can determine whether executing the result action of the new triggering action group will trigger any existing action groups. Based on this determination, the controller device can track the effects of those result actions on other existing triggering action groups, thereby constructing a series of triggering action groups; the series can end when the controller finds that its result action does not trigger any other triggering action group. The series can also be constructed in the reverse direction, for example, by determining whether the result action of a new triggering action group corresponds to the triggering event or triggering condition of an existing triggering action group. When constructing a series of triggering action groups, inconsistencies can be detected between any number of links in the series (e.g., executing action group A triggers action group B, which triggers action group C, which in turn triggers action group A again). In some implementations, the controller device can also determine whether the new triggering action group is "internal consistent," for example, whether the result action includes a triggering event. For consistency checks, manually triggered action groups can be ignored, as users can break the "series" by not performing manual triggers.

[0100] Using a suitable interface, users can create environment models (e.g., Figure 3 The user can use a home environment model (300) and add attachments to it. Users can also modify the environment model, for example, by moving attachments to different rooms, defining new rooms, zones, and / or service groups, etc. In the controller (which can be...), Figure 2 An application executing on the coordinator 210 or any controller 202 may provide a user interface screen or other user interface for managing the environment, and instructions received via the user interface may be interpreted as changes to the environment model. Specific examples of user interfaces are described below. The controller may periodically (e.g., when changes are made and / or in response to explicit instructions from the user to save changes) save a copy of the environment model.

[0101] Figure 7 This is a flowchart 700 of a process for setting up a home environment according to an embodiment of the present invention. Process 700 can be implemented, for example, on a controller executing an appropriate application (…). Figure 2 It may be implemented in either controller 202 or coordinator 210. In some implementations, process 700 may be used only when the controller executing the process is in a home local environment.

[0102] At box 702, the user can create a new home environment model, for example, by invoking the controller's "New Environment" function. This can be based on... Figure 3 Environment 300 initializes the environment model using default configurations for rooms and zones. For example, the default could be providing a general "Room 1" located within a general "Zone 1" or simply a "Anywhere" room not assigned to a zone. At box 704, users can define rooms and zones based on their specific household. For example, users can rename one or more general rooms and zones, add new rooms, add new zones, place rooms within zones, etc.

[0103] Once at least one room is defined (either by default or via user input), the user can add attachments to the environment model. For example, at box 706, the user can pair an attachment located somewhere in the home (e.g., by performing a pairing establishment or pairing addition as described above). In some implementations where attachments communicate via a coordinator, if the user does not interact directly with coordinator 210, box 706 may include a transmission to coordinator 210 such that coordinator 210 pairs with the attachment. Once a pairing has been established, the user can add an attachment to the home at box 708. Box 708 may include, for example, the user specifying a room assignment for the attachment. In some implementations, the room assignment may default to the same room as the last added attachment, any room, or some other default room. At box 710, the user can add another attachment (returning to box 706) or end the process (box 712). The user who ends process 700 can later return to the process to add additional attachments. For example, by adding or removing rooms or regions, moving rooms between regions, moving attachments between rooms, defining service groups and / or action groups, etc., the process 700 can be enhanced using other boxes that allow users to modify the environment model.

[0104] Process 700 can be modified as needed, and the order of operations can be changed, and the operations can be repeated. For example, in some implementations, the controller can pair with one or more attachments before creating the environment model. During subsequent environment model creation, the controller can automatically or based on user input add attachments to the environment model. As another example, the controller can pair with several attachments and then add all paired attachments to the environment model in a single operation. Other modifications are also possible.

[0105] It should be understood that the attachment network and environment model described herein are exemplary, and variations and modifications are possible. The environment model is not limited to a home environment but can be extended to any type of environment in which attachments may exist (office buildings, schools, campuses with multiple buildings, urban public infrastructure, etc.). Specific processes and user interfaces for defining and modifying the environment model and for adding attachments to the environment model can be implemented as needed. Examples are described below.

[0106] Exemplary synchronization technology

[0107] Imagine multiple users who can share access to an environment and its accessories. For example, all members of a family can access a home environment and the accessories present therein. It is also imagined that a single user may have multiple devices that they wish to use to operate accessories within the environment. Therefore, it might be desirable for multiple users to access and / or modify the environment model and control the accessories. For example, if a user adds an accessory to the environment, it might be desirable for other users who frequently enter and exit the environment to see that accessory when interacting with the environment model on their devices.

[0108] Therefore, it may be useful to maintain an access list associated with the environment model. This access list could identify individual users and / or devices (including any attachments present in the model) that are permitted to access and / or modify the environment model. The access list could also indicate the type of permission granted to each user (e.g., viewing and / or editing).

[0109] Figure 8 Table 800 is a list of access points provided for an environment model according to an embodiment of the present invention. For illustrative purposes, it is assumed that Table 800 is related to... Figure 3The family environment model 300 is associated and models the family as four individuals residing there: “Mom,” “Dad,” “Jill,” and “Jack.” It is assumed that Mom and Dad are adults, while Jill and Jack are minors. It is also assumed that others may sometimes need access to the family environment model 300. For example, a “nanny” could be someone hired to look after children when adults are not home. For each person (user) with access to the environment, Table 800 may list the user identifier (field 802), the controller device belonging to that user (field 804), and the permissions granted to that user (field 806). As shown, a user can have any number of devices. (Dad has five, Mom and Jill each have two, and Jack and the nanny each have one.) In this example, permissions can be granted based on the user rather than just the device, allowing the user experience to be consistent across devices. Device-based permissions can also be implemented if needed, and combinations of user-based and device-based permissions can be supported. For example, a user with full access permissions (e.g., Figure 8 The father in the story may have devices shared with other users, such as a set-top box for TV (which may be used by the whole family), and the father may not expect other users to have full access to the environment model while they are using the set-top box. Therefore, as... Figure 8 As shown, the set-top box can have different licenses from other devices from Dad.

[0110] In some implementations, the user identifier can be any name or other identifier that a user consistently uses for their controller device. For example, in some implementations, all user controller devices can be linked to a cloud-based data service where the user maintains an account. Through the user's account, the cloud-based data service can provide data storage and retrieval for the user's devices. In some implementations, the cloud-based data service can also provide device management services, such as synchronizing data between user devices (assuming all devices are linked to the same user account), updating device firmware, providing access to applications that can run on the device (and updates to such applications), and so on. When a controller device is linked to a cloud-based data service, the user identifier 802 can be a username (or other user ID) used to access the cloud-based data service. This provides a convenient system of globally unique usernames.

[0111] As shown in Table 800, each user can be assigned a license level. In this example, three license levels are defined. Users with a "Full" license can control attachments in environment model 300, modify environment model 300, and add or remove users from access table 800. Users with a "Family" license can control attachments in environment model 300 and modify environment model 300, but cannot add or remove users. Users with a "Basic" license can control attachments in environment model 300, but cannot change the model or add or remove users.

[0112] Users with “guest” permissions may have more restricted access. For example, a guest permission might restrict access to which attachments can be controlled when they are available, and different combinations of access restrictions can be defined for different levels of “guest” permissions. Thus, for example, a nanny user might have access to all attachments, but only during the period when the nanny is expected to care for the children. Another user, such as a gardener, might only have access to attachments in external area 306(3), allowing the gardener access to gardening tools and sprinkler systems stored in the garage, but not access to the interior of the house. Guest permissions or other forms of restricted access can support fine-grained control over access to attachments in environment model 300. Restrictions can be based on specific attachments, rooms, or areas; time constraints (e.g., specific hours of day, days of week, or date ranges); usage constraints (e.g., restrictions on the number of hours a user can operate the TV each day or week); whether the controller being accessed is in the local environment or far away from it; and so on.

[0113] In some implementations, the permission levels associated with adding and removing users may be independent of the permission levels associated with modifying the environment model. For example, one user may be permitted to do two things, while another user may be permitted to modify the environment model but not add or remove users. It should be noted that the permission levels associated with the environment model may be independent of the privilege levels associated with specific attachments (e.g., administrator and user privileges as described in U.S. Provisional Patent Application No. 62 / 005,764, U.S. Provisional Patent Application No. 62 / 094,391, and U.S. Application No. 14 / 725,891 cited above), and in some implementations, attachments may be uncontrollable except by controllers with appropriate privileges, regardless of any permissions that may be established in Table 800. For example, only the coordinator device may have privileges to operate on a specific attachment; other controllers with appropriate privileges may operate the attachment directly by communicating with the coordinator.

[0114] When the user communicates with the coordinator (e.g., Figure 2In some implementations where the coordinator 210 interacts with attachments to control them, the coordinator 210 may query table 800 to determine whether a user is authorized to perform the requested operation (e.g., control a specific attachment) and may only transmit control messages to the attachment if the user is authorized. In cases where interaction with attachments in environment 300 is mediated by the coordinator 210, assuming the coordinator 210 has privileges to control all attachments in environment 300, the attachment may know or not know which controller is the ultimate source of the control messages received from the coordinator 210.

[0115] When multiple user and / or controller devices share access to an environment model and one controller device makes changes to the model, it may be desirable to propagate those changes to other controller devices that have access to the model. This can be accomplished, at least in part, by utilizing synchronization techniques that propagate data updates between user devices.

[0116] Figure 9 The technique referred to in this paper as "direct" synchronization is illustrated. For illustrative purposes, it is assumed that... Figure 8 The diagram illustrates the users and devices that can access the environment model. As mentioned above, it is further assumed that each user has an account at a cloud-based data service, and all user devices are linked to that account, thereby allowing the synchronization of data objects between user devices.

[0117] Further assuming that the environment model has been modified (Δ) using Dad's laptop 902. That is, Dad's laptop 902 now has a modified environment model 904 (denoted as "HOME+Δ"). The change Δ should be propagated to other devices.

[0118] To propagate change Δ to Dad's other devices—phone 906, tablet 908, coordinator 910, and set-top box 911—the synchronization features of a cloud-based data service can be utilized, provided that the home model 904 is considered as a data object in which the cloud-based data service is included in its synchronization operation. Therefore, when Dad's laptop 902 synchronizes to the cloud-based data service (referred to herein as the "cloud synchronization" process), the cloud-based service will automatically receive the updated home model 904, and the update will be propagated to those devices the next time Dad's phone 906, tablet 908, and coordinator 910 perform a cloud synchronization, as indicated by the dashed arrows. Cloud synchronization can be performed automatically (e.g., in response to a change) or upon explicit request from the user (in this case, Dad).

[0119] The change Δ should also be propagated to the controller devices of other users. In some implementations, the device that is the source of the change (in this case, Dad's laptop 902) can connect to the controller device and transmit the change Δ to each other. However, this can be cumbersome, especially if there are a large number of controller devices.

[0120] To reduce the number of required connections and communications, cloud synchronization between other user devices can be utilized. For example, Dad's laptop 902 can read... Figure 8 The access list 800 identifies one device for each other user, such as Mom's tablet 920, Jill's phone 922, Jack's phone 924, and the nanny's phone 926. Dad's laptop 902 can transmit changes to the identified devices. This communication can use local files, which may or may not require user intervention, to share functionality. In some implementations, cloud-synchronized data services can facilitate locating other users' devices and / or transmitting data to them. Once a change is received on one user device, it can be propagated to the user's other devices via cloud synchronization. Thus, Mom's phone 928 and Jill's laptop 930 can be updated via cloud synchronization, as indicated by the dashed arrows.

[0121] In some implementations, cloud synchronization via cloud-based data services can be used to further simplify the dissemination of changes. Figure 10 A cloud-based technique for transmitting updates to an environment model, according to an embodiment of the present invention, is illustrated. Figure 10 It shows the relationship with Figure 9The same devices. In this case, each device has a connection to a cloud-based data service 1042 via network 1040 (e.g., the Internet). The cloud-based data service 1042 may maintain a repository 1044 (e.g., a database or other data storage device) for environmental information. The repository 1044 may store a “master” copy of the home environment model 300 as a home data bundle 1046. In some implementations, the contents of the home data bundle 1046 may be opaque to the cloud-based data service 1042. For example, the home data bundle 1046 may be encrypted using a key shared among the user devices but not with the cloud-based data service 1042. In one specific implementation, the father (or the father’s device) may define the key and then provide it to the mother, Jack, Jill, and nanny (or their devices) via direct communication. The repository 1044 may also store a “synchronized” list 1048 associated with the home data bundle 1046. Synchronization list 1048 can identify users and devices associated with home data bundle 1046 in a format readable by cloud-based data service 1042, thereby allowing cloud-based data service 1042 to control access to home data bundle 1046. In some implementations, synchronization list 1048 may include access levels for specific users and / or devices. For example, a specific user may be permitted to receive updates to home data bundle 1046, but not to propagate updates to home data bundle 1046. Synchronization list 1048 may be created by devices associated with a fully licensed home environment model (e.g., Dad's laptop 902); in some implementations, synchronization list 1048 may be automatically generated based on access list 800.

[0122] In this example, such as Figure 9 As shown, the father can use his laptop 902 to make changes to the home environment model. That is, the father's laptop 902 now has a modified environment model 904 (represented as "HOME+Δ"). The change Δ should be propagated to other devices.

[0123] In this example, propagation is accomplished via cloud synchronization. For instance, when the father's laptop 902 is synchronized to the cloud, the change Δ can be propagated to the cloud-based data service 1042 and added to the family data bundle 1046. In some implementations, adding the change Δ to the family data bundle 1046 may involve overwriting all family data bundles 1046; in other implementations, the change Δ can be added as, for example, an incremental update instruction with a timestamp.

[0124] When cloud synchronization is performed on another device, such as Mom's tablet 928, the cloud-based data service 1042 can (e.g., from the synchronization list 1048) determine that Mom's tablet 928 should receive an update to the family data bundle 1046. In some implementations, this may involve sending the entire family data bundle 1046 to Mom's tablet 928 if it is more up-to-date than the version of the family environment model 1004 residing in Mom's tablet; in other implementations, an incremental update instruction may be sent (e.g., based on an update received since the last cloud synchronization with Mom's tablet). In either case, Mom's tablet 928 may automatically receive the change Δ via cloud synchronization without communicating directly with Dad's laptop 902. The same applies to the remaining devices in devices 906-930: each device performs cloud synchronization and can receive any updates to the family data bundle 1046, including the change Δ, previously received at the cloud-based data service 1042.

[0125] In some implementations, it may be desirable not to share the entire home data bundle 1046 with all users. Therefore, a set of “domain-specific” data bundles (not shown) can be used instead of a single home data bundle 1046, each encrypted with a different key. Each domain-specific data bundle can be a version of the home data bundle 1046 including information from an environment model associated with a specific user or a specific license group. Each domain-specific data bundle can be identified by its associated user identifier or license identifier. For example, a “full” domain-specific data bundle may include all information about the home model, including information about authorized users and license levels, while a “basic” domain-specific data bundle may omit information about authorized users and license levels (because, as defined above, basic licenses do not include licenses to change user status). A “guest” domain-specific data bundle may include information about accessories that allow guests to operate and omit all other information. For example, if guests are not allowed to operate the master bedroom TV 304(3), that accessory can simply be omitted from the “guest” domain-specific data bundle. As another example, domain-specific data bundles can be generated for each user (e.g., Dad, Mom, Jack, Jill, and Nanny) and may contain information about attachments associated with that user. Each domain-specific data bundle can be encrypted using a key known to the administrator (in this case, Dad; more generally, a user with full permissions) and can be selectively shared with users associated with that domain, for example, through direct communication between Dad (or a device within Dad's devices) and each other user (or their devices). These keys do not need to be known by the cloud-based data service 1042.

[0126] When using domain-specific data bundles, each user's device can be cloud-synchronized with the appropriate domain-specific data bundle. For example, if a domain-specific data bundle is generated for each user, each user's device will be synchronized with that user's domain-specific data bundle. The mapping from user to domain-specific data bundle can be based on the user's account identifier at the cloud-based data service 1042, and the unsynchronized list 1048 can map the user account ID to the correct domain-specific data bundle. If the domain-specific data bundle is generated for each license group, a license group from the synchronization list 1048 can be assigned to each user's account, allowing the user's device to receive the appropriate domain-specific data bundle.

[0127] Using domain-specific data bundles provides further protection against unauthorized use of the attachment. For example, instead of relying on controller 202 or coordinator 210 to implement any appropriate access restrictions, no information about the specific attachment prevents controller 202 from engaging in unauthorized communication with attachment 204. Furthermore, if each domain-specific data bundle is encrypted using a different key, a device receiving a data bundle intended for the "wrong" domain will be unable to decrypt it.

[0128] In some implementations, the family data bundle 1046 may represent an environment model shared by all users in the family. For example, an attachment name (or room name) assigned to an attachment (or room) by the user's father may be propagated to all other users through synchronization. Some implementations may allow users (even users with basic or guest permissions) to define "personal overlays" for the environment model. Personal overlays may include customizations based on user preferences, such as changing the names of attachments, rooms, areas, etc. In some implementations, personal overlays may include aliases for individual elements of the shared environment model (e.g., attachments, rooms, areas), but not any new or different elements. In some implementations, personal overlays may include user-specific, personally defined service groups and / or action groups. Personal overlays may be synchronized between user devices, but not with devices belonging to other users.

[0129] By way of illustration, the environment model represented in the home data bundle 1046 may include a room named "Jill's Bedroom". User Jill may wish to rename this room to "My Room". In some implementations, Jill may do so, for example, via a controller user interface on her phone 922. Jill's phone 922 can then recognize Jill's bedroom as "My Room" in its user interface. Since user Jill does not have permission to modify the environment model, the changes do not propagate to other users' devices. However, they can be synchronized between Jill's devices. For example, while Jill's phone 922 is synchronizing with the cloud-based data service 1042, Jill's phone 922 may send personal coverage data (θ) to the cloud-based data service 1042. The cloud-based data service 1042 may store personal coverage 1050 for each user in association with the home data bundle 1046. Each personal coverage 1050 (or its changes) may propagate to the device of the user who created it, but not to the devices of other users. Therefore, changes to Jill's personal overlay (denoted as +θ) can be propagated to Jill's laptop 930, but not to other controller devices. Similarly, the user mother can create different personal overlays (Π), which can be synced from the mother's tablet 928 to the mother's phone 920 via cloud-based data service 1024 (as shown in Π), but not to other controller devices.

[0130] In addition to or alternative to those mentioned above, other technologies may be implemented for synchronizing home environment models among user devices. In some implementations, cloud synchronization (e.g., ...) can be used for all devices and users with accounts at the cloud-based data service 1042. Figure 10 (as shown), and can be used by users on devices without an account at the cloud-based data service 1042 for direct synchronization (e.g., as shown). Figure 9 (As shown in the diagram). In some implementations, it may be an operational requirement that all users have accounts at the cloud-based data service 1042.

[0131] Exemplary User Interface

[0132] Any of the controller devices described above may provide a user interface to facilitate user access to the automation environment, including control accessories and configuration environments. An example of a graphical user interface will now be described. It should be understood that other user interfaces may be substituted.

[0133] Figure 11An example user interface screen 1100 according to an embodiment of the present invention is shown. Like all user interface screens described herein, screen 1100 may be presented on the display of a controller device. In some embodiments, the user interface screen (including screen 1100 and other screens described herein) may occupy the entire display area (e.g., on a mobile phone or other device with a smaller display); in other embodiments, the user interface screen may occupy a portion of the display area (e.g., a window or panel on a virtual desktop displayed on a desktop or desktop computer). The user interface may incorporate a variety of user-selectable graphical controls to invoke the functions of the application that generated the interface screen. For example, if the user interface is presented on a touchscreen display, the user can touch a control to select it; if the user interface is presented on a non-touchscreen display, the user can operate a pointing device (e.g., a mouse, trackpad, etc.) to position a cursor above a control and then select the control by tapping or clicking. Other input technologies may also be used (e.g., mapping keys on a keyboard to controls, voice input, etc.).

[0134] User interface screen 1100 may be the start screen displayed when a user first launches the application (also referred to as the "application") to configure the environment model and control the automated environment using the environment model. In the examples herein, the automated environment is assumed to be a home, but it should be understood that similar interfaces can be used to configure and control other automated environments. Screen 1100 (and some other screens described herein) provides a "Control" button 1102 and a "Configure" button 1104. Selecting the "Control" button 1102 puts the application into "Control" mode, where the application can receive user input to control any attachments that have been added to the model of the automated environment; selecting the "Configure" button 1104 puts the application into "Configure" mode, where the application can receive user input to configure the model of the automated environment, such as adding attachments; defining environments, rooms, and areas; assigning attachments to rooms; and / or defining action groups, trigger conditions, service groups, etc. In some implementations, this mode may be available from any screen changes presented on the "Control" button 1104 and the "Configure" button 1104 and enable user action.

[0135] Screen 1100 displays the defined automation environment model ( Figure 11 The term "family" in the context of the list 1110 and the list 1110 of the input controller 1112 used to add families to list 1110, thus allowing the user to define multiple families. For example... Figure 11As shown, list 1110 can initially be empty and may require the user to define at least one family before controlling any attachments; therefore, the "Control" button 1102 can be disabled. List 1114 can be used to identify one of the defined families as the "basic" family. The primary family can be used by the controller as the default for routing any attachment control commands it may receive (even when the control and configuration application are inactive or in the background). The user can override the default by specifying a different family when sending attachment control commands to the controller. Figure 11 In this configuration, no families have been defined yet, so list 1114 is empty. Therefore, the user's first task is, for example, to define at least one family by manipulating input control 1112. In some implementations, the user may be prompted to define a family upon first launching the control and configuration application.

[0136] Figure 12 An example user interface screen 1200 for defining a new family is shown. For example, in response to a user action on input control 1112, screen 1200 may be presented as a pop-up window above screen 1100. Screen 1200 may prompt the user to enter the name of the new family in text box 1202. A virtual keyboard (not shown) may be presented to facilitate name input. The user may select control 1204 to add a family to list 1110 or select control 1206 to cancel the action.

[0137] Figure 13 Showing the user in Figure 12 Screen 1200 shows an example of the user interface screen 1100 after the family name is entered. A family entry 1302 named "Apartment" has been added to the family list 1110. A selection control 1304 is provided to allow the user to select a family named "Apartment" for further configuration. Furthermore, the primary family field 1114 now uses an icon 1306 to identify "Apartment" as the primary family. If additional families have been defined, field 1114 can include a list of all defined families, and the user can tap the appropriate list entry to designate the corresponding family as the primary family, and can move the icon 1306 to the designated entry. In some implementations, the first family added by the user to the family list 1110 automatically becomes the primary family and can remain so until the user explicitly selects a different primary family. Designating a primary family allows users to more easily interact with attachments in their most frequently visited environment, while also allowing the same application to support attachment interaction in multiple different environments.

[0138] After defining a family, users can configure it by, for example, tapping or clicking the selection control 1304 to select the family. Figure 14An example of a main configuration user interface screen 1400 that may be presented when a user selects a family to configure ("apartment" in this example). In this example, the family can be modeled, for example, by assigning attachments to rooms defined within the family, as described above. Rooms can then be grouped into areas. Action groups, triggers, and service groups can also be defined to facilitate interaction with the family. Users can also be associated with families and granted various levels of permissions. Therefore, the main configuration screen 1400 may include sections corresponding to each of these configuration options. For example, as... Figure 14 As shown, screen 1400 may provide a section 1402 for adding and / or configuring accessories within the home; a section 1404 for defining and managing rooms; a section 1406 for defining and managing areas; and a section 1408 for adding and managing users to be granted at least some access rights to the home. Additional sections may be provided ( Figure 14 (Not shown in the image) This screen (1400) is used to define and manage action groups, triggers, service groups, etc. In some implementations, screen 1400 can scroll to display additional sections; an example is shown below.

[0139] Initially, such as Figure 14 As shown, the attachment section 1402 can be empty except for the "Add Attachment" control 1412. The "Control" button 1102 is disabled until at least one attachment is added, thus preventing the application from entering control mode until at least one controllable attachment is added to the home. In this example, no attachments are automatically added to the home. Some implementations may support automatic attachment addition, for example, if the controller device is already aware of one or more attachments and / or communicates with them when the controller application is launched. (For example, the controller device may have been previously configured to stream music to a speaker system.) Any known attachments can be automatically added to the home and listed in section 1402. Alternatively, the user may be prompted to add any known attachments to the home. The user can also add attachments by manipulating control 1412.

[0140] An example of a user interface for adding attachments will now be described. In this example, the user can discover attachments to be added to the model, assign them to rooms, and facilitate the pairing process between the controller (or coordinator) and the attachments. In some implementations, the attachment addition process is not considered complete until all these stages are successfully completed. Figure 15 A user interface screen 1500 for finding an accessory to be added, according to an embodiment of the present invention, is shown. For example, in response to user operation... Figure 14The control 1412 can be displayed on screen 1500. For example, when the user operates the control 1412, the controller can begin scanning for beacon signals, announcements, etc., from nearby wireless accessories. Conventional wireless network scanning techniques can be used, including active and / or passive scanning; other techniques for detecting accessories can also be implemented. A status bar 1502 can indicate that the controller is scanning for new accessories. In some embodiments, if the scan is complete, the status bar 1502 can disappear, and user-operable controls can be provided to allow the user to instruct the controller to initiate a new scan.

[0141] List 1504 may present a list of all attachments discovered by the controller. In some embodiments, list 1504 may be automatically populated as the controller continues scanning for attachments. For example, attachments may be listed alphabetically using human-readable names provided in their beacons or notification signals. Attachments may indicate whether they are bridging attachments by including “bridging” in, for example, their attachment names or some other visual indicator (as described in U.S. Provisional Patent Application Nos. 62 / 005,764, 62 / 094,391, and 14 / 725,912 cited above and above). The user may select an attachment to add, for example, by operating the “Select” control 1506 for that attachment or by tapping or clicking the attachment name. The “Done” control 1508 may be selected to end the attachment addition and return to screen 1400.

[0142] If a user selects an attachment to add, for example, via control 1506, the attachment interface screen can guide the user throughout the attachment addition process. As mentioned above, some implementations may require all stages to be completed before the user interface presents the attachment as a component of the home. Figure 16 A user interface screen 1600 for adding attachments according to an embodiment of the present invention is shown. For example, screen 1600 may be presented in response to a user's operation of control 1506 on screen 1500. The name of the selected attachment (which may be the same as the name in list 1504) may be presented in field 1602. The user can assign attachments to rooms by selecting rooms from list 1604. In this example, only one room is defined and selected by default, as shown by the "Selected" icon 1606. If multiple rooms are defined, all rooms may be included in list 1604, and the user can select the desired room from the list, for example, by tapping the room name. In some embodiments, the family model may include default rooms automatically created when the family is defined. In some embodiments, this may correspond to rooms anywhere described above. New attachments may initially be assigned to the default room, and the user can change the assignment by selecting the desired room from list 1604.

[0143] The “Identify” control 1608 can be operated by the user to verify which attachment is being added. In some implementations, when the user operates control 1608, the controller sends a “Display” request to the attachment. The attachment can respond to the “Display” request by taking some action that can be observed by the user. For example, the attachment can flash, make a sound, vibrate, move, and / or perform other actions that can be observed by the user. This allows the user to confirm that the attachment being added is the one the user expected to add. The user can operate the “Add” control 1610 to continue the attachment adding process or operate the “Back” control 1612 to return to screen 1500.

[0144] If a user operates the "Add" control 1610, the controller can communicate with the attachment to determine whether it is certified for use with the automated environment control application (and associated software). For example, the certification status can be determined based on a security certificate obtained by the controller from the attachment. In some implementations, adding attachments can be prohibited if the attachment is not certified. In other implementations, the user can be permitted to add uncertified attachments. If this is the case, the user can be warned of the potential interoperability issue. For example, as... Figure 17 As shown, the warning message 1700 can be displayed above screen 1600 as, for example, a pop-up window. Despite the warning, the user can also choose to add an attachment via operation control 1702 or cancel the operation via operation control 1704. In the latter case, the interface can return to... Figure 15 The screen has 1500 pixels to allow users to select different accessories.

[0145] Assuming the user chooses to continue (or if the attachment is verified), the controller can proceed with the pairing establishment operation with the attachment (e.g., as described above). In some implementations, pairing establishment may require the user to enter a pairing code (e.g., a PIN or password) provided by the attachment. Figure 18A A user interface screen 1800 for manually entering accessory pairing codes according to an embodiment of the present invention is shown. For example, screen 1800 may be presented as a pop-up window above screen 1600. A virtual keypad 1802 may be presented to facilitate input of the pairing code. Numeric, alphanumeric, or other keypads may be presented as needed. The user can input the pairing code by operating the virtual keypad 1802. The user can obtain the pairing code to be entered from the accessory. For example, the pairing code may be printed on or inside the accessory casing, packaging, or instruction manual, anywhere accessible to the user. In some embodiments, if the accessory has display capabilities, the accessory may display its pairing code to the user.

[0146] Figure 18BAnother user interface screen 1850 is shown, illustrating the use of Optical Character Recognition (OCR) to capture accessory pairing codes according to an embodiment of the invention. OCR capture can use the controller's camera to capture an image of a machine-readable pairing code for the accessory. The machine-readable code can be, for example, a barcode, a QR code, a machine-readable alphanumeric code, a dynamic code (e.g., a changing dot pattern on the screen), or any other form used to present information that can be imaged by the camera and analyzed using a computer-implemented algorithm to extract information. The code capture screen 1850 can be presented above screen 1600 as, for example, a pop-up window. A prompt 1852 may prompt the user to point the controller's camera at the machine-readable code (in this example, a barcode). A live preview pane 1854 may present an image of the captured camera data, thereby assisting the user in correctly aligning the image. In some embodiments, the controller may dynamically process the captured preview image and automatically determine when a sufficiently high-quality image has been received. Upon receiving such an image, the controller may modify the UI to indicate success. For example, the image in the live preview pane 1854 may remain static and brighten to indicate success, or the border of the live preview pane 1854 may change color. Other success metrics can also be used to notify the user that they no longer need to aim at the camera. Additional interfaces and technologies can also be used to obtain pairing codes from attachments.

[0147] Regardless of how it is captured, the pairing code provides the controller with confirmation that the attachment being added is the one the user expected to add. Furthermore, the pairing code can be incorporated into the pairing process and used to generate a shared secret between the controller and the attachment. Requiring the user to provide the pairing code out of band enhances security, especially in situations where the pairing code is difficult to guess.

[0148] Assuming the pairing is successfully established, an attachment is added to the family. If an error occurs, the user will be notified and prompted to retry or cancel the attachment addition operation.

[0149] After adding (or canceling) an attachment, the user interface can display an updated attachment list screen 1500, for example, as shown below. Figure 19 As shown in the diagram. Added attachments may have an associated "edit" control 1902 that is visually different from the "select" control 1506 provided for each attachment that has not yet been added. Editing attachments is described below. From screen 1500, the user can select another attachment to add or end the operation by performing the "done" operation 1508.

[0150] If the user interacts with the "Done" control 1508, the user interface can display an updated main configuration screen 1400, for example... Figure 20As shown in the diagram. Attachment 2002 has been added to the home and now appears in the attachment section 1402. The user can operate the "Information" control 2004 to obtain additional information about attachment 2002, and in some embodiments, also edit or control attachment 2002. (In some embodiments, information can be obtained by selecting the attachment name, and a visually distinct "Information" control element is not required.) It should be noted that after one or more attachments have been added to the home, the "Control" button 1102 may become active, allowing the user to switch the application from configuration mode to control mode. An example of operation in control mode is described below.

[0151] In response to a user action on the "Information" control 2004, details about attachment 2002 can be presented. The specific information presented may depend on the available information. In some embodiments, a uniform protocol for controlling attachments in an automated environment can define a standard attachment model. For example, as described in U.S. Provisional Patent Application No. 61 / 935,967 and U.S. Patent Application No. 14 / 614,914 cited above, any attachment can be modeled as a collection of "services," where each service relates to a function that the attachment can perform, and each service can be modeled as a discrete collection of "characteristics," the values ​​of which represent aspects of the current or desired ("target") state of the attachment.

[0152] Figure 21 A user interface screen 2100 according to an embodiment of the present invention is shown. For example, in response to user operation of control 2004 on screen 2000, screen 2100 can be used to present attachment information to the user. In this example, screen 2100 may present a list 2102 of services defined by attachment 2002. The information in list 2102 may be generated based on attachment definition records provided by the attachments, such as those described in U.S. Provisional Patent Application No. 61 / 935,967 and U.S. Patent Application No. 14 / 614,914 cited above.

[0153] Users can select a service from screen 2100 and view the features associated with that service. Figure 22A user interface screen 2200 according to an embodiment of the present invention is shown. Screen 2200 can be used to present information about accessory characteristics. Characteristics are listed in section 2202 and the value (state) of each characteristic is shown. In cases where the user can change the characteristic, a control element can be presented to allow the user to make the change. For example, for a garage door opener service, the user can open or close the door by changing the target door state, and control element 2204 can be provided to select the target door state (open or closed in this example). The current door state shown at 2206 is a read-only characteristic, indicating whether the door is currently open or closed. Characteristics marked "read-only" are read-only with respect to the controller, but can be updated (written) by the accessory based on monitoring its actual state (e.g., whether the door is actually open or closed). In this example, the door is closed.

[0154] In some implementations, a user can control the accessory via screen 2200. For example, a user can operate the "Control" button 1102 to switch the application to control mode without leaving screen 2200. In control mode, control elements 2204 and 2208 (and / or any other control elements that may be present) can become active input controls, allowing the user to change the value of the controller-writable (target) characteristic, which can result in a change in the actual (current) state of the accessory. For example, while the application is in control mode, the user can operate control 2204 to set the value of the "Target Door State" characteristic to "Open" (e.g., by tapping the word "Open") to open the (realistic) garage door represented by this accessory. The controller can send a message to the accessory indicating the new value of the "Target Door State" characteristic. In response, the accessory can determine that the target door state (open) no longer matches the value of the "Current Door State" characteristic (closed, as shown at 2206). The accessory can interpret this as an instruction to open the garage door, making the current state match the target. For example, the accessory can actuate a motorized door opener mechanism to open the door. Once the door is opened, the accessory can update the value of the "Current Door Status" property and send a message to the controller to notify of the update. The controller can then update screen 2206 to display the new "Current Door Status" value. Other behaviors of the accessory can be controlled in a similar manner by reading values ​​from and writing values ​​to the appropriate property. In some implementations, control elements 2204 and 2208 may be active when the application is in control mode and inactive when the application is in configuration mode.

[0155] In configuration mode, the user can return from screen 2200 to the main configuration screen 1400. For example, the user can use the "Back" button 2210 to return to... Figure 21 On screen 2100, then press the "Back" button 2110 to return to... Figure 20 The screen resolution is 1400.

[0156] Refer again Figure 20 In some implementations, it is necessary to assign all attachments in the environment to rooms. Currently defined rooms can be seen in section 1404. To simplify the process for users who do not intend to define multiple rooms, section 1404 may be initially populated with a default room 1414; the default room may be named based on the family name. In some implementations, the default room may correspond to any room described above. If a user defines their rooms, those rooms may also be included in section 1404. Each room may have an associated selection control 1416 available for selecting that room. Operable control 1418 is used to add rooms to the family.

[0157] Figure 23 A user interface screen 2300 for adding rooms to a home according to an embodiment of the present invention is shown. For example, screen 2300 may be presented in response to user operation control 1418. Screen 2300 may be presented as a pop-up window above screen 1400, prompting the user to enter a name for the new room in text box 2302. A virtual keyboard (not shown) may be presented to facilitate name input. The user may select control 2304 to add a room to list 1404, or select control 2306 to cancel the operation.

[0158] Once a room is added to the list in section 1404, users can select a room from section 1404 to view information about it. Figure 24 A screen 2400, which can be presented for newly added rooms according to an embodiment of the present invention, is shown. The screen 2400 can provide a list 2402 of attachments already assigned to the rooms. For newly created rooms, the list 2402 can be empty, as shown.

[0159] In some implementations, users can add attachments to a room by editing the attachments. Figure 25 An interface screen 2500 for editing attachments according to an embodiment of the present invention is shown. For example, it can respond to a user's input... Figure 19 The operation of the "edit" control 1902 on screen 1900 or in response to the user's input from screen 1400 ( Figure 20 Section 1402 selects attachment 2002 to display screen 2500. Screen 2500 can typically be similar to Figure 16Screen 1600. For example, screen 2500 can recognize an attachment to be edited in section 2502. Attachments can be edited by changing their room assignments. For example, section 2504 can present a list of all defined rooms. For example, the currently assigned room can be indicated by a “Selected” icon 2506, and the user can change the selection by, for example, tapping on the desired room. When the user taps on a different room, the “Selected” icon 2506 can be offset to indicate the new room. The “Recognition” control 2508 may be similar to that described above. Figure 16 It works like control 1608. Users can save changes using control 2510 or cancel changes using control 2512. In either case, the user interface can be flipped back to the screen that invoked the editing operation, such as screen 1900 (…). Figure 19 ) or screen 1400 ( Figure 20 ).

[0160] This allows users to add attachments to a room. For example, a user can add an attachment to a room called the "living room". Figure 26 It shows Figure 24 An updated version of the interface screen 2400 is presented after an attachment is added to a room referred to as the "living room". It is not empty; list 2402 now displays the assigned attachments at 2602. Any number of attachments can be added to the room, and list 2402 can be scrolled if the number of attachments exceeds the display area. In some implementations, the user can edit an attachment by selecting it from list 2402, which can result in a call to an editing interface such as screen 2500 described above. For example, if the user sees an attachment incorrectly assigned to the "living room", the user can immediately edit the attachment to change the room assignment. Control 2404 can be used to return to screen 2000.

[0161] Refer again Figure 20 Screen 1400 can also be used to configure areas within the environment. As mentioned above, an area can be a collection of rooms and can be defined based on any physical or logical grouping of rooms that the user deems worthwhile. Defining areas can be optional, and therefore segment 1406 can be initially empty, except for the "Add Area" control 1420. If desired, control 1420 can be manipulated to add areas to the home. Figure 27 A user interface screen 2700 for adding areas to a home according to an embodiment of the present invention is shown. For example, screen 2700 can be presented in response to user operation control 1420. Screen 2700 can be presented as a pop-up window above screen 1400, prompting the user to enter a name for the new area in text box 2702. A virtual keyboard (not shown) can be presented to facilitate name input. The user can select control 2704 to add an area to list 1406, or select control 2706 to cancel the operation.

[0162] Once a region is added, rooms can be assigned to that region. Figure 28 An interface screen 2800 according to an embodiment of the present invention is shown, which can be used to add rooms to areas. For example, screen 2800 may be presented in response to a user entering a name for a new area on screen 2700 or in response to a user selecting an area from segment 1406 of screen 2000. Screen 2800 may display the name of the area and a list 2804 of all rooms currently assigned to that area at field 2802. Operable controls 2806 are used to add rooms to the area. In response to a user selecting control 2806, the user interface may present a list of all rooms (not shown) and indicate whether each room is currently assigned to the area. The user may select rooms to add to and / or remove from the area. In some embodiments, rooms may be assigned to multiple areas. For example, a user may define an "Upstairs" area and a "Bedroom" area. An upstairs bedroom may be in both areas; an upstairs bathroom may be in the "Upstairs" area but not in the "Bedroom" area; a downstairs bedroom may be in the "Bedroom" area but not in the "Upstairs" area. In some embodiments, rooms may be assigned to areas by editing rooms, in addition to or instead of editing areas. For example, a user interface for modifying the assignment of rooms to zones could present a list of rooms and indicate, for each room, which zone it is assigned to (if any).

[0163] Refer again Figure 14 and Figure 20 As mentioned above, screen 1400 is scrollable to present additional configuration options. Figure 29 An example of screen 1400 after scrolling down is shown. Section 2908 allows the user to manage other users associated with the family. For example, section 2908 can display a list of all other users that have been added to the family. This list can initially be empty, or it can only include users of the controller where screen 1400 is located (in...). Figure 29 (I am identified as "me"). The controller user can add another user to the household, for example, via operation control 2910. In some embodiments, in response to user operation control 2910, the user interface may prompt the user to enter an identifier for the user to be added, such as the identifier of the new user's account with the aforementioned cloud-based data service. The interface may also prompt the user to establish permissions for the new user, for example, as described above. In some embodiments, additional interface controls may be used to remove a user.

[0164] Section 2912 can be used to define and manage action groups, where an "action group" refers to a set of control messages to be sent to various attachments as described above. In the examples presented herein, action groups can be defined independently of any automatic triggers, allowing the same action group to be manually invoked and / or associated with one or more triggers. Any defined action groups can be listed in section 2912. This list can be initially empty. The "Add" control 2914 is operable to access the interface used to define action groups.

[0165] Figure 30 An example of a user interface screen 3000 for defining action groups according to an embodiment of the present invention is shown. For example, in response to user operation... Figure 29 Control 2914 can be used to display screen 3000. Field 3002 can be used to name action groups. For example, a user can click or tap within field 3002 and then type a name, for example, using a virtual keyboard (not shown). Action list 3004 identifies all actions already included in an action group. Initially, no actions are included. Attachment list 3006 can include all attachments in the household (in this example, only one attachment, but there could be any number of attachments). The user can select the attachment from which one or more actions in the action group should be performed. A service and features screen, similar to screen 2100 described above, can then be displayed to the user. Figure 21 ) and 2200 ( Figure 22 Or the same as above. Through these screens, users can specify the actions that the attachment should take in conjunction with the action group. For example, for a garage door opener attachment, users can specify the closing (setting the target door state to "closed") and locking (setting the target lock mechanism state to "locked") actions. These actions can be added to the action group. Figure 31 An updated version of the user interface screen 3000 is shown, in which the name field 3002 is populated and some actions are added to the action list 3004. Any number of actions performed by any number of different attachments (or different services of the same attachment) can be included in an action group. The user can save the action group via operation control 3010 or cancel the operation via operation control 3012. In some embodiments, a new action group cannot be saved until the user provides a name and at least one action.

[0166] Once a new action group is defined, the user interface can display an updated version of screen 1400 that includes the action group. Figure 32 An updated version of screen 1400 after adding an action group is shown according to an embodiment of the present invention. Section 2912 has been updated to include the use of screen 3000 ( Figure 31The action group is defined. Users can select information control 3216 to view and / or edit the action group. In some implementations, users can manually invoke the action group, for example, by switching the application to control mode using the "Control" button 1102 and then selecting the action group from section 2912.

[0167] After defining one or more action groups, the trigger section 3220 can become active. Section 3220 can list all defined triggers and may include control 3222 to allow the user to add new triggers. In the simplified example shown, the trigger is a "time" trigger that causes the action group to execute at a user-specified time. It should be understood that other types of triggers can also be defined. Furthermore, in the simplified example shown, a trigger event is defined, and the trigger condition is assumed to be the aforementioned "always". Other interfaces can conveniently define trigger conditions and trigger events.

[0168] Figure 33 A user interface screen 3300 for defining time triggers according to an embodiment of the present invention is shown. For example, screen 3300 may be presented in response to user action control 3222 or another control, indicating that a time trigger has been defined. A trigger can be named by entering text into the name field 3302. A status control 3304 can be used to enable or disable the trigger; an "enabled" trigger will automatically execute its action upon detecting a trigger event, while a "disabled" trigger will not execute its action group regardless of whether a trigger event is detected. A condition section 3322 can be used to add trigger conditions, for example, by operating the add control 332. Operating the add control 3322 may result in the presentation of a pop-up interface (not shown) to define the trigger conditions. An action group list 3306 may present a list of all defined action groups. For each action group to be executed in response to a trigger, an "Included" icon 3308 may be presented along with the action group. For each action group not to be executed, the icon 3308 may be omitted. Users can toggle the included / excluded status of an action group, for example, by tapping the name of the action group to make icon 3308 appear or disappear as needed. In some implementations, no action groups are excluded by default, and users can tap the name of each action group that is not to be executed in response to this trigger.

[0169] The date and time field 3308 can be used to specify the date and time for a time-based trigger event, and the cycle field 3310 can be used to define a cycle pattern for the trigger event. A regular or other date selection interface, such as a date picker, drop-down list, keyboard input, etc., can be used. Although fields 3308 and 3310 in this example are specific to time-based trigger events, those skilled in the art who understand this disclosure will appreciate that other types of trigger events and / or trigger conditions can be defined by providing appropriate fields in the user interface.

[0170] Once a trigger and associated action group are defined, the user can use the "Save" control 3320 to save the defined trigger, or the "Cancel" control 3322 to cancel the operation without saving any changes made. In some implementations, a new trigger cannot be saved until the user provides a name, at least one action group, and a trigger event. After saving a new trigger, the user interface can return to the main configuration screen 1400. Figure 34 This shows the addition of section 3220. Figure 33 An example of the main configuration screen 1400 after a trigger is defined. In this example, the "Enabled" icon 3424 indicates whether the trigger is enabled or disabled, allowing the user to see the status without opening the trigger for viewing or editing. In some implementations, the "Enabled" icon 3424 may be a user-operable toggle switch to enable or disable a specific trigger directly from the interface 1400.

[0171] Screen 1400 can also be used, for example, with segment 3430 to define and manage service groups. As mentioned above, a service group can be a set of attachments or ancillary services that a user may regularly expect to use together. A user can define a new service group, for example, by manipulating control 3432. In response, the user interface can present additional screens to allow the user to define or edit service groups.

[0172] Figure 35 A user interface screen 3500 for creating a service group according to an embodiment of the present invention is shown. For example, in response to user operation control 3432, screen 3500 can be presented as a pop-up window above screen 1400. Screen 3500 may prompt the user to enter a name for the new service group in text box 3502. A virtual keyboard (not shown) may be presented to facilitate name input. The user can select control 3504 to continue the process of creating the service group, or select control 3506 to cancel the operation.

[0173] Figure 36 A user interface screen 3600, according to an embodiment of the invention, is shown that can be presented after a user (e.g., using screen 3500) names a new service group. Operable controls 3602 allow the user to identify the service to be added to the service group, and list 3604 can present a list of services already included in the service group. Initially, list 3604 can be empty. Once a service is added, its name appears in area 3604, and controls on the user interface (not shown) allow the user to view information about the service or remove a service from the service group.

[0174] Figure 37An example of a user interface screen 3700 for adding services to a service group according to an embodiment of the present invention is shown. For example, screen 3700 can be displayed in response to user operation control 3602. Screen 3700 can present a list of all services that have been added to all accessories of the home.

[0175] 3702. In some implementations, no services are included by default, and users can access them from a list.

[0176] 3702 Select a service to add a service. For each service currently included in the service group, an "Included" icon 3704 may be displayed along with the service name. For each service not included, icon 3704 may be omitted. In some embodiments, for services not included, a control element may be provided instead of icon 3704 to allow the user to add a service to the service group (control element 3706) or identify attachments that include the listed services (control element 3708). As described above, identifying attachments may include sending a "display" request from the controller to the attachment, which the attachment may respond by performing some action observable by the user.

[0177] In some implementations, controls 3706 and 3708 may be omitted, and the user can toggle the included / excluded status of a service, for example, by tapping them to make icon 3704 appear or disappear as needed. When the user is satisfied with the included services, the user can operate the "Done" control 3710 to return to screen 1400. In some implementations, new service groups cannot be saved unless they have a name and include at least one service.

[0178] The above-described operations and interface can be used to configure an automation environment. In some implementations, the same application and similar interface may also be used, for example, to control the home by interacting with an accessory. For example, from screen 1400 (e.g., as...) Figure 20 , Figure 29 , Figure 32 and Figure 34 (As shown in either of the above), the user can switch from the "Configuration" mode to the "Control" mode by operating, for example, the "Control" button 1102.

[0179] Figure 38A user interface screen 3800 for controlling an accessory is shown according to an embodiment of the present invention. For example, screen 3800 may be presented in response to a user switching from screen 1400 to control mode. Screen 3800 may include a list 3802 of all services that have been added to all accessories in the home. In the example shown, services are grouped by service type. Other arrangements are also possible. In some embodiments, the arrangement may mimic the physical or logical layout of the home, for example, by showing services grouped by room or area. (Services are a functional model of an accessory and can be considered as being assigned to the same room as the accessory to which they belong.) In some embodiments, controls may be provided to allow the user to select and modify how the list of services is arranged. The user can select an item from list 3802. In response, the application may present a similar display as described above. Figure 22 The control interface of the 2200 screen.

[0180] It should be understood that the user interface screens described herein are exemplary, and variations and modifications are possible. The arrangement and appearance of the interface elements can be changed as needed. Specific sequences of screens can also be altered. Information shown herein on a single screen can be divided across multiple screens, and vice versa. In some embodiments, the list can be expanded and contracted. In some embodiments, the included search or filtering options can be combined with various operations. For example, instead of viewing screen 1500 (… Figure 15 The application can filter the list of detected attachments based on specified criteria, allowing users to provide all or part of the attachment name (or some other characteristic, such as an attachment with a "light bulb" service). Furthermore, when multiple users and / or controllers share access to the environment, it is not necessary to manually configure the environment model on each controller. As mentioned above, the environment model can be synchronized between devices. Therefore, one controller can be used to manually configure the environment model, and other controllers can obtain the environment model (or a portion thereof) from the first controller via direct or cloud-based synchronization.

[0181] In some implementations, voice input may be used for some or all of the operations described herein, and feedback to the user may be provided as needed via visual (e.g., on a display) and / or auditory (e.g., speaking words using a speech synthesizer).

[0182] Exemplary device

[0183] The embodiments described herein can be implemented in electronic devices that are of generally conventional design. Such devices are suitable for conforming to a unified accessory protocol that supports command and control operations by which a controller (first electronic device) controls the operation of an accessory (second electronic device). In some cases, such as with the aforementioned coordinator or agent, the device may combine features or aspects of the controller and the accessory.

[0184] Figure 39 This is a simplified block diagram of a controller 3900 according to an embodiment of the present invention. The controller 3900 may implement any or all of the controller functions, behaviors, and capabilities described herein, as well as other functions, behaviors, and capabilities not explicitly described. The controller 3900 may include a processing subsystem 3910, a storage device 3912, a user interface 3914, a communication interface 3916, a secure storage module 3918, and an encryption logic module 3920. The controller 3900 may also include other components (not explicitly shown), such as a battery, a power controller, and other components operable to provide various enhanced capabilities. In various embodiments, the controller 3900 may be implemented in a desktop computer, laptop computer, tablet computer, smartphone, other mobile phone, wearable computing device, or other system with any desired form factor. Furthermore, as described above, the controller 3900 may be partially implemented in a base station and partially implemented in a mobile unit that communicates with the base station and provides a user interface.

[0185] Storage device 3912 may be implemented, for example, using a disk, flash memory, or any other non-transitory storage medium or combination of media, and may include volatile and / or non-volatile media. In some embodiments, storage device 3912 may store one or more application and / or operating system programs to be executed by processing subsystem 3910, including programs that implement the various operations described above to be performed by the controller. For example, storage device 3912 may store a unified controller application that can read an attachment description record and generate a graphical user interface for controlling attachments based on the information therein (e.g., as described in U.S. Provisional Patent Application No. 61 / 935,967 and U.S. Patent Application No. 14 / 614,914 cited above). In some embodiments, some (or all) of the controller functions described herein may be implemented in the operating system program rather than in the application. In some embodiments, storage device 3912 may also store applications designed for specific attachments or specific categories of attachments (e.g., an IP camera application for managing IP camera attachments, or a security application for interacting with door lock attachments).

[0186] The user interface 3914 may include input devices such as a touchpad, touchscreen, scroll wheel, click wheel, dial pad, button, switch, keypad, microphone, etc.; and output devices such as a video screen, indicator lights, speaker, headphone jack, etc., along with supporting electronics (e.g., digital-to-analog converters or analog-to-digital converters, signal processors, etc.). Users can operate the input devices of the user interface 3914 to invoke the functions of the controller 3900, and can view and / or listen to the output from the controller 3900 via the output devices of the user interface 3914.

[0187] The processing subsystem 3910 may be implemented as one or more integrated circuits, such as one or more single-core or multi-core microprocessors or microcontrollers known in the art. In operation, the processing system 3910 may control the operation of the controller 3900. In various embodiments, the processing subsystem 3910 may execute various programs in response to program code and may maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed may reside in the processing subsystem 3910 and / or a storage medium such as a storage device 3912.

[0188] With appropriate programming, the processing subsystem 3910 can provide various functions to the controller 3900. For example, in some embodiments, the processing subsystem 3910 can implement various processes (or portions thereof) described above as being implemented by the controller. The processing subsystem 3910 can also execute other programs to control other functions of the controller 3900, including applications that can be stored in the storage device 3912. In some embodiments, these applications can interact with the attachment, for example, by generating messages to be sent to the attachment and / or by receiving responses from the attachment. Such interactions can be facilitated by, for example, the attachment management daemon and / or other operating system processes, as described above.

[0189] Communication interface 3916 provides voice and / or data communication capabilities to controller 3900. In some embodiments, communication interface 3916 may include: radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular phone technology, data network technologies such as 3G, 4G / LTE, Wi-Fi, other IEEE 802.11 series standards, or other mobile communication technologies, or any combination thereof), components for short-range wireless communication (e.g., using Bluetooth and / or Bluetooth LE standards, NFC, etc.), and / or other components. In some embodiments, as a supplement to or alternative to the wireless interface, communication interface 3916 may provide wired network connectivity (e.g., Ethernet). Communication interface 3916 may be implemented using a combination of hardware components (e.g., driver circuitry, antenna, modulator / demodulator, encoder / decoder, and other analog signal processing circuitry and / or digital signal processing circuitry) and software components. In some embodiments, communication interface 3916 may use the same or different transmissions, and simultaneously or at different times support multiple communication channels.

[0190] The secure storage module 3918 may be an integrated circuit or the like that securely stores password information for the controller 3900. Examples of information that can be stored in the secure storage module 3918 include the controller's long-term public key and key 3922 (LTPKC, LTSKC mentioned above) and a list of paired attachments 3924 (e.g., a lookup table mapping attachment IDs to the attachment's long-term public key LTPKA for attachments that have completed the above-mentioned pairing establishment or pairing addition process).

[0191] In some implementations, cryptographic operations may be implemented in an encryption logic module 3920 that communicates with the secure storage module 3918. Physically, the encryption logic module 3920 may be implemented, as needed, in the same integrated circuit as the secure storage module 3918 or a different integrated circuit (e.g., the processor in the processing subsystem 3910). The encryption logic module 3920 may include various logic circuits (fixed or programmable as needed) that implement or support cryptographic operations of the controller 3900, including any or all of the aforementioned cryptographic operations. The secure storage module 3918 and / or the encryption logic module 3920 may be presented as a “black box” to the rest of the controller 3900. Thus, for example, the communication interface 3916 may receive messages in an encrypted form that it cannot decrypt and may simply transmit messages to the processing subsystem 3910. The processing subsystem 3910 may also be unable to decrypt the messages, but it may recognize the messages as encrypted and transmit them to the encryption logic module 3920. The encryption logic module 3920 can decrypt messages (e.g., using information retrieved from the secure storage module 3918) and determine which information to return to the processing subsystem 3910. As a result, specific information may only be available within the secure storage module 3918 and the encryption logic module 3920. If the secure storage module 3918 and the encryption logic module 3920 are implemented on a single integrated circuit that executes only code from an internal secure store, this makes information retrieval very difficult, providing a high level of security. Other specific implementations are also possible.

[0192] Figure 40 This is a simplified block diagram of Annex 4000 according to an embodiment of the present invention. Annex 4000 may implement any or all of the Annex functions, behaviors, and capabilities described herein, as well as other functions, behaviors, and capabilities not explicitly described. Annex 4000 may include a storage device 4028, a processing subsystem 4030, a user interface 4032, Annex-specific hardware 4034, a communication interface 4036, a secure storage module 4038, and an encryption logic module 4040. Annex 4000 may also include other components (not explicitly shown), such as a battery, a power controller, and other components operable to provide various enhanced capabilities.

[0193] Annex 4000 represents a broad class of accessories that can be operated by a controller such as controller 3900, and these accessories can vary considerably in terms of capability, complexity, and form factor. Various accessories may include... Figure 40Components not explicitly shown include, but are not limited to, storage devices (disks, flash memory, etc.) with fixed or removable storage media; video screens, speakers, or ports for connecting to external audio / video devices; camera components such as lenses, image sensors, and controls for the image sensor (e.g., aperture, zoom, exposure time, frame rate, etc.); microphones for recording audio (alone or in conjunction with video recording); and so on.

[0194] Storage device 4028 may be implemented, for example, using a disk, flash memory, or any other non-transitory storage medium or combination of media, and may include volatile and / or non-volatile media. In some embodiments, storage device 4028 may store one or more programs (e.g., firmware) to be executed by processing subsystem 4030, including programs that implement various operations described above to be performed by the attachment and operations related to specific attachment behavior. Storage device 4028 may also store attachment objects or attachment definition records, such as those available to the controller device during device discovery, as described in U.S. Provisional Patent Application No. 61 / 935,967 cited above. Storage device 4028 may also store attachment status information and any other data that may be used during the operation of attachment 4000.

[0195] Processing subsystem 4030 may include, for example, one or more single-core or multi-core microprocessors and / or microcontrollers that execute program code to perform various functions associated with accessory 4000. For example, processing subsystem 4030 may implement the various processes (or portions thereof) described above as implemented by the accessory, for example, by executing program code stored in storage device 4028. Processing subsystem 4030 may also execute other programs to control other functions of accessory 4030. In some cases, programs executed by processing subsystem 4030 may interact with a controller (e.g., controller 3900), for example, by generating messages to be sent to and / or receiving messages from the controller.

[0196] User interface 4032 may include user-operable input devices such as touchpads, touchscreens, scroll wheels, click wheels, dial pads, buttons, switches, keypads, microphones, etc.; and output devices such as video screens, indicator lights, speakers, headphone jacks, etc., along with supporting electronics (e.g., digital-to-analog converters or analog-to-digital converters, signal processors, etc.). Depending on the specific implementation of accessory 4000, the user can operate the input devices of user interface 4032 to invoke the functions of accessory 4000, and can view and / or listen to the output from accessory 4000 via the output devices of user interface 4032. Some accessories may provide a minimal user interface or no user interface at all. In the absence of a user interface in an accessory, the user can still interact with the accessory using a controller (e.g., controller 3900).

[0197] The accessory-specific hardware 4034 may include any other components that may be present in the accessory 4000 to enable its functionality. For example, in various embodiments, the accessory-specific hardware 4034 may include one or more storage devices using fixed or removable storage media; a GPS receiver; power and / or power management circuitry; a camera; a microphone; one or more actuators; control switches; environmental sensors (e.g., temperature sensors, pressure sensors, accelerometers, chemical sensors, etc.), and so on. It should be understood that any type of accessory functionality can be supported by providing appropriate accessory-specific hardware 4034, and the accessory-specific hardware may include mechanical components as well as electrical or electronic components.

[0198] Communication interface 4036 may provide voice and / or data communication capabilities for annex 4000. In some embodiments, communication interface 4036 may include: radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular phone technology, data network technologies such as 3G, 4G / LTE, Wi-Fi, other IEEE 802.11 series standards, or other mobile communication technologies, or any combination thereof), components for short-range wireless communication (e.g., using Bluetooth and / or Bluetooth LE standards, NFC, etc.), and / or other components. In some embodiments, as a supplement to or alternative to the wireless interface, communication interface 4036 may provide wired network connectivity (e.g., Ethernet). Communication interface 4036 may be implemented using a combination of hardware components (e.g., driver circuitry, antenna, modulator / demodulator, encoder / decoder, and other analog signal processing circuitry and / or digital signal processing circuitry) and software components. In some embodiments, communication interface 4036 may use the same or different transmissions, and simultaneously or at different times support multiple communication channels.

[0199] The secure storage module 4038 may be an integrated circuit or similar device capable of securely storing password information for the attachment 4000. Examples of information that can be stored in the secure storage module 4038 include the attachment's long-term public key and key 4042 (LTPKA, LTSKA mentioned above) and a list of pairing controllers 4044 (e.g., a lookup table mapping controller IDs to the controller's long-term public key LTPKC for controllers that have completed the pairing establishment or pairing addition process described above). In some embodiments, the secure storage module 4038 may be omitted; the pairing controller's key and list may be stored in storage device 4028.

[0200] In some implementations, cryptographic operations may be implemented in an encryption logic module 4040 that communicates with the secure storage module 4038. Physically, the encryption logic module 4040 may be implemented, as needed, in the same integrated circuit as the secure storage module 4038 or a different integrated circuit (e.g., the processor in the processing subsystem 4030). The encryption logic module 4040 may include various logic circuits (fixed or programmable as needed) that implement or support the cryptographic operations of Annex 4000, including any or all of the cryptographic operations described above. The secure storage module 4038 and / or the encryption logic module 4040 may be presented as a “black box” with respect to the rest of Annex 4000. Thus, for example, the communication interface 4036 may receive messages in an encrypted form that it cannot decrypt and may simply transmit messages to the processing subsystem 4030. The processing subsystem 4030 may also be unable to decrypt the messages, but it may recognize the messages as encrypted and transmit them to the encryption logic module 4040. Encryption logic module 4040 can decrypt messages (e.g., using information retrieved from secure storage module 4038) and determine which information to return to processing subsystem 4030. As a result, specific information may only be available within secure storage module 4038 and encryption logic module 4040. If secure storage module 4038 and encryption logic module 4040 are implemented on a single integrated circuit that executes only code from an internal secure store, this makes information retrieval very difficult, providing a high level of security. Other implementations are also possible.

[0201] Annex 4000 can be any electronic device that interacts with controller 3900. In some embodiments, controller 3900 can provide remote control via operation of annex 4000 as described above. For example, controller 3900 can provide annex 4000 with a remote user interface that may include input and output controls (e.g., a display screen showing current status information obtained from annex 4000, and input controls such as a touch screen overlay for allowing changes to the status information). In various embodiments, controller 3900 can control any function of annex 4000 and can also receive data from annex 4000.

[0202] It should be understood that the system configurations and components described herein are exemplary, and variations and modifications are possible. It should be understood that specific implementations of controller 3900 can perform all the operations described above as being performed by the controller, and specific implementations of accessory 4000 can perform any or all of the operations described above as being performed by the accessory. Agents, bridges, tunnels, or coordinators may combine components of controller 3900 and accessory 4000 using the same or different hardware as needed. Controllers and / or accessories may have other capabilities not explicitly described herein (e.g., mobile phone, Global Positioning System (GPS), broadband data communications, Internet connectivity, etc.). Depending on the specific implementation, these devices may be interoperable to provide any functionality supported by either (or both) device or to provide functionality partially implemented in each device. In some implementations, a particular accessory may have some functionality that is not accessible or invoked through a particular controller but is accessible through another controller or through direct interaction with the accessory.

[0203] Furthermore, although the controller and accessories are described herein with reference to specific blocks, it should be understood that these blocks are defined for descriptive convenience and are not intended to imply a specific physical arrangement of component parts. Moreover, blocks do not necessarily correspond to physically different components. For example, by programming the processor or providing appropriate control circuitry, and depending on how the initial configuration is obtained, the blocks can be configured to perform various operations. Embodiments of the invention can be implemented in a variety of devices, including electronic devices implemented using any combination of circuitry and software.

[0204] Other implementation plans

[0205] Although the invention has been described in conjunction with specific embodiments, those skilled in the art will recognize that many modifications are possible. Controller networks and / or accessory networks may include as many or as few devices as desired. The use of agents or coordinators is not required; regardless of the number of accessories or controllers, (at least in principle) pairing can always be established between each controller and each accessory, enabling all controllers to operate by directly controlling the accessories. With an accessory network model provided (e.g., the environment model described above), each controller can obtain a copy of the model (e.g., via the synchronization described above) and can provide access to the model through its user interface.

[0206] Furthermore, while it's possible, but not necessarily required, for all controllers to be permitted access to all attachments through the agent or controller, even when an agent or controller is present. Some controllers may be restricted from accessing attachments when not in a local environment, and some attachments may require controllers to access them directly rather than through an agent or coordinator.

[0207] In some implementations, the environment model or other model of the attachment network may include the identification of one or more controller devices as preferred (or licensed) agents or coordinators. In some cases, multiple coordinators and / or agents may be specified. For example, Figure 8 The access list may include flags or other status indicators to identify devices with coordinator and / or agent functionality. In cases where the annex network model includes agent or coordinator identification, this facilitates the selection of the agent or coordinator by another controller (e.g., in...). Figure 6 (During the process shown).

[0208] It should also be understood that, in conjunction with the appendix network access list, for example... Figure 8 The access list shown may, but does not necessarily, eliminate the need for controllers to pair with attachments. For example, in some of the embodiments described above, all communication with attachments may be mediated by the coordinator. If this is the case, other controllers may need to pair with the coordinator but not with individual attachments. In some embodiments, controller pairing with the coordinator may be a prerequisite for being added to the access list used for the attachment network.

[0209] Furthermore, some implementations manage security on a per-user basis rather than a per-controller basis. For example, in the above implementations, each controller may have its own long-term public key and long-term private key, independent of any other controller. In other implementations, long-term public key / private key pairs may be assigned to users (rather than specific controllers) and shared among all controllers belonging to that user. For example, long-term public key / private key pairs may be associated with a user account on a cloud-based data service and propagated to devices where the user has linked to the account. Appropriate measures can be implemented to securely propagate the key pairs. When managing long-term keys based on users rather than controllers, an attachment (or coordinator) may pair with users rather than controllers; subsequently, the attachment may accept messages from any controller device presenting the user's identifier and sufficient evidence that it possesses the user's long-term private key.

[0210] Embodiments of the present invention can be implemented using any combination of dedicated components and / or programmable processors and / or other programmable devices. The various processes described herein can be implemented on the same or different processors in any combination. Where components are described as being configured to perform certain operations, such configurations can be implemented, for example, by designing electronic circuits to perform operations, by programming programmable electronic circuits (such as microprocessors) to perform operations, or any combination thereof. Furthermore, although the above embodiments may refer to specific hardware and software components, those skilled in the art will understand that different combinations of hardware and / or software components may also be used, and specific operations described as implemented in hardware may also be implemented in software, or vice versa.

[0211] Computer programs incorporating various features of this invention can be encoded and stored on a variety of computer-readable storage media; suitable media include magnetic disks or magnetic tapes, optical storage media such as optical discs (CDs) or DVDs (Digital Versatile Optical Discs), flash memory, and other non-transitory media. (It should be understood that “storage” of data is different from data propagation using transient media such as carrier waves.) The computer-readable medium encoded with program code can be packaged together with a compatible electronic device, or the program code can be provided independently of the electronic device (e.g., via Internet download or as a separately packaged computer-readable storage medium).

[0212] Therefore, although the invention has been described with respect to specific embodiments, it should be understood that the invention is intended to cover all modifications and equivalents within the scope of the following claims.

Claims

1. A method for accessory management, comprising: identifying, by a controller device, a plurality of accessories in a physical environment; generating, by the controller device, an action group, the action group including at least a first accessory and a second accessory of the plurality of accessories, the action group further including a first resultant state of the first accessory and a second resultant state of the second accessory; identifying, by the controller device, a trigger event and a trigger condition associated with the action group, wherein the trigger event includes a detectable occurrence event corresponding to a location of a device, a state of an accessory, or a time or date, wherein the trigger condition includes a condition corresponding to the detectable occurrence event, and wherein an occurrence of the trigger event and corresponding trigger condition are configured to trigger the action group; detecting, by the controller device, an occurrence of the trigger event, wherein the trigger event is based on a state change of a first accessory; in response to detecting the trigger event: testing, by the controller device, whether the trigger condition is satisfied, wherein testing whether the trigger condition is satisfied includes sending a read request to the second accessory that determines a state of the second accessory; and in accordance with the trigger condition being satisfied: sending a second instruction to the second accessory that transitions from a second current state to the second resultant state.

2. The method of claim 1, wherein, generating the action group based at least in part on input received at a user interface.

3. The method of any one of claims 1 or 2, wherein, identifying the trigger event based at least in part on input received at a user interface.

4. The method of claim 1, wherein, detecting the occurrence of the trigger event is based at least in part on receiving information from a coordinator device, wherein the information identifies the trigger event.

5. The method of claim 1, further comprising identifying a trigger condition associated with the action group, a positive outcome of the trigger condition being configured to trigger the action group.

6. The method of claim 5, further comprising determining a positive outcome of the trigger condition, and wherein the transmission of the second instruction is further in accordance with the determination of the positive outcome of the trigger condition.

7. The method of any one of claims 5 or 6, wherein, identifying the trigger condition based at least in part on input received at a user interface.

8. The method of claim 6, wherein, the determination of the positive outcome of the trigger condition is based at least in part on receiving information from a coordinator device, wherein the information identifies the trigger event.

9. The method of claim 1, further comprising establishing, by the controller device, a pairing with each of the plurality of accessories, wherein establishing a pairing with an accessory includes providing a long-term public key of the controller device to the accessory and obtaining a long-term public key of the accessory from the accessory.

10. The method of claim 1 or 9, wherein, the plurality of accessories are individually controllable based at least in part on a comparison between a current state and a target state.

11. The method of claim 10, wherein, the target state includes at least one of the first resultant state or the second resultant state.

12. The method of claim 1, wherein, the physical environment includes a plurality of rooms and a plurality of zones.

13. The method of claim 12, wherein, each zone contains a subset of the plurality of rooms.

14. The method of claim 13, wherein, accessories assigned to a common room or a common zone in an environment model are controllable as a group.

15. The method of any of claims 13 or 14, further comprising generating an accessory service group that includes accessories in different rooms of the plurality of rooms.

16. The method of claim 15, further comprising generating an attachment service packet comprising the first attachment and the second attachment.

17. The method of claim 15, wherein, the attachment service packet comprises the group of actions.

18. The method of claim 13, wherein, the first result state and the second result state are the same state.

19. A computer-readable storage medium having stored thereon program instructions that, when executed by a processor in a controller device, cause the controller device to perform the method of any one of claims 1-18.

20. A controller device comprising: a computer-readable storage medium having stored thereon program instructions; and a processor coupled to the computer-readable storage medium and configured to execute the program instructions to cause the controller device to perform the method of any one of claims 1-18.

21. A system for attachment management comprising means for performing the method of any one of claims 1-18.

Images