Home automation (HA) system and related methods including desired scenario implementation based on a user-selectable list of addressable HA devices

Abstract

The present disclosure relates to a home automation system, which may include an addressable device and a device scenario controller. The controller may obtain a first desired scenario including a first trigger action and a response event, and present a first user-selectable list of addressable devices that can implement the first desired scenario. The controller may determine the addressable device selected by the first user, and use the addressable device selected by the first user to perform a first response event when the first trigger event occurs. The controller may obtain a second desired scenario including a second trigger action and a response event from the cloud, present a second user-selectable list of corresponding addressable devices that can implement the second desired scenario, determine the addressable device selected by the second user, and use the addressable device selected by the second user to perform a second response event when the second trigger event occurs. When the addressable device cannot implement the scenario, the controller may present a purchase offer for additional addressable devices.

Description

Technical Field

[0001] The present embodiments are directed to the field of electronics, and more particularly to home automation systems and related methods. Background Art

[0002] There are many home automation systems and practices that attempt to allow automatic control of electrical devices within a house. Home automation is becoming more and more popular due to the increasing availability of smartphones and tablets. As pointed out in the article "The Problem With Home Automation's Internet Of Things (IoT)" published in Forbes on September 26, 2013, home automation is usually an expensive system for wealthy consumers to control lights, home theater, security, air conditioning and home audio. This market has expanded, and there are now many do-it-yourself (DIY) products available, and although the products are useful, they can be difficult to aggregate. In other words, as explained in the article, if a consumer purchases a Nest thermostat, a Kwikset door lock, a Phillips Hue lighting device, a Lutron light switch, a Sonos audio system, and a Belkin wireless plug, then difficulties arise. The consumer will need multiple applications, each of which requires time to set up, learn, and use. In addition, the article points out that there is no easy way to get the devices to work together, such as if the consumer wants to use one device to trigger an event based on another event from another device.

[0003] Multiple communication protocols can also be problematic. In particular, different devices may operate using different communication protocols, such as Wifi, Zigbee, Zwave, Insteon, Itron, RadioRA2, and others. This can bring additional difficulties to home automation.

[0004] One approach to addressing these shortcomings is for consumers (which can include users and / or businesses) to use a service and device aggregator that provides an application and integrated wireless adapter unit. Users would sign up with such a provider for many years. Unfortunately, as noted in the article, consumers may not benefit from state-of-the-art hardware and software.

[0005] As noted in the Forbes article, another approach is to provide a single application that attempts to integrate different applications and integrate wireless adapters, such as by using each of the different communication protocols. Further improvements to the operation and integration of devices may be desirable.

[0006] U.S. Patent No. 8,490,006 issued to Reeser et al. relates to scenario creation for building automation systems. More specifically, Reeser et al. disclose a home automation system that logs the use of automation devices such as lights, appliances, door and window openings, motion, etc., and suggests home automation scenarios based on analysis of the logs. The system device inventory can be compared to scenarios used by other systems with similar device inventories to develop suggested scenarios. A server can analyze insurance claim data and police reports to suggest scenarios that, from the perspective of the system inventory, can help minimize specific losses.

[0007] U.S. Patent Application Publication No. 2015 / 0160797 to Shearer et al. relates to a system for rule-based automation and notification. More specifically, Shearer et al. disclose a graphical user interface that can display a list of rules associated with general automation rules and context-related automation rules. The list items can be associated with suggested automation rules, notification rules, or both. The list of rule items can include items associated with attribute contexts or external condition contexts. Each item can be included. Summary of the invention

[0008] A home automation (HA) system may include a plurality of addressable HA devices at a given location and a HA device scenario controller. The HA device scenario controller may be configured to obtain a first desired scenario including at least one first trigger action and at least one first response event from a user, and present a first user-selectable list of corresponding addressable HA devices in the addressable HA devices that can implement the first desired scenario. The HA scenario controller may also be configured to determine the first user-selected addressable HA device in the addressable HA devices, and upon the occurrence of at least one first trigger event, use the first user-selected addressable HA device in the addressable HA devices to perform at least one first response event, thereby implementing the first desired scenario. The HA device scenario controller may also be configured to obtain a second desired scenario including at least one second trigger action and at least one second response event from the cloud, present a second user-selectable list of corresponding addressable HA devices in the addressable HA devices that can implement the second desired scenario, determine the second user-selected addressable HA device in the addressable HA devices, and upon the occurrence of at least one second trigger event, use the second user-selected addressable HA device in the addressable HA devices to perform at least one second response event, thereby implementing the second desired scenario. Thus, for example, a scene can be established without initial knowledge of the addressable devices used to implement the scene, and a scene can be shared using different addressable HA devices.

[0009] For example, the HA device scenario controller may be configured to obtain at least one first and second trigger action without obtaining a corresponding addressable HA device in the addressable HA devices for implementing the at least one first and second trigger action. The HA device scenario controller may be configured to obtain at least one first and second response event without obtaining a corresponding addressable HA device in the addressable HA devices for implementing the at least one first and second response event.

[0010] The HA system may further include a user interface device coupled to the HA device scene controller. The user interface device may include a display, a user input device, and a user interface controller coupled to the display and the user input device. The user interface controller may be configured to display the first and second user selectable lists on the display, and allow the user to select the first and second user selected addressable devices in the addressable devices via the user input device.

[0011] For example, the HA device scene controller may be configured to generate a notification for the user when at least one first or second trigger action occurs.For example, the device scene controller may be configured to wirelessly obtain at least one first and second trigger action and at least one first and second response event.

[0012] The HA device scene controller may be configured to wirelessly implement the first and second desired scenes.For example, each of the plurality of addressable HA devices may include one of a motion detector, a thermostat, a light switch, an audio controller, a door lock, and a camera.

[0013] On the other hand, for an HA system, the system may include multiple addressable HA devices at a given location and an HA device scenario controller. The HA device scenario controller is configured to obtain a first desired scenario including at least one first trigger action and at least one first response event from a user, and determine when the multiple addressable HA devices at the given location are unable to implement the scenario, and present a purchase offer for additional addressable HA devices. The HA device controller is also configured to determine when the multiple addressable HA devices at the given location are able to implement the scenario, and present a first user-selectable list of corresponding addressable HA devices in the addressable HA devices that can implement the first desired scenario. The HA device controller is also configured to determine the addressable HA device selected by the first user in the addressable HA devices, and when at least one first trigger event occurs, use the addressable HA device selected by the first user in the addressable HA devices to execute at least one first response event, thereby implementing the first desired scenario.

[0014] One method aspect is directed to a method for implementing first and second desired scenarios in an HA system including a plurality of addressable HA devices at a given location. The method may include using an HA device scenario controller to obtain a first desired scenario including at least one first trigger action and at least one first response event from a user, and presenting a first user-selectable list of corresponding addressable HA devices in the addressable HA devices that can implement the first desired scenario. The HA scenario controller is also used to determine the addressable HA device selected by the first user in the addressable HA devices, and when at least one first trigger event occurs, use the addressable HA device selected by the first user in the addressable HA devices to perform at least one first response event, thereby implementing the first desired scenario. The HA scenario controller is also used to obtain a second desired scenario including at least one second trigger action and at least one second response event from the cloud, present a second user-selectable list of corresponding addressable HA devices in the addressable HA devices that can implement the second desired scenario, determine the addressable HA device selected by the second user in the addressable HA devices, and when at least one second trigger event occurs, use the addressable HA device selected by the second user in the addressable HA devices to perform at least one second response event, thereby implementing the second desired scenario.

[0015] Another method aspect is directed to a method for implementing a first desired scenario in an HA system including multiple addressable HA devices at a given location. The method may include using an HA device scenario controller to obtain a first desired scenario including at least one first trigger action and at least one first response event from a user, and determining when multiple addressable HA devices at a given location are unable to implement the scenario, and presenting a purchase offer for additional addressable HA devices. The HA device controller may also be used to determine when multiple addressable HA devices at a given location are able to implement the scenario, and present a first user-selectable list of corresponding addressable HA devices in the addressable HA devices that are able to implement the first desired scenario. The HA scenario controller may also be used to determine a first user-selected addressable HA device in the addressable HA devices, and when at least one first trigger event occurs, use the first user-selected addressable HA device in the addressable HA devices to perform at least one first response event, thereby implementing the first desired scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A is a schematic diagram of an electronic device integration system according to an embodiment of the present invention.

[0017] Figure 1B is a schematic diagram of a HA system according to an embodiment.

[0018] Figure 2A It is used in Figure 1A A schematic block diagram of a message queue in a system.

[0019] Figure 2B is a schematic block diagram of a HA system including a message queue according to an embodiment.

[0020] Figure 3 It is used in Figure 1A Schematic diagram of the action server in the system.

[0021] Figure 4 It is used in Figure 1A Schematic diagram of the operation of the analysis server in the system.

[0022] Figure 5 It is used in Figure 1A Schematic diagram of the camera server in the system.

[0023] Figure 6 It is used in Figure 1A Schematic diagram of the configuration server in the system.

[0024] Figure 7 It is used in Figure 1A Schematic diagram of the debugging server in the system.

[0025] Fig. 8A It is used in Figure 1A Schematic diagram of a discovery server in a system.

[0026] Figure 8B yes Fig. 8A Another diagram of the discovery server.

[0027] Fig. 9 It is used in Figure 1A Schematic diagram of a notification server in a system.

[0028] Fig.10 It is used in Figure 1A Schematic diagram of a loader server in a system.

[0029] Fig.11 It is used in Figure 1A Schematic diagram of the state server in the system.

[0030] Fig.12 It is used in Figure 1A Schematic diagram of the web server in the system.

[0031] Fig.13A yes Figure 1A Schematic diagram of a security server in a system.

[0032] Fig. 13B is another schematic diagram of a security server according to an embodiment.

[0033] Fig.14Ais Figure 1A Figure 1 is a diagram of a user interface that displays contextual help on a remote device of a system.

[0034] Fig. 14B is Figure 1A Figure 1 is a diagram of a user interface that displays contextual help on a remote device of a system.

[0035] Fig.15A It is shown in Figure 1A Diagram of a user interface for addressable devices arranged by room on a remote device of a system.

[0036] Fig. 15B It is shown in Figure 1A Diagram of a user interface of addressable devices arranged by device type on a remote device of a system.

[0037] Fig. 15C It is shown in Figure 1A A diagram of a user interface of addressable devices arranged by scene type on a remote device of a system.

[0038] Fig.16 Is to show that Figure 1A A diagram of a system of LED addressable devices used together with a color picker user interface.

[0039] Fig.17 is a schematic block diagram of a remote device and an LED bulb addressable device according to an embodiment of the present invention.

[0040] Fig.18 is a schematic diagram of an interface between multiple hub devices according to an embodiment of the present invention.

[0041] Fig.19 yes Figure 1A Schematic diagram of the bridge in the system.

[0042] Fig. 20 When adding a new bridge Figure 1A Schematic diagram of the operation of the system.

[0043] Fig.21A It is a graphic Figure 1A Figure 2. Sandboxing process in a system.

[0044] Fig. 21B It is a graphic Figure 1A Another illustration of sandboxing in a system.

[0045] Fig. 22 It is a graphic Figure 1A A diagram of the response scenario definitions in the system.

[0046] Fig.23 It is a graphic Figure 1AA flowchart of the response scenarios of the ingredients in the system.

[0047] Fig.24 Is to show recommended purchases based on factors to complete Figure 1A A diagram of a user interface for a scenario in the system.

[0048] Fig.25 is for Figure 1A A diagram of a user interface showing a scenario in a system of FIG. 5 showing the ability of a user to select from a list of element tiles.

[0049] Fig.26 is for Figure 1A A diagram of a user interface showing a device operation block suggested based on user input in a scenario in a system of FIG.

[0050] Fig. 27 is for Figure 1A A diagram of a user interface showing a prompt for user input to select what device to provide an element in a system scenario.

[0051] Fig.28 It is shown in Figure 1A A diagram of the user interfaces for different scenarios for a given set of elements or devices in a system.

[0052] Fig.29A It is shown that for Figure 1A Diagram of a user interface for selecting a device in a system to map scenes to user input prompts for home-specific devices.

[0053] Fig.29B yes Figure 1A A schematic block diagram of the operation of the HA device scenario controller in the HA system.

[0054] Fig. 30A is a block diagram of an electronic device integration system according to another embodiment of the present invention.

[0055] Fig. 30B is a schematic diagram of a HA system for generating a user health score according to an embodiment.

[0056] Fig.31 is a diagram of a hub device for detecting proximity to a remote device according to an embodiment of the present invention.

[0057] Fig.32 is a schematic diagram of a system for integrating multiple electronic devices according to an embodiment of the present invention.

[0058] Fig.33 2 is a schematic diagram of an electronic device integration system including a short-range communication protocol ID device according to an embodiment of the present invention.

[0059] Fig.34 is a diagram illustrating a user interface for event generation on a remote device for a multiple electronic device integration system according to an embodiment of the present invention.

[0060] Fig.35 is a schematic diagram of a climate control system according to an embodiment.

[0061] Fig.36 It is a graphic Fig.35 Diagram of the operation of the climate control system.

[0062] Figures 37A-37E is a schematic block diagram of a HA system illustrating communications between an addressable HA device and a remote access wireless communication device according to an embodiment. DETAILED DESCRIPTION

[0063] The present invention will now be described more completely below with reference to the accompanying drawings, in which preferred embodiments of the present invention are shown. However, the present invention may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided to make this disclosure thorough and complete and to fully convey the scope of the present invention to those skilled in the art. The same reference numerals always refer to the same elements, and primary symbols are used in alternative embodiments to indicate similar elements.

[0064] First reference Figure 1A , the electronic device integration system illustratively has the form of a home automation (HA) system 20 and is referred to as a K4Connect system. The HA system 20 illustratively includes a plurality of addressable devices 31a-31n, a home device 32, a remote device 36, and a cloud device 33. Although the HA system 20 is described herein, it should be understood that the system is not limited to use in the home and can be used in any occasion, commercial, industrial, residential, etc.

[0065] The addressable devices 31a-31n may include controllable devices and / or sensors, such as motion detectors, thermostats, light switches, audio controls, door locks, and / or cameras. Of course, the addressable devices may include additional or other devices.

[0066] Although a cloud device 33 or a hardware server is described, it will be appreciated by those skilled in the art that the processing and functions performed by the cloud device can be performed by a processor 46 or by multiple processors in different geographic locations and through different networks understood by those skilled in the art as a cloud. The home device 32 can be a personal computer, a tablet computer, an independent computing device, or any other computing device. The HA system 20 can also include a hub device 34 (i.e., K4Hub). In some embodiments, the hub device 34 and the home device 32 can be within the home 47 and wirelessly connected to a home network that can provide communication with the Internet. The functions and interconnections of these devices within the system will be described in further detail below.

[0067] The K4Home software program operates the K4Connect HA system 20 for home, office, business and building automation for addressable devices 31a-31n that can be connected to the program. The K4Home software is available as a software package only that can be loaded onto a personal computer or other small computer device (e.g., home device 32). The functions of the K4Home software are performed by corresponding processors or processing circuitry on one or more devices running the K4Home software (e.g., processor 38 of home device 32), as will be described below.

[0068] The K4Hub 34 is a device that can also run the K4Home software and host the system architecture on the device. The K4Hub 34 includes a housing 41 and a hub processing circuit system 42 carried by the housing. The K4Hub 34 also includes a plurality of radio ports 43a-43n, such as a universal serial bus (USB) port carried by the housing 41 and used to couple to any one of a plurality of radio controllers 44a-44n. The K4Hub 34 runs the system locally and can communicate directly with the addressable devices 31a-31n, rather than being routed through cloud-based processing. In other words, the hub processing circuit system 42 cooperates with the inserted radio controllers 44a-44n to communicate with the addressable devices 31a-31n based on the corresponding protocol.

[0069] The radio controllers 44a-44n can each be used for a given radio protocol. For example, a Z-wave radio controller can be plugged into one of the radio ports 43a-43n, which allows the K4Hub 34 to communicate with Z-wave based addressable devices. A second or third radio controller can be plugged into the K4Hub's radio ports 43a-43n to add the ability to communicate with controllable devices using second and third radio protocols.

[0070] The K4Hub 34 is an improvement over the prior art because it reduces the latency and system failures common to current home automation devices that require a network connection. Similar to the K4Hub 34, the K4Home software running on a personal computer (e.g., home device 32) can be enhanced with additional home automation communication protocols (such as ZigBee and Z-wave) by attaching a port via the K4Hub or the computer's USB port.

[0071] K4App is the location of the user interface 35 of the K4Connect HA system 20 and allows the user to access the K4Home software and control the K4Connect system 20 through or from a remote device 36 (e.g., a smartphone or tablet device including a display 48 and a processor 49 coupled to the display). The user interface 35 can also be accessed by a desktop application for a personal computer and / or an on-screen application for a television. There can be more than one remote device 36, and each remote device can be a different type of device.

[0072] In some embodiments, the remote device 36 can connect "locally" without going through or communicating with the cloud device 33. This can be particularly advantageous because the communication can operate locally without relying on network connectivity and independently of the Internet. In addition, the communication can be relatively faster and more reliable.

[0073] A remote server or cloud device 33 running software referred to as K4Away is a cloud-based subscription system that provides connectivity between local K4Home software running, for example, on a home device 32 or K4Hub 34, and K4Apps outside the local home network (e.g., running on a remote device 36). K4Away also provides connectivity between the K4Home software and the K4Connect system analysis and help system. In some embodiments, the K4App can connect directly to the K4Home (i.e., home device 32 or K4Hub 34) without communicating through the cloud device 33, or without communicating indirectly through the cloud device.

[0074] Reference now Figure 1B , the above components of the HA system 20 will be described. The HA system 20 includes addressable HA devices 31a-31n, each of which is configured to communicate wirelessly using a corresponding HA wireless communication protocol in different HA wireless communication protocols. The addressable HA devices 31a-31n may include any one of a motion detector, a thermostat, a light switch, an audio controller, a door lock, and / or a camera. Of course, the addressable HA devices 31a-31n may include other and / or additional devices.

[0075] The HA system 20 also includes HA wireless radio controllers 44a-44n, each of which is configured to also wirelessly communicate using a corresponding different HA wireless communication protocol in the different HA wireless communication protocols. Each HA wireless radio controller 44a-44n includes a circuit system 441a-441n and a connector 442a-442n coupled thereto. For example, the HA wireless radio controllers 44a-44n can be Zigbee controllers, Z-wave controllers, and / or other types of controllers.

[0076] The HA system 20 also includes an HA hub device 34, which includes a housing 41 and wireless radio port connectors 43a-43n carried by the housing. Each port connector 43a-43n is configured to be coupled to a corresponding connector 442a-442n of a corresponding HA wireless radio controller 44a-44n. The port connectors 43a-43n can be, for example, USB connectors, and / or other or additional types of connectors. The HA hub device 34 also includes a hub processing circuit system 42 coupled to the wireless radio port connectors 43a-43n. The hub processing circuit system 42 communicates with the addressable HA devices 31a-31n based on the corresponding HA wireless communication protocol. In some embodiments, the HA wireless radio controllers 44a-44n can, for example, communicate directly with the addressable devices via the HA hub device 34, rather than being routed through cloud-based processing, as will be appreciated by those skilled in the art.

[0077] One method aspect is directed to a method for communicating in a HA system 20. The method includes using HA wireless radio controllers 44a-44n to wirelessly communicate using corresponding different HA wireless communication protocols in different HA wireless communication protocols. The method also includes using HA hub device 34 to communicate with addressable HA devices 31a-31n based on the corresponding HA wireless communication protocol.

[0078] Now additionally refer to Figure 2A , the main functionality of the HA system 20 (i.e., K4Connect) is based on a separate independent message queue server 50, which is a combination of an independent local message queue 51 located on a device running the K4Home software and a cloud message queue 52 hosted on a cloud device 33 (i.e., K4Away), where the cloud device 33 provides connectivity to registered devices outside the local home network. For example, the communication between the message queues 51, 52 and the connected addressable devices 31a-31n, the connected servers, and the connected bridges uses web sockets as the transmission medium.

[0079] The local message queue 51 and the cloud message queue 52 both function independently, but maintain a persistent connection so that communications to and from connected devices (e.g., servers and bridges) remain available regardless of the user's location. The persistent connection is initiated from the local message queue 51 to reduce security issues that can be inherent when piercing the firewall of the local network. For example, having the connection originate from within a firewalled system allows message queues 51, 52 to be connected more easily while maintaining the security integrity of the home system. In other words, each remote device 36 is connected to the cloud message queue 52, without being directly connected to the local message queue 51 or any one of K4Home32 or K4Hub 34. In addition, for increased security, the communication between the local message queue 51 and the cloud message queue 52, the connected addressable devices 31a-31n, the server and the bridge can be SSL encrypted, including on the local network. When the K4App is connected to the cloud or remote server 33, for example, via a remote device 36, the persistent connection allows the user to connect to the cloud server to serve as a direct connection to the local message queue 51.

[0080] The local message queue 51 receives messages from the cloud message queue 52 and distributes the messages to the cloud message queue 52, and receives messages from the local server 81 and the device bridge 82 and distributes the messages to the local server 81 and the device bridge 82. This distribution technology of messages allows the independence of each component of the program and leaves the logic or prescribed actions to the individual server or bridge. This independence of the components of the program can also reduce the possibility of system crash errors. For example, this also allows the continuous introduction of new bridges and compatibility with new devices without updating the complete software package.

[0081] As will be appreciated by those skilled in the art, typical prior art automation integration systems either exchange messages entirely within the home network or entirely over the Internet by traversing a firewall. The embodiments described herein advantageously provide a hybrid messaging approach that includes the increased speed of "in-home" message processing (processing over the Internet adds latency) and has the increased security of the Internet (no penetration of firewalls to expose the home network).

[0082] Reference now Figure 2B , another aspect of the HA system 20 will now be described with respect to the local message queue 51 and the cloud message queue 52. The HA system 20 includes addressable HA devices 31a-31n, each having respective device capabilities, device configurations, and device states associated therewith.

[0083] For example, each device configuration may include at least one of a device address, a device location, and a device identifier. An exemplary device configuration may include an IP address of the device, a location of the device in a house, and a channel location (e.g., left, right) in an audio configuration. Of course, a device configuration may include other and / or additional elements.

[0084] The capabilities of each device may include at least one of a sensing function and an output function. For example, with respect to a light switch, the device capabilities may include the capabilities of "on", "off", and being at different "dimmer levels".

[0085] The state of each device may include a current state among a plurality of possible states. For example, for a light switch, the current state may be "on", "off", and "dim to a given level".

[0086] The HA system 20 includes a cloud message queue controller 521 and a cloud message queue storage 522 coupled thereto in the cloud, the cloud message queue storage 522 being used to store device configurations, device capabilities, and device states of multiple addressable HA devices 31a-31n. For example, the cloud message queue controller 521 and the cloud message queue storage 522 may be part of the cloud message queue 52.

[0087] The HA system 20 further includes a home device message queue controller 511 and a home device message queue memory 512 coupled thereto, the home device message queue memory 512 being used to store device configurations, device capabilities, and device states of a plurality of addressable HA devices 31a-31n. For example, the home device message queue controller 511 and the home device message queue memory 512 may be part of the local message queue 51.

[0088] The cloud message queue controller 521 and the home device message queue controller 511 synchronize the device configuration, device capabilities, and device status of the addressable HA devices 31a-31n. The cloud message queue controller 521 exchanges messages related to the addressable HA devices with the local message queue controller 511, such as for communication with the addressable devices 31a-31n and for synchronization. For example, such messages may include messages related to the operation and control of the addressable HA devices 31a-31n.

[0089] The local client device 36a or remote device (e.g., running K4App) includes a local client device controller 361a and a local client device memory 362a coupled thereto, the local client device memory 362a being used to store device configurations, device capabilities, and device states of the addressable HA devices 31a-31n when synchronized with the local message queue controller 511. The local client device controller 361a exchanges messages (e.g., sensing, responding, and controlling operations) related to the addressable HA devices 31a-31n with the local message queue controller 511.

[0090] The cloud client device 36b or remote device (e.g., running K4App) includes a cloud client device controller 361b and a cloud client device memory 362b coupled thereto, which is used to store device configurations, device capabilities, and device states of the addressable HA devices 31a-31n when synchronized with the cloud message queue controller 521. The cloud client device controller 361b exchanges messages related to the addressable HA devices 31a-31n (e.g., sensing, response, and control operations) with the cloud message queue controller 521.

[0091] As will be appreciated by those skilled in the art, by synchronizing the device configurations, device capabilities, and device status or messages of the addressable HA devices 31a-31n, communications with the cloud client device 36a or the local client device 36b can be faster because, for example, processing of messages, responses, status queries, instructions, etc. can be processed on the cloud client device or the local client device, or at the nearest location to the cloud or message queue (i.e., requests or communications generally may not have to go to one or the other of the local message queue 51 or the cloud message queue 52).

[0092] One method aspect is directed to a method for communicating with multiple addressable HA devices 31a-31n, each addressable HA device 31a-31n having corresponding device capabilities, device configurations, and device states associated therewith. The method includes using a cloud message queue controller 521 and a cloud message queue memory 522 coupled thereto in the cloud, the cloud message queue memory 522 being used to store the device configurations, device capabilities, and device states of the multiple addressable HA devices. The method also includes using a home device message queue controller 511 and a home device message queue memory 512 coupled thereto, the home device message queue memory 512 being used to store the device configurations, device capabilities, and device states of the multiple addressable HA devices. The cloud message queue controller 521 and the home device message queue controller 511 synchronize the device configurations, device capabilities, and device states of the multiple addressable HA devices 31a-31n.

[0093] Now additionally refer to Figure 4-13, for example, using the K4Home program executed by the home device 32 or K4Hub 34 to provide an independent server or functional module for each function of the HA system 20. For security, the server 81 is separated from the bridge 82 running on the HA system 20, and the system can be allowed to run independently as a whole. The server 81 on the home automation integration system 20 includes an action server 69, an analysis server 54, a camera server 61, a configuration server 62, a debugging server 63, a discovery server 55, a loader server 64, a message server 65, a notification server 66, a status server 67, an update server 59, a web server 68, and a security server 56. If new functions are required, more servers can be added to the software. Although the term "server" is used herein, it should be understood that, for example, as described above, a server can be one or more independent software processes executed on one or more processors on any device. The functions of each server 81 are performed by a processor, a controller and / or related circuit system, particularly on the device where it is executed (e.g., a home device processor 38 or a hub device processing circuit system 42), as will be recognized by those skilled in the art.

[0094] The action server runs continuously on the HA system 20, more specifically the home device 32, and executes the response scenarios of the K4Home system or the components within the home ( Figure 3 The analysis server 54 records user and system actions to the cloud storage system or server 33, and receives suggestions for possible response scenarios that the user can implement or actions that the user can take to improve their K4Home HA system 20 ( Figure 4 ).

[0095] In the initial K4Home system setup, the analysis server 54 requests notifications from the servers 81 and bridges 82 on the system. The servers 81 and bridges 82 on the K4Home system 20 return notifications, which allow the analysis server 54 to subscribe to individual servers and bridges. Once subscribed, the servers and bridges 82 send individual events, commands, and variable changes to the analysis server 54, which keeps a log of the data sent.

[0096] At certain intervals (which can be periodic or regular), the analysis server 54 reports the collected data to the cloud system or cloud device 33 via a private globally unique identifier (GUID). The cloud-based analysis or device 33 processes and reviews the anonymous data, storing the data in a cloud database. This data is then used to review the functionality of the K4Home HA system 20, which can reveal any problems that may exist in the software. This HA system 20 can also use the data collected from the security server to assess any security threats and develop mitigation plans. The cloud-based analysis or cloud device 33 also reviews the K4Home system and recommends devices and response scenarios to the private GUID. Once the information in the cloud has been analyzed and collected by the cloud device 33, it will be pushed back to the local analysis server 54 the next time to "check in" to the cloud.

[0097] Camera Server 61 ( Figure 5 ) locates camera images / videos and streams the images / videos to the system. For example, the camera server also acts as an image proxy for remote users who cannot connect directly to the camera.

[0098] Configuration Server 62( Figure 6 ) stores the persistent configuration of the home automation integration system 20. The configuration server 62 also uses the device description during the device connection process to cooperate with the device setup wizard to set up the addressable devices 31a-31n on the HA system 20. The debugging server 63 enables bridge debugging ( Figure 7 ).

[0099] Discovery Server 55( Fig. 8A ) find addressable devices to connect to the K4Connect system 20. The discovery server 55 uses the signatures of devices (e.g., addressable devices 31a-31n) in its search to discover devices that are not inherently discoverable for connection to the system. With typical prior art home automation integration systems, certain addressable devices do not automatically broadcast their availability and therefore have to be connected manually by the user. Manual entry often involves advanced technical knowledge or having to follow detailed complex instructions to add a device to their home automation system, such as manually entering an IP address, device ID, and / or other identifying information. The discovery server 55 reduces these complexities.

[0100] The sample codes executed on the discovery server 55 with respect to the network device and the USB device are as follows:

[0101]

[0102]

[0103] The discovery server 55 is typically always running a process of monitoring the system home automation integration 20, either passively waiting for a signal from a new controllable device or scanning the system for signatures of addressable devices 31a-31n. The discovery server 55 runs UPNP and MDNS processes using a text matching process of the signatures from the addressable devices 31a-31n to identify the controllable devices. The discovery server 55 also runs a multicast process and connects to these unconnected addressable devices 83, for example, through a challenge response.

[0104] Advantageous elements of the discovery server 55 are ARP scanning and udev scanning. The ARP scan runs a port match for the loaded controllable device signature and runs a challenge-response process to identify the addressable devices 31a-31n. For example, the discovery server 55 can query the port with data and obtain an identification response based on the query. The ARP scan also identifies devices through MAC address matching. Another advantageous element is the UDEV scan, which uses USB matching for devices connected to hardware running K4Home and running TTY matching, which uses a challenge-response process to identify devices. As will be appreciated by those skilled in the art, any number of elements or network characteristics that define a controllable device signature may be used.

[0105] Once the discovery server 55 discovers a new addressable device (i.e., new to the system 20), it sends a notification to the configuration server 62 and notification server 66 via the message queue 51 ( Figure 7 ), then the notification server 66 notifies the user of the newly discovered addressable device and begins the wizard setup process. When new addressable devices become available (e.g., new to the market, not just the system), for which there is no recognizable signature, new signature filters can be added to the discovery server 55.

[0106] In some embodiments, advertisements from addressable devices 31a-31n can be used to limit addressable devices. For example, based on the advertisements, a controllable speaker device can appear to be a generic device to the home automation integration system 20. However, queries based on a subset of addresses or signature elements can be used, which can increase the speed of controllable device discovery. For example, signature elements can be used to qualify or limit device types, and discovery can continue based on that subset.

[0107] Now additionally refer to Figure 8B, the discovery server 55 will be described with respect to the HA system 20. The addressable HA devices 31a-31n each have a corresponding HA device signature associated therewith, and are each configured to communicate wirelessly using a corresponding different wireless communication protocol in the different wireless communication protocols. The addressable HA devices 31a-31n may include any one of a motion detector, a thermostat, a light switch, an audio controller, a door lock, and / or a camera. Of course, the addressable HA devices 31a-31n may include other and / or additional devices.

[0108] The discovery server 55 may be in the form of a controller 551 and a memory 552 coupled thereto. The memory 552 stores HA device signatures for paired and unpaired addressable HA devices in the addressable HA devices 31a-31n. The HA device signature may include, for example, a MAC address, port data, and / or a universal serial bus (USB) identifier.

[0109] The controller 551 polls the addressable HA devices 31a-31n and determines an unpaired addressable HA device from a plurality of addressable HA devices based on the polling. The controller 551 may poll the addressable HA devices 31a-31n by polling broadcasts from the addressable HA devices and / or by scanning the addressable devices in response to a given HA device signature among the stored HA device signatures stored in the memory 552.

[0110] The controller 551 also compares the associated HA device signature of the unpaired addressable HA device with the stored HA device signature. The controller 551 may compare the associated HA device signature of the unpaired addressable HA device with the stored HA device signature based on at least one of a Universal Plug and Play (UPnP) process and a Multicast Domain Name System (mDNS) process. For example, any one of the UPnP and mDNS processes may be performed based on a text matching process.

[0111] In some embodiments, the addressable HA devices 31a-31n may each have port data associated therewith, in which case the controller 551 may poll the addressable HA devices based on an address resolution protocol (ARP) scan and compare the associated HA device signatures of the unpaired addressable HA devices with the stored HA device signatures based on the port data from the ARP scan.

[0112] Alternatively or additionally, the controller 551 may poll the addressable HA devices 31a - 31n based on a udev scan, in which case the controller compares the associated HA device signatures of the unpaired addressable HA devices with the stored HA device signatures based on the udev scan.

[0113] When there is a match between the HA device signature of the unpaired addressable HA device and one of the stored HA device signatures, the controller 551 allows the unpaired addressable HA device to be paired to communicate with the unpaired addressable HA device using the corresponding wireless communication protocol. The controller 551 can prompt the user to approve the pairing of the unpaired addressable HA device. The pairing of the unpaired addressable HA device can be based on a challenge response from an electronic device associated with the user (e.g., the remote device 36).

[0114] The communication interface 553 provides communication between the controller 551 and the cloud (eg, the cloud device 33 ). The controller 551 communicates with the cloud device 33 via the communication interface 553 to update the HA device signature stored in the memory 552 .

[0115] The HA system 20 also includes radio controllers 44a-44n coupled to the controller 551. Each of the addressable devices 31a-31n is configured to wirelessly communicate with the controller 551 via the corresponding radio controller 44a-44n.

[0116] A method aspect is directed to a method for allowing pairing of unpaired addressable HA devices 31a-31n in a HA system 20. The method includes using a controller 551 and a memory 552 coupled thereto, the memory 552 storing a plurality of HA device signatures for paired and unpaired addressable HA devices in a plurality of addressable HA devices to poll the plurality of addressable HA devices and determine an unpaired addressable HA device from the plurality of addressable HA devices based on the polling. The controller 551 and the memory 552 are also used to compare the associated HA device signatures of the unpaired addressable HA devices 31a-31n with the stored HA device signatures, and when there is a match between the HA device signature of the unpaired addressable HA device and one of the stored HA device signatures, allow the unpaired addressable HA device to be paired to communicate with the unpaired addressable HA device using a corresponding wireless communication protocol.

[0117] Loader server 64 loads bridge 82 and server 81 ( Fig.10 ). The message server 65 runs or operates the message queue 51. The notification server 66 sends notifications from the system 20 to the user interface 35 at the remote device 36, for example ( Fig. 9 ).

[0118] The state server 67 is used to store a log of the last known state without having to poll the system-wide state machine of the devices in the system. Fig.11This is accomplished by having the state server 67 execute as a separate state machine that tracks the last known state of the system. As will be appreciated by those skilled in the art, the state server 67 is an advantageous improvement over the current common practice, for example, where the system stores state information in the driver stack. The web server 68 runs the user interface content ( Fig.12 ). In some embodiments, user interface content may be stored locally.

[0119] The security server 56 performs security processing of the home automation integration system 20 ( Fig.13A ). The security server 56 listens for open communication ports 84 that are not used by the home automation integration system 20. This allows the security server 56 to log when a device (e.g., an addressable device 31a-31n or a remote device 36) scans or connects to the port. The security server 56 can then ignore any devices that are known to scan or connect and do not pose a threat to the system and log when it receives an unknown or unexpected scan. For example, an open port may be scanned or connected to by a connected user's remote iPhone, but since this is an expected action from the iPhone, the security server 56 does not automatically consider this to be a threat to the home automation integration system 20. If a connected home automation or addressable device 31a-31n (e.g., a refrigerator) performs the same scan or connects to the open port, the security server 56 logs the action and then reports the log to the analysis server 54. The security server 56 knows what can be considered normal behavior for the addressable devices 31a-31n with the help of signature files included for all known controllable devices. In other words, because the types of remote and controllable devices coupled to the home automation integration system 20 on the network are known, the traffic between the devices can be monitored to maintain security. If it is determined that the traffic or communication associated with a particular device is unstable, the security server 56 can identify the device as hijacked and / or malware and be marked for reporting to the analysis server 54. The analysis server 54 uploads the data to the cloud device 33 for security analysis.

[0120] The following is an example security server signature that describes what might be considered normal behavior for a network device:

[0121]

[0122] The cloud server or cloud device 33 can perform analysis to evaluate or classify patterns and recommend actions for the security server 56. Some examples of actions taken by the security server 56 include notifying the user of the abnormal action of the device, disconnecting the damaged device from the K4Connect system 20, or ignoring if the action is not malicious. The K4Connect system 20 can identify vulnerabilities or attacks in the manufacturer's smart devices in some cases, and can provide information about the vulnerability to the manufacturer. Of course, the cloud device 33 can recommend other and / or additional actions to the security server based on the analysis.

[0123] Reference now Fig. 13B , the security server 56 will now be described with respect to the HA system 20. The HA system 20 includes addressable HA devices 31a-31n, each of which has a corresponding HA device signature associated therewith, which can be stored in the memory 562. The HA device signature can include data about the expected actions of the addressable HA device 31a-31n. For example, the HA device signature can also include a MAC address, port data, and a universal serial bus (USB) identifier. Of course, the HA device signature can include additional identifiers and / or any combination thereof that can be used as a basis for characterizing the operational behavior of the addressable HA devices 31a-31n.

[0124] The addressable HA devices 31a-31n may include any one of a motion detector, a thermostat, a light switch, an audio controller, a door lock, and / or a camera. Of course, the addressable HA devices 31a-31n may include other and / or additional devices. The addressable devices 31a-31n communicate wirelessly using corresponding different wireless communication protocols in different wireless communication protocols.

[0125] The HA system 20 includes an HA security controller 561 coupled to a memory 562 and communicating with addressable HA devices 31a-31n via corresponding communication ports, for example, by scanning or polling the communication ports. A given communication port is not currently in use or open. When a given addressable HA device 31a-31n communicates via a given communication port that is not currently in use, the HA security controller 561 determines whether the given addressable HA device is operating abnormally based on the corresponding HA device signature and transmits it to the cloud 33 to verify whether the given addressable HA device is operating abnormally. When a given addressable HA device 31a-31n is verified to be operating abnormally, the HA security controller 561 terminates communication with the given addressable HA device.

[0126] The HA security controller 561 also generates a notification when a given addressable HA device 31a-31n is verified to be operating abnormally. In some embodiments, the addressable HA devices 31a-31n each have a manufacturer associated therewith, and the HA security controller 561 can transmit the notification to the corresponding manufacturer associated with the given addressable HA device that is verified to be operating abnormally. Of course, as will be appreciated by those skilled in the art, the HA security controller 561 can transmit the notification to another device and / or entity.

[0127] The HA system 20 may further include a communication interface 563 that provides communication between the HA security controller 561 and the cloud 33. The HA security controller 561 communicates with the cloud 33 via the communication interface 563, for example, to update the HA device signature stored in the memory 562.

[0128] The HA system 20 also includes radio controllers 44a-44n coupled to the HA security controller 561. Each of the addressable devices 31a-31n may be configured to wirelessly communicate with the HA security controller 561 via the corresponding radio controller 44a-44n.

[0129] A method aspect is directed to a method for communicating in a HA system 20. The method includes using an HA security controller 561 to communicate with an addressable HA device 31a-31n via a corresponding communication port in a communication port, wherein a given communication port is not currently in use. The method also includes using the HA security controller 561 to determine whether a given addressable HA device among the addressable HA devices 31a-31n is operating abnormally based on a corresponding HA device signature when the given addressable HA device in the addressable HA devices 31a-31n communicates via a given communication port that is not currently in use, communicating with the cloud 33 to verify whether the given addressable HA device is operating abnormally, and terminating communication with the given addressable HA device and generating a notification when the given addressable HA device is verified to be operating abnormally.

[0130] Another aspect is directed to a setup wizard for the K4Home software. The setup wizard can provide an increasingly simple and relatively uniform setup process for each device connected to the K4Connect system 20 (and in particular, to the K4Home). The setup wizard can limit the actionable items on each screen step of the wizard to maintain simplicity. For example, the setup wizard can allow a question and a data input received from that question before moving to the next step in the wizard.

[0131] Each setup wizard is based on pre-built templates that allow software developers to collect data for setting up a device without having to build new user interface components. For example, each setup wizard can be customized for developers and bridge builders. Customization can be achieved by allowing each setup wizard to have a unique style sheet while keeping the base style consistent. In addition to the base style, the user interface in each setup wizard can be changeable, but it is expected that these changes are within specific parameters. If the developer does not have a suitable template available, for example, a user interface component can be created. Custom templates will still use K4Connect components when available, and custom templates cannot conflict with the relatively simple and unified setup process provided by the K4Home software.

[0132] Additional references Fig.14A 14b, each setup wizard may also provide contextual help by, for example, providing a progress bar 71 on the display 48 of the remote device 36 or as part of the user interface 35 including, for example, a help button 72 on the progress bar. The help button 72 links to the help corresponding to the user's current step in the setup wizard. In other words, depending on where the user is in the setup process, different instructions will be presented to the user on the display 48. This can be particularly advantageous because it helps the user with steps that often present problems in the setup and can make the user experience more adaptable and easier than current home automation setups.

[0133] Reference now Fig.15A -15c, the user interface 35 may provide several different ways to control the K4Connect system 20 or the addressable devices 31a-31n on the K4Connect system. For example, the K4Connect system 20 may be controlled by room ( Fig.15A ), by scene( Fig. 15B ) and by device type ( Fig. 15C ) to control. Of course, the K4Connect system 20 can be controlled in other ways or using other technologies.

[0134] The user interface 35, which can be presented via the display 48 of the remote device 36 (e.g., a touch screen display of a mobile phone), allows the user to view the addressable devices 31a-31n by device category of the addressable devices 31a-31n or by location (FIG 15a). The user can also directly switch from the addressable device selection according to the location to the addressable device category view. The user interface 35 also advantageously tracks the history of the devices used by tracking the last contacted device. This can allow the user to directly access the recently used addressable devices 31a-31n faster, rather than searching back through the previous page of the user interface. The user interface 35 can also provide increased usability by allowing the entire screen of the remote device 36 (e.g., a touch screen remote device) to be used to adjust the addressable devices 31a-31n instead of positioning a single point on the touch screen display for adjustment. In some embodiments, for example, the addressable devices 31a-31n can be controlled via the user interface 35 by means of voice recognition. Other types of control, such as biometrics or gesture (e.g., arm, hand, eye) recognition, can also be used and / or additionally.

[0135] Reference now Fig.16 , when one or more of the addressable devices 31a-31n are in the form of light emitting diode (LED) bulbs, the user interface 35 includes an LED color picker 75 functionality. The LED color picker 75 provides a more accurate method of setting the color in the controllable multi-color LED bulb 31a. Currently, the user selects a color from a palette and the bulb will adjust to the closest possible color. This can result in variations between what the user selects from the display 48 and the actual output from the multi-color LED bulb 31a.

[0136] With the help of the processor 49 of the remote device 36, the LED color picker 75 detects the colors that the multi-color LED bulb 31a can produce and presents these color options to the user. This is done, for example, by determining the CIE delta of the multi-color LED bulb 31a. The CIE delta can be determined by the manufacturer, and its data can be stored in the remote device 36 or received from the cloud device 33.

[0137] refer to Fig.17In another embodiment, when the remote device 36' includes a camera 86', the processor 49' of the remote device can cooperate with the camera to capture the color actually illuminated by the multi-color LED bulb 31a'. The processor 49' of the remote device 36' then displays the available colors of the multi-color LED bulb 31a' on the display 48' based on the stored CIE increment information or the captured image. The color is calculated in the CIE triangle relative to finding the color at the end of the increment. In addition, in some embodiments, as discussed above, in the case of a multi-color LED bulb, the bulb may, for example, include CIE increments based on the model of the bulb. The user then selects the exact color from the options on the display 48', and the multi-color LED bulb 31a' changes to the selected color. In contrast to current methods of selecting colors based on approximations, this matches the user's expectations of the bulb output.

[0138] As those skilled in the art will recognize, the capabilities of a multicolor LED bulb 31a are typically much less than that shown in a typical CIE diagram. The embodiments described herein advantageously determine the color display capabilities of a multicolor LED bulb 31a and allow those actual colors to be selected rather than approximated.

[0139] Now additionally refer to Fig.18 The user interface 35 also provides an interface for interacting with multiple K4Hub hubs or hub devices (e.g., home hub 34a and office hub 34b). Currently in the home automation market, end users are either unable to set up multiple hubs in their homes, or the hubs are combined in a cloud system to prevent users from being able to clearly distinguish between systems. The K4Connect system 20 connects (e.g.,

[0140] Automatically) to a local hub and connect to any other hubs through the cloud to advantageously allow the user the option of controlling multiple hubs from the user interface 35.

[0141] When connected to a local network, for example, via Wifi, the user interface 35 of the remote device 36 can automatically connect (i.e., without user intervention) to the hub devices 34a, 34b on the same local network. When using a cellular connection or Wifi network that is not connected to the hub devices 34a, 34b, the user interface 35 allows the user to pick which of the multiple systems they want to view. For example, in a first scenario, a connection is made to a hub device 34a located in the user's office. The K4App or user interface 35 controls the addressable devices 31a-31n from the office hub 34a, but the user can choose to switch the user interface to control other connected hubs. In a second scenario, when the user is only connected to a cellular network (such as an LTE network), the user interface 35 provides the user with the option to choose between the connected hubs (if there is more than one), so that the user can choose between the home hub 34b or the office hub 34a. In a third scenario, the user is connected to the home hub 34b and the user interface 35 automatically controls the addressable devices 31a-31n at home, but the user can switch to controlling the office hub 34a on the user interface.

[0142] For example, when a home device 32 or a hub device 34 (i.e., a device running the K4 family) detects a new addressable device 31a-31n, new software to support the newly detected addressable device may be downloaded. For example, an "app store" for controllable devices may provide support or drivers for the newly detected controllable device. For example, the "app store" may be hosted by a cloud server 33 or a third-party provider. With respect to an app store available on a cloud server, the cloud server may store addressable device drivers in memory. When a home device 32 or a hub device 34 detects a new addressable device 31a-31n, the home device or hub device may "pull down" the corresponding driver or software instead of the entire software package.

[0143] Now additionally refer to Fig.19 , more details of the bridge 82 will now be described. The K4Connect bridge 82 provides a translation layer for the message queue or message queue server 50 to communicate with the addressable devices 31a-31n connected to the K4Connect system 20. When a user or a predefined scenario executes a command on the K4Connect system 20, the message queue 50 sends a generic form of message to the associated bridge 82 through the Node.js API. The generic form of the message can be sent through different APIs or through different technologies, as will be recognized by those skilled in the art. The bridge 82 then translates the generic command into a specific command for the addressable device 31a-31n and sends the translated command to the addressable device.

[0144] The independence of the bridge 82 advantageously allows developers to write a bridge for nearly any controllable device independent of the entire K4Connect system 20. After the bridge 82, which may generally be stored separately from the message queue 50, is coded, it can be downloaded and integrated into the message queue 50 without having to update the entire K4Connect software program.

[0145] More specifically, for example, when a home device 32 or hub device 34 detects a new addressable device 31a-31n, new software (i.e., a bridge) for supporting the newly detected controllable device may be downloaded. For example, an "app store" for controllable devices may provide support or a bridge for the newly detected controllable device. For example, the "app store" may be hosted by a cloud server 33 or a third-party provider. With respect to the app store available on the cloud device 33, the cloud device may store the addressable device bridge in memory. When a home device 32 or hub device 34 detects a new addressable device 31a-31n, the home device or hub device may "pull down" the corresponding bridge or software instead of the entire software package.

[0146] The independence of each bridge also allows for better use of bandwidth and storage space on the home K4Connect system 20. By not downloading an entire software update package each time a bridge is updated, the user and K4Connect conserve Internet bandwidth and data. Moreover, the ability to download only the bridge 82 desired by each user allows the user to conserve storage space on the device running K4Home (e.g., home device 32 and / or K4Hub 34). For example, such reserved storage space allows the K4Connect system 20 to provision a relatively large number of bridges for new home automation devices without much concern for bloated software or limited storage space on the user's device.

[0147] Now specific reference Fig. 20, when a new bridge 91 is created and loaded into the K4Connect system 20, the update server 59 on each K4Connect system connects to the cloud device 33 or K4Away and is notified when the system performs an update. As will be appreciated by those skilled in the art, the update server 59 can perform updates by communicating with the cloud server and determining whether an update exists based on communication with it (e.g., based on date, update ID, etc.). The device signature and device description of the new bridge are sent to the update server 59. The one or more files associated with the device signature and description are generally much smaller than the complete bridge file, which is downloaded if the new controllable device is eventually connected to the K4Connect system 20. The update server 59 sends the device signature to the discovery server 35 and the device description to the configuration server 62. The device signature allows the discovery server 35 to scan available ports and discern whether a new addressable device 31a-31n that can be connected through the new bridge 91 is in the home. The device description includes a wizard process for setting up a new controllable device, such as described above. When the discovery server 55 finds a new controllable device that can be connected through the new bridge 91, the discovery server 55 sends a message to the configuration server 62, which notifies the configuration server of the new addressable device. The discovery server 55 also sends a new addressable device notification to the notification server 66, which launches the user interface 35 on the display 48 of the remote device 36 to notify the user of the new addressable device. The bridge wizard 92 is also launched. The bridge wizard 92 collects information for device description and requests from the configuration server 62.

[0148] Once the information has been gathered and the user provides responses, e.g., via the bridge wizard 92, the configuration server 62 notifies the loader server 64 of the newly configured addressable device. The loader server 64 requests a complete bridge download from the update server 59, and the update server requests the complete bridge from the cloud device 33 or K4Away. The update server 59 sends the complete bridge download to the loader server 64, which stores the file and starts the new bridge. The newly connected controllable device is thus connected to the K4Connect system 20.

[0149] Now additionally refer to Fig.21A, the bridges 82a-82c of the K4Connect system 20 are also what one skilled in the art may refer to as "sandboxes" so that the system state is not easily interrupted in the event of a given bridge failure. If one of the bridges 82a-82c fails or if the connection to the message queue 50 fails, the remaining system components continue to function. The bridges 82a-82c perform communications between themselves and the message queue 50 so that if communications fail, the bridges will generally restart communications. For example, if a bridge 82a-82c has an error, the loader server 64 reloads the bridge 82a-82c. If an error occurs in a bridge 82a-82c, these sandboxed processes limit or reduce the restart of the entire software program running on the home device 32 or hub device 34. However, one impact to the K4Connect system 20 may be the inability to control the specific addressable devices 31a-31n associated with the failed bridges 82a-82c, which may be quickly repaired when the loader server 64 reloads the bridges. The functionality of the message queue 50, other servers, and other bridges is generally unaffected. As noted above, bridges may be installed on demand (e.g., as needed) for communicating with addressable HA devices.

[0150] Reference now Fig. 21B , the "sandboxed" bridges 82a-82c will now be described with respect to the HA system 20. The HA system 20 includes addressable HA devices 31a-31n. The addressable HA devices 31a-31n may include any of a motion detector, a thermostat, a light switch, an audio controller, a door lock, and / or a camera. Of course, the addressable HA devices 31a-31n may include other and / or additional devices. The addressable devices 31a-31n communicate wirelessly using corresponding different wireless communication protocols in different wireless communication protocols.

[0151] The processor 641 and the memory 642 associated with the processor can cooperate to perform the functions described above with respect to the sandboxed bridges 82a-82c. More specifically, the processor 641 and the memory 642 are configured to implement the message queue 50. That is, the message queue 50 generates a generic message for a corresponding addressable HA device in the addressable HA devices 31a-31n. The processor 641 and the memory 642 also implement the sandboxed bridges 82a-82c. Each sandboxed bridge 82a-82c converts the generic message from the message queue 50 into a specific message for a given addressable HA device in the addressable HA devices 31a-31n. The specific message can be a specific control and / or status message specific to the corresponding sandboxed bridge 82a-82c.

[0152] Once one of the sandboxed bridges 82a-82c fails, the processor 641 and the memory 642 implement reloading the failed sandboxed bridge 82a-82c while maintaining the operation of the other sandboxed bridges. For example, the processor 641 can determine the failed one of the sandboxed bridges 82a-82c based on the communication between the sandboxed bridge and the message queue 50 and / or the communication between the sandboxed bridges 82a-82c.

[0153] The HA system 20 also includes radio controllers 44a-44n coupled to the processor 641. Each of the addressable devices 31a-31n may be configured to wirelessly communicate with the processor 641 via a corresponding radio controller 44a-44n.

[0154] One method aspect is directed to a method of maintaining operation of a plurality of sandboxed bridges 82a-82c in a HA system 20. The method includes using a processor 641 and a memory 642 associated therewith to generate a plurality of generic messages for corresponding addressable HA devices in a plurality of addressable HA devices 31a-31n via a message queue 50, and using the plurality of sandboxed bridges 82a-82c to convert the generic messages from the message queue into specific messages for a given addressable HA device in the addressable HA devices. The method also includes using the processor 641 and the memory 642 to reload the failed sandboxed bridge when one of the plurality of sandboxed bridges 82a-82c fails, while maintaining operation of the other sandboxed bridges.

[0155] Now additionally refer to Fig. 22 , the K4Home software that may be executed on a home device 32 or a hub device 34 also features response scenarios that serve as a list of elements 95 of the K4Connect system 20 that may then cause actions in the addressable devices 31a-31n connected to the system 20. Based on the state of the system 20, the response scenarios may also return plain language notifications to the user, for example, at the user interface 35.

[0156] Standard response scenarios can be set up by the user using a scenario wizard. The scenario wizard includes a list of addressable devices 31a-31n and command event variables or triggers. The user selects the trigger for the scenario, the affected addressable devices 31a-31n, and the actions or states that the addressable devices will take to respond to the scenario, for example, via the user interface 35 of the K4App.

[0157] The standard response scenario may be initiated by a list of triggers detected by the program or by the user activating a scenario 93 in the user interface 35. An example of a command variable or trigger list is as follows: Trigger 1 is a time period, Trigger 2 is the mobile control device connecting to the network, Trigger 3 is a set day, and Trigger 4 is a connected motion detector sensing motion.

[0158] The scene has specified which set of user-defined components or addressable devices 31a-31n will be touched for the scene and what states those addressable devices should take. For example, controllable device 1 31a is a television (TV), addressable device 2 31b is a set of lights in the TV room, addressable device 3 31c is the room thermostat in the TV room, and addressable device n 31n controls the operation of a coffee machine. The system 20 generates commands that are sent to the addressable devices. The addressable devices 31a-31n respond based on the commands.

[0159] For example, Trigger 1 is activated between 7-9pm, Trigger 2 is activated when a given user's smartphone or remote device 36 is connected to the local network, Trigger 3 is activated during weekdays, and Trigger 4 is activated by motion detected by the living room motion detector. Based on the triggers, the user-defined components turn on the TV to a given channel, dim the lights in the TV room, adjust the thermostat to 72 degrees, and start brewing evening decaf coffee.

[0160] Standard response scenarios can also be shared between users using cloud devices 33 or K4Away and a marketplace listing available scenarios. The K4Connect system 20 can also suggest possible other and / or additional addressable devices 31a-31n to connect to add functionality and more response scenarios to individual users of K4Home.

[0161] Once the user has completed the response scenario wizard or has added a shared response scenario, the remote device 36 can display a modeled animation of the scenario via the user interface 35, which shows what the scenario looks like when activated. The user can also access animations of the scenario and its triggers that will function throughout the day.

[0162] Reference now Fig.23 -29a, another aspect of K4Home is what can be called an attribute-based element-responsive scenario, which allows the use of different addressable devices that can produce the same attribute in each recipe. For example, the scenario is based on a specific attribute, rather than the scenario being tied to a specific addressable device 31a-31n for a given function. This advantageously allows the responsive scenario to be implemented using the same elements required by the responsive scenario but without using exactly the same devices.

[0163] For example, if a given user wants to know when another user is at home, they can set up a response scenario that identifies addressable devices 31a-31n, which can be used to indicate whether someone is at home. For a given user, an addressable device 31a-31n or an element in a response scenario can be a deactivated alarm system that gives the desired attributes when it is tripped to trigger a response scenario. The response scenario then causes the K4Connect system 20 to send a notification (e.g., a plain language notification) to the given user that another user is at home. This scenario can then be shared with a third user who does not yet have an alarm system but has a motion detector 101, where the motion detector 101 falls into the same list of devices that can give the desired attributes to complete the scenario. In other words, regardless of the specific addressable device 31a-31n, the scenario is associated with the desired outcome. In the case where the scenario is almost complete or can be enhanced by adding more controllable devices, the K4Connect system 20 notifies the user of possible scenario-based response scenarios, for example via a user interface 35 on a remote device 36, and links them to an online market where the user can download the addressable devices 31a-31n for free or for purchase.

[0164] Another example of a response scenario based on a list of elements is that if the system 20 indicates that a scenario has not been met, it can send a plain language notification that the scenario has not been met. For example, if a person has not arrived at the home at a certain time, the scenario includes the elements of presence (with the help of motion detectors, cameras, and connected smartphones (i.e., remote devices 36)) and time. The lack of presence at a specific time triggers the scenario and alerts the user. Of course, other people can also be alerted, such as a monitoring center and / or other designated personnel.

[0165] The user can create an element response scenario (box 114) or download a shared response scenario from the cloud device 33. K4Home then determines whether all elements exist in the K4Connect system 20 (box 102). If an addressable device 31a-31n that can provide element attributes is connected to the K4Connect system 20, the system determines the status of the addressable device (box 106) by polling the addressable device (box 103). If all elements of the scenario are met (box 104), the K4Connect system 20 executes the scenario (box 108). If the elements of the K4Connect system 20 do not meet the conditions (box 104), the system can either poll the attribute status again (box 103) or wait for a specified amount of time set by K4Home or the response scenario. If any of the elements / attributes are not available in the K4Connect system 20 because the addressable device 31a-31n that can provide the element is not connected, then K4Home sends a message to the analysis server 54 requesting the suggested controllable devices from the cloud device 33 (box 110) and may also cooperate to present the user with the purchase of the suggested controllable devices, for example, on the user interface 35 (box 110). Fig.24 At block 112, a new addressable device may be installed.

[0166] In another example, a user can download a response scene that uses a camera to record motion events within a specific time period. For example, a given user wants to record when his dog climbs onto the living room sofa during the given user's work from 8 am to 5 pm. The given user then constructs a scene with three elements: the ability to record video (provided by a camera connected to K4Home), motion (provided by the built-in motion detector of the same camera), and time period. The given user then shares this on the response scene market on K4Away or cloud device 33. For example, another user downloads the scene and intends to use the scene for home security at night. The other user has a camera, but no ability to sense motion. K4Home suggests that the other user install an independent motion sensor to be able to use the scene, and provides a link to the K4Store or cloud device 33 where any of many brands and types of motion detectors can be purchased. The other user then installs the motion sensor, and now, since all elements are met, the response scene can be enabled. The other user uses the same basic response scene to record any movement in his living room from 10 pm to 6 am, while using different devices to provide elements.

[0167] For example, a user may generate a responsive scene to provide an idea of ​​"home". A responsive scene may be generated relative to the user such that "when I am home, I want light in the living room". The K4Connect system 20 indicates or displays, for example, via a menu, "You have x devices that can be used to determine if I am home, and here are the devices that provide light". In other words, scenes are constructed first, and then addressable devices 31a-31n that can make scenes are provided.

[0168] Reference now Fig.29B , the element response scenarios will now be described as they relate to the HA system 20. The HA system 20 includes addressable HA devices 31a-31n at a given location. The addressable HA devices 31a-31n include any of a motion detector, a thermostat, a light switch, an audio controller, a door lock, and / or a camera. Of course, the addressable HA devices 31a-31n may include additional and / or other devices.

[0169] The HA system 20 also includes a HA device scene controller 581, which obtains a first desired scene including a first trigger action and a first response event from a user, for example, wirelessly. For example, the first trigger may be "when I get home" and the first response event may be "turn on the lights in the living room." In fact, the first trigger action and the first response event do not identify which addressable HA device 32a-32n is responsible for implementing the first trigger action and the first response event. The HA device scene controller 581 may obtain the first trigger action and the first response event, for example, from a user interface device 360 ​​(and more specifically, a user input device 351 coupled to a user interface controller 353 to allow user input). The user interface device 360 ​​may be a remote device, such as a tablet computer, a smart phone, and the like. There may be more than one first trigger action and any number of first response events.

[0170] The HA device scenario controller 581 also presents, for example, a first user-selectable list of corresponding addressable HA devices among the addressable HA devices 31a-31n that are capable of implementing the first desired scenario on the display 354 of the user interface device 360 ​​coupled to the user interface controller 353. In other words, the HA device scenario controller 581 presents the addressable HA devices 31a-31n that correspond to the first trigger action and the first response event or are about to perform the first trigger action and the first response event.

[0171] The HA device scenario controller 581 also determines the first user-selected addressable HA device among the addressable HA devices 31a-31n, and when a first trigger event occurs, uses the first user-selected addressable HA device to perform a first response event, thereby implementing a first desired scenario. The first desired scenario can be executed wirelessly, for example, the HA device scenario controller 581 can communicate wirelessly with the addressable HA devices 31a-31n to implement the first desired scenario. In some embodiments, the HA device scenario controller 581 can generate a notification when a trigger event occurs.

[0172] HA device scenario controller 581 also obtains, for example wirelessly, a second desired scenario including a second trigger action and a second response event from cloud 331. The second trigger action and the second response event are obtained without identifying the addressable HA devices 31a-31n responsible for implementing the second trigger action and the second response event.

[0173] The HA device scenario controller 581 can present, for example, on the display 354, a second user-selectable list of corresponding addressable HA devices 31a-31n that can realize the second desired scenario. In other words, the second scenario is obtained as a shared scenario, such as from another person's HA system. The HA device scenario controller 581 also determines the addressable HA devices 31a-31n selected by the second user, and similarly to the above, for example, wirelessly performs a second response event using the addressable HA devices selected by the second user when a second trigger event occurs, thereby realizing the second desired scenario.

[0174] The HA device scenario controller 581 may also determine when the addressable HA device 31a-31n at a given location is not capable of implementing the scenario. In this case, the HA device scenario controller 581 presents a purchase offer for an additional addressable HA device, for example, on the display 354. For example, the user may purchase the additional addressable HA device by clicking on a hyperlink.

[0175] One method aspect is directed to a method for implementing a first desired scenario and a second desired scenario in an HA system 20. The method includes obtaining a first desired scenario including a first trigger action and a first response event from a user using an HA device scenario controller 581 and presenting a first user-selectable list of corresponding addressable HA devices 31a-31n that can implement the first desired scenario. The HA device scenario controller 581 is also used to determine the addressable HA devices 31a-31n selected by the first user, and when the first trigger event occurs, perform a first response event using the addressable HA devices selected by the first user, thereby implementing the first desired scenario.

[0176] The HA device scenario controller 581 is also used to obtain a second desired scenario including a second trigger action and a second response event from the cloud 331, and present a second user-selectable list of corresponding addressable HA devices 31a-31n that can implement the second desired scenario. The device scenario controller 581 is also used to determine the addressable HA device selected by the second user among the addressable HA devices, and when at least one second trigger event occurs, use the addressable HA device 31a-31n selected by the second user to perform the second response event, thereby implementing the second desired scenario. In some embodiments, the HA device scenario controller 581 is used to determine when the addressable HA device 31a-31n at a given location is not able to implement the scenario, and present a purchase offer for additional addressable HA devices.

[0177] The development kits will now be described as they relate to the K4Connect system 20. The K4Connect system 20 provides software and hardware development kits. The software development kit builds a complete device stack for developers to interact with the message queue and handles all communications with the message queue. For example, the built-in bridge editor allows developers to create and edit bridges from a web browser, and the description editor creates a device description XML file.

[0178] The hardware development kit allows developers to connect controllable devices directly to the message queue without the need for an intermediate bridge. For example, when developers add a communication protocol to their controllable device, the K4Connect system 20 (particularly its communication components) can be integrated into their hardware to bypass the bridge on the system and communicate directly with the message queue.

[0179] Further details of the cloud device 33 or K4Away will now be described. In addition to the functionality of K4Away already described, K4Away also hosts an external API that provides an interface for devices that cannot connect to Internet-based services on their own. When connected to the K4Connect system 20 and K4Away, previously unconnected devices can be accessed by external services (such as IFTTT, Evernote, and Facebook) through their connection to K4Away.

[0180] Regarding security, the security model of the K4Connect system 20 is based on providing a relatively high level of security for the system. Each phone or remote device 36 is authenticated at two levels. The first level is the device-specific allowances added by the system administrator. The second level is the user login on the remote device 36. This two-tier system reduces the occurrence of unauthorized device logins even if there is a valid user login.

[0181] The K4Connect system 20 also provides security through its privacy approach in the collection of analytical data. The data is stored on two separate servers. One server holds a token representing the anonymous user, while the other server holds usage and analytical data. The connection between the two servers occurs when the user authorizes it for technical assistance. When a response scenario or device recommendation is sent to the user, the recommendation is typically only sent to the token representing the user. The user always remains anonymous. In other words, a portion of the information about the user can be selectively used to provide technical support, similar to a "need to know" basis.

[0182] The K4Connect system 20 also uses a secure method that grants users full rights and ownership of the collected data. The K4Connect system 20 collects and analyzes data from users and stores it on separate secure servers. After a threshold period of time (e.g., one year), the data is permanently deleted. This method includes the ability for users to override the ability to permanently delete their data at any time.

[0183] The K4Hub 34 can also be used as a Wifi router connected to a home router, so that all devices connected to the K4Connect system 20 are routed through the private Wifi network of the K4Hub 34. This advantageously allows separation between devices (such as personal computers) connected to the K4Connect system. This separation can reduce the chance of an attack on a personal computer affecting the network between the devices of the K4Connect system 20.

[0184] Reference now Fig. 30A On the other hand, a health-related device for use on a K4Connect system 20" is provided. A health-related device used in conjunction with the K4Connect system 20" may be referred to as K4Life. However, it should be noted that other and / or additional devices, whether health-related or not, may be part of the K4Life system. Similar to the K4Connect system described above, the K4Life system 20" includes addressable devices 31a"-31n", some of which may be in the form of health devices that measure human health-related data (such as the number of steps taken, blood pressure, weight, and other metrics). In other words, the K4Life system 20" performs the functions of the K4Connect system described above, and includes further health-related functions as will be described in further detail below. For example, the health device may include, for example, one or more bed sensors, motion detectors, fitness tracking devices, blood pressure cuffs / monitors, weight scales, and temperature probes. Of course, other and / or additional health devices or sensors, for example from the K4Home system, may be used.

[0185] In addition, K4App also provides social interactions such as photo sharing and live video chat. More specifically, when a live video chat starts, the K4Life system 20" can report the start time and duration of the live video chat to a central server (e.g., a cloud device 33" or a local server device).

[0186] The K4Life system 20" (e.g., analytics server 54") calculates a score (which may be referred to as a K4Score) that indicates the overall health of a user. The K4Score is determined by combining directly measured health data, activity levels measured from use of addressable devices or health devices 31a"-31n", and social engagement measured through use of the K4App. The K4Score may include or be based on other and / or additional information. For example, historical trends in this score may be used to predict improvements or declines in a user's health. Of course, this data may also be used for other purposes, such as transmission to other users (such as health care professionals, monitoring stations, etc.). For example, people who are sedentary, have irregular sleep patterns, and have little social interaction may be identified as having potential health problems. An example scenario in which the K4Life system 20" and K4Score may be relatively advantageous is when the system is used by elderly parents whose children wish to check on their parents' health or if the user simply wants to know about their own health.

[0187] In some embodiments, health or activity data can be viewed by family members or in a group living setting (such as an assisted living facility) or by a live or remote supervisor. Health data can also be displayed, for example, via a user interface 35" of a remote device 36" to show an individual user's health score or an aggregated health score for a community of users.

[0188] Reference now Fig. 30B , the health-related aspects of K4Connect or the HA system 20" will now be described. The HA system 20" includes addressable HA devices 31a"-31n". The addressable HA devices 31a"-31n" may include motion detectors, thermostats, light switches, audio controllers, door locks, cameras and / or health-related sensors (e.g., room occupancy sensors, bed sensors, pedometers, heart rate monitors, blood pressure monitors, temperature sensors, and weight scales). Of course, the addressable HA devices 31a"-31n" may include other and / or additional devices. The addressable devices 31a"-31n" communicate wirelessly using corresponding different wireless communication protocols in different wireless communication protocols.

[0189] The HA system 20" also includes a user interface device 36", which allows users to engage in social networking and generates user social networking data based thereon, such as data related to which social networking applications are used and the amount of time spent using each social networking application. The user interface device 36" includes a portable housing 361", a display 48" carried by the portable housing, a wireless communication circuit system 362" carried by the portable housing, and a user interface device controller 49" coupled to the display and a wireless communication circuit system for performing at least one wireless communication function. For example, the user interface device 36" can be a smart phone or a tablet computer, and can execute any number of social networking applications, such as photo sharing, live video chat, and social media applications.

[0190] The HA system 20" also includes a controller 381" and a memory 382" coupled thereto, the memory 382" storing measured user health data and determining user physical activity data based on the addressable HA devices 31a"-31n". The physical activity may be determined based on a period of time.

[0191] The controller 381" also generates a user health score based on the user social networking data, the user health data, and the user physical activity data, and transmits the user health score via the cloud 331". For example, the controller 381" may also generate a user health score at intervals over a period of time based on the determined physical activity level.

[0192] For example, the controller 381" can generate a notification when the user health score exceeds a threshold. More specifically, if the user health score indicates poor health, a notification (such as an email, SMS message, visual notification on a display, etc.) can be generated and transmitted to the electronic device 361" via the cloud 331". In some embodiments, if the user health score continues to decrease during the time period, the controller 381" can generate a notification. Once the user health score is transmitted to the cloud 331", it can be downloaded, for example, by the electronic device 361" for storage, viewing, analysis and / or other data processing, as will be appreciated by those skilled in the art.

[0193] One method aspect is directed to a method for transmitting a user health score in a HA system 20". The method includes allowing a user to socially network via a user interface device 36" and generating user social networking data based thereon. The method also includes using a controller 361" and a memory 362" coupled thereto for storing measured user health data to determine user physical activity data based on multiple addressable HA devices, generate a user health score based on the user social networking data, the user health data, and the user physical activity data, and transmit the user health score via the cloud.

[0194] Reference now Fig.31, the K4Connect system 20'" can also be used for location determination. The K4Connect system 20'" can detect mobile devices (i.e., remote devices 36'"') that are within a specified range of a K4Hub 34'"'. For example, these detections can be reported to a central server (e.g., a cloud device or K4Away), where they are used to estimate the location of the person or device within a home or facility. Since more than one K4Hub 34'" can detect a mobile or remote device 36'"' at one time, the K4Connect system 20'"' reduces duplicate data by comparing the detection strength of overlapping data and determining which K4Hub is closest to the detected person or device. Of course, the K4Connect system 20'"' described in this embodiment can be particularly useful for use with the above-mentioned K4Life system.

[0195] Reference now Fig.32 In another embodiment, multiple K4Life (or K4Connect) systems 20a""-20n"" may be used together in a system that may be referred to as a K4Community. The K4Community system advantageously allows data from multiple K4Life or K4Connect systems 20a""-20n"" such as in a cloud device or K4Away to be aggregated so as to be analyzed for comparison within the community. Data from other controllers and / or devices may also be aggregated. Of course, any or each system 20a""-20n"" may process or aggregate data, such as in its entirety or in a shared or load-balanced arrangement. Additionally, users in a K4Community 20a""-20n"" may be able to communicate with one another and, in some embodiments, see how others are performing relative to a given user's performance. As will be appreciated by those skilled in the art, because the health-related data is being collected and potentially exchanged, the health-related data is maintained anonymously and may be encrypted until the user or owner of the health data agrees to share or actually shares it.

[0196] In some embodiments, the K4Life or K4Community system may not be limited to health-related devices and health-related data. For example, the principles of the above system can be applied to utility management (e.g., apartment utility load control management). In such an embodiment, for example, sensors or controllable devices can be used to monitor energy and water usage, and build profiles based on this. Specific tenants who use more utilities relative to other tenants can be identified. Common areas can also be monitored and scored. Scores can also be assigned to each tenant.

[0197] refer to Fig.33In another embodiment, the K4Life system 120 can be used in a health care setting to determine how much time a health care professional has given to a patient or user. In one specific example, the system 120 can be used in a nursing home to monitor how much time a nurse spends with a user / patient, and when and whether the nurse is in the room 147 with the patient. The system 120 (and in particular the hub device 134) includes a short-range communication protocol controller 199 (such as Bluetooth). Of course, in this or other embodiments, the hub device 134 can be used interchangeably with home devices. Each nurse will also wear an identification device or tag 197, which includes a circuit system 196 configured to communicate with the system via a short-range communication protocol. When the nurse is in the same room as the user or patient and is within communication range, the system and the tag communicate, and the time and duration of the communication are logged. This information can be used in the K4Community environment, as will be appreciated by those skilled in the art.

[0198] Reference now Fig.34 In another embodiment, for example, in a K4Community system such as a medical facility, events or tickets based on addressable devices can be generated. Those events can be logged and / or assigned to staff and displayed on the user interface 135' of the remote device 136'. For example, when a staff member arrives at the room of a person associated with the event generation, the arrival time of the staff member can be logged, such as described above.

[0199] Although several embodiments have been described as including software executed by a processor or processing circuit system of an electronic device, it will be understood by those skilled in the art that software may include firmware, machine code, or a configuration of a processor or processing circuit system. Moreover, although several embodiments have been described, it will be appreciated that the functions described in any given embodiment may be used together with other and / or additional functions, for example, as described in different embodiments. Further, although the term "home" has been used to describe certain devices and / or locations (e.g., with respect to home automation), it will be appreciated by those skilled in the art that the system and its components may be used in other locations, such as apartments, health centers, and the like. Therefore, the term "home" is not particularly limited to the user's home. Moreover, although a processor and / or controller have been described herein, it will be appreciated that the processor and / or controller may include a circuit system for performing the corresponding functions and may also include a memory. For example, a memory may also be coupled to the processor and / or controller.

[0200] Method aspects include making a home automation integration system as described in any of the embodiments described herein, including, for example, K4Connect, K4Life, and K4Community. Other method aspects include the operation of the system or its various components and performing any of the functions detailed above (e.g., integration, communication, display, etc.).

[0201] Another aspect is directed to a non-transitory computer-readable medium storing instructions for performing any of the functions of the systems and methods described herein. For example, the functions of K4App, K4Home, and K4Away can be embodied as computer-executable instructions stored on a non-transitory computer-readable medium. Of course, other functions described herein can be embodied on a non-transitory computer-readable medium.

[0202] Reference now Fig.35 , another embodiment is directed to a climate control system 1020 that includes a heating, ventilation, and air conditioning (HVAC) system 1021 for an indoor building area 1022. The HVAC system 1021 can be switched between operating modes for heating and cooling. The climate control system 1020 includes a home automation (HA) thermostat device 1030 in the indoor building area 1022. The HA thermostat device 1030 includes a housing 1031 and an indoor temperature sensor 1032 carried by the housing. The indoor temperature sensor 1032 senses an indoor temperature of the indoor building area 1022.

[0203] The temperature controller 1033 is carried by the housing 1031. The HA thermostat device 1030 also includes wireless communication circuitry 1034 coupled to the temperature controller 1033. The wireless communication circuitry 1034 may be configured to communicate, for example, via Wifi, cellular, or other protocols.

[0204] The temperature controller 1033 obtains the set point temperature for the indoor building area 1022. The set point temperature may be obtained wirelessly, for example, via the wireless communication circuitry 1034. The set point temperature may be obtained from an input device, a remote electronic device, and / or other device, as will be appreciated by those skilled in the art.

[0205] The HA thermostat device 1030 also includes a user set point temperature input device 1035 and a display 1036, both of which are carried by the housing 1031 and coupled to the temperature controller 1033. The user set point temperature input device 1035 can be in the form of a touch display, a button, a rotatable dial, or other input device, as will be appreciated by those skilled in the art. The user set point temperature input device 1035 can be used to set the set point temperature. The temperature controller 1033 can cooperate with the display 1036 to display the indoor temperature and the set point temperature.

[0206] The set point temperature may also be generated or set based on the HA controller 1037 (e.g., as described above) and coupled to the HA thermostat device 1030 and configured to generate the set point temperature. As described above, the HA controller 1037 may be coupled to addressable HA devices 1038a-1038n (e.g., motion detectors, lighting devices, etc.). The HA controller 1037 generates the set point temperature based on one of the addressable HA devices 1038a-1038n. For example, based on motion detected from a motion detector, the HA controller 1037 may communicate with the HA thermostat device 1030 to set the set point temperature (i.e., set the set point temperature lower when someone is at home). Of course, the set point temperature may be set based on other types of addressable HA devices 1038a-1038n.

[0207] The temperature controller 1033 also obtains the outside temperature from outside the indoor building area 1022. The outside temperature can be obtained wirelessly, for example, via the Internet. For example, the outside temperature can be an outdoor temperature, or can be an indoor temperature of a space or area that can be considered to be outside the indoor building area 1022, for example. In some embodiments, more than one temperature sensor (indoor and / or outdoor) can be used to obtain the outside temperature.

[0208] The temperature controller 1033 determines the crossing of the outside temperature by the set point temperature and switches the HVAC system 1021 between operating modes based on the crossing of the outside temperature by the set point temperature and the indoor temperature moving beyond the set point temperature by a threshold temperature difference (e.g., one degree). Other threshold temperature differences may be used.

[0209] Now additionally refer to Fig.36 , the operation of the climate control system 1020 is illustrated by means of a graph 1040 and corresponding displays 1036a-1036e, which show corresponding indoor temperatures 1043a-1043e and set point temperatures 1044a-1044e at different points in time identified on the graph. In graph 1040, the outside temperature is shown by line 1041, and the actual or indoor temperature is shown by line 1042. Illustratively, when the outside temperature passes through deadband 1045, the indoor or room temperature momentarily deviates from the desired or set point temperature.

[0210] One method aspect is directed to a method of operating a climate control system 1020. The method includes sensing an indoor temperature of an indoor building area 1022 via an indoor temperature sensor 1032. The method also includes using a HA thermostat device 1030 in the indoor building area 1022 to obtain a set point temperature of the indoor building area, obtain an outside temperature from outside the indoor building area, determine a crossing of the outside temperature of the set point temperature, and switch the HVAC system 1021 between operating modes based on the crossing of the outside temperature of the set point temperature and the indoor temperature of the indoor building area 1022 moving beyond the set point temperature by a threshold temperature difference.

[0211] Reference now Figure 37A-3 7e, in another embodiment of the HA system 2020, it may be desirable to remotely access the addressable HA devices 2031a-2031n. For example, remote access to the addressable HA devices 2031a-2031n (also referred to as IOT devices) may be particularly helpful for troubleshooting issues with a given addressable HA device and / or updating software or configuration.

[0212] The addressable HA devices 2031a-2031n are typically behind one or more network address translation (NAT) routers and / or firewalls and are therefore generally not accessible via the Internet, as will be appreciated by those skilled in the art. Thus, in order to access the addressable HA devices 2031a-2031n, on-demand secure shell (SSH) tunneling may be used.

[0213] On-demand SSH tunneling allows a given addressable HA device in the addressable HA devices 2031a-2031n to communicate with a known host, for example, via a periodic connection, to retrieve tunneling instructions. For example, the tunneling instructions can thus allow remote access to a given addressable HA device via the SSH protocol while reducing overhead. Of course, other protocols (e.g., secure protocols) may also be used.

[0214] To establish a remote connection to an addressable HA or IOT device 2031a-2031n, a request to open a tunnel is issued for a given addressable HA device. This may be performed by a remote user, for example, using on-demand SSH tunneling via a remote access wireless communication device 2036, thereby creating a file (e.g., a publicly accessible) location specific to a given addressable HA device. Fig.37A ). For example, the web-visible location may be on server 2099 or other web-visible location. In an example embodiment, the file may be an Amazon Simple Storage Service (S3) file that is a hash of the unique identifier of a given addressable HA device and the last unique cloud session identifier. The S3 file may include other and / or additional information about the addressable HA devices 2031a-2031n.

[0215] The cloud server 2033 may make device-specific instructions for a given HA device 2031a-2031n available at a known location (e.g., on server 2099). Fig.37B ). In some embodiments, device-specific instructions may be collocated on the cloud server 2033.

[0216] The addressable HA devices 2031a-2031n communicate with this location, for example, by periodically polling. For example, the location may be polled every few minutes. Of course, the addressable HA devices 2031a-2031n may communicate with or poll the location at longer, shorter, and / or different intervals. For example, based on the polling, the addressable HA devices 2031a-2031n find the tunneling instructions ( Fig.37C ). In one example, the instruction may be a json file including cloud-visible host / port / username / password. Of course, the instruction may be embodied in different types of files, and / or other data elements may be stored in the instruction file.

[0217] A given addressable HA device 2031a-2031n opens an SSH tunnel to the cloud server 2033 according to instructions retrieved from a web accessible location ( Fig.37D The remote user can then connect to the tunnel's cloud via the remote access wireless communication device 2036, allowing communication with a given addressable HA device 2031a-2031n, for example, by logging into it as if it were Internet visible ( Fig.37E ).

[0218] Many modifications and other embodiments of the present invention will occur to those skilled in the art having benefit of the teachings given in the foregoing description and the associated drawings. Therefore, it is to be understood that the present invention is not limited to the specific embodiments disclosed, and these modifications and embodiments are intended to be included within the scope of the appended claims.

Claims

1. A home automation HA system, include: Multiple addressable HA devices at a given location; The HA device scene controller is configured as follows: obtaining from a user a first desired scenario including at least one first trigger action and at least one first response event regardless of the capabilities of the plurality of addressable HA devices, and after obtaining the first desired scenario, presenting a first user-selectable list of corresponding addressable HA devices among the addressable HA devices that are capable of implementing the first desired scenario including the at least one first trigger action and the at least one first response event, determining a first user-selected addressable HA device among the addressable HA devices that can realize the first desired scenario, When the at least one first triggering action occurs, the at least one first response event is executed using the first user-selected addressable HA device among the addressable HA devices, thereby realizing a first desired scenario, obtaining from the cloud a second desired scenario including at least one second trigger action and at least one second response event, After obtaining the second desired scenario, presenting a second user-selectable list of corresponding addressable HA devices that can implement the second desired scenario including at least one second trigger action and at least one second response event, determining a second user-selected addressable HA device of the addressable HA device, and When the at least one second triggering action occurs, the at least one second response event is performed using the second user-selected addressable HA device of the addressable HA devices, thereby achieving a second desired scenario.

2. The HA system of claim 1, wherein the HA device scenario controller is configured to obtain the at least one first and second trigger actions without obtaining a corresponding addressable HA device in the addressable HA device for implementing the at least one first and second trigger actions.

3. The HA system of claim 1, wherein the HA device scenario controller is configured to obtain the at least one first and second response events without obtaining a corresponding addressable HA device in the addressable HA device for implementing the at least one first and second response events.

4. The HA system of claim 1, further comprising a user interface device coupled to the HA device scene controller, and the user interface device include: monitor; User input devices; as well as A user interface controller is coupled to the display and the user input device and is configured to: displaying the first and second user-selectable lists on the display, and Via the user input device, a user is allowed to select the first and second user-selected addressable HA devices among the addressable HA devices.

5. The HA system of claim 1, wherein the HA device scenario controller is configured to generate a notification for a user when the at least one first or second triggering action occurs. 6 . The HA system of claim 1 , wherein the HA device scene controller is configured to wirelessly obtain the at least one first and second trigger actions and the at least one first and second response events.

7. The HA system of claim 1, wherein the HA device scenario controller is configured to implement the first and second desired scenarios wirelessly.

8. The HA system of claim 1, wherein each of the plurality of addressable HA devices comprises one of a motion detector, a thermostat, a light switch, an audio controller, a door lock, and a camera.

9. A home automation HA system, include: Multiple addressable HA devices at a given location; The HA device scene controller is configured as follows: obtaining, from a user, a first desired scenario including at least one first trigger action and at least one first response event for operating a corresponding addressable HA device of the plurality of addressable HA devices regardless of capabilities of the plurality of addressable HA devices, determining when the plurality of addressable HA devices at the given location are unable to achieve the first desired scenario including at least one first trigger action and at least one first response event, and after obtaining the first desired scenario, presenting a purchase offer for an additional addressable HA device that is capable of achieving at least a portion of the first desired scenario, determining when the plurality of addressable HA devices at the given location are capable of implementing the scenario, and presenting a first user-selectable list of corresponding ones of the addressable HA devices that are capable of implementing the first desired scenario, determining a first user-selected addressable HA device of the addressable HA device, and When the at least one first triggering action occurs, the at least one first response event is executed using the addressable HA device selected by the first user among the addressable HA devices, thereby realizing the first desired scenario.

10. The HA system of claim 9, wherein the HA device scenario controller is configured to obtain the at least one first trigger action without obtaining a corresponding addressable HA device among the addressable HA devices for implementing the first trigger action.

11. The HA system of claim 9, wherein the HA device scenario controller is configured to obtain the at least one first response event without obtaining a corresponding addressable HA device among the addressable HA devices for implementing the at least one first response event.

12. The HA system of claim 9, further comprising a user interface device coupled to the HA device scene controller, the user interface device include: monitor; User input devices; as well as A user interface controller is coupled to the display and the user input device and is configured to: displaying the purchase offer and the first user-selectable list on the display, and Via the user input device, a user is allowed to select a first user-selected addressable HA device among the addressable HA devices.

13. The HA system of claim 9, wherein the HA device scenario controller is configured to generate a notification for a user when the at least one first triggering action occurs.

14. The HA system of claim 9, wherein the HA device scene controller is configured to wirelessly obtain the at least one first trigger action and the at least one first response event.

15. The HA system of claim 9, wherein the HA device scenario controller is configured to implement the first and second desired scenarios wirelessly.

16. The HA system of claim 9, wherein each of the plurality of addressable HA devices comprises one of a motion detector, a thermostat, a light switch, an audio controller, a door lock, and a camera.

17. A method for implementing first and second desired scenarios in a home automation HA system, the HA system comprising a plurality of addressable HA devices at a given location, the method include: Using HA device scene controller: obtaining from a user a first desired scenario including at least one first trigger action and at least one first response event regardless of the capabilities of the plurality of addressable HA devices, and after obtaining the first desired scenario, presenting a first user-selectable list of corresponding addressable HA devices among the addressable HA devices that are capable of implementing the first desired scenario including the at least one first trigger action and the at least one first response event, determining a first user-selected addressable HA device among the addressable HA devices that can realize the first desired scenario, When the at least one first triggering action occurs, the at least one first response event is executed using the first user-selected addressable HA device among the addressable HA devices, thereby realizing the first desired scenario, obtaining from the cloud a second desired scenario including at least one second trigger action and at least one second response event, After obtaining the second desired scenario, presenting a second user-selectable list of corresponding addressable HA devices in the addressable HA devices that can implement the second desired scenario including at least one second trigger action and at least one second response event, determining a second user-selected addressable HA device of the addressable HA device, and Upon occurrence of the at least one second triggering action, the at least one second response event is performed using the second user-selected addressable HA device of the addressable HA devices, thereby achieving the second desired scenario.

18. The method of claim 17, wherein the HA device scenario controller obtains the at least one first and second trigger actions without obtaining a corresponding addressable HA device in the addressable HA devices for implementing the at least one first and second trigger actions.

19. The method of claim 17, wherein the HA device scenario controller obtains the at least one first and second response events without obtaining a corresponding addressable HA device among the addressable HA devices for implementing the at least one first and second response events.

20. The method of claim 17, further comprising displaying the first and second user-selectable lists on a display using a user interface device coupled to the HA device scene controller, and allowing a user to select first and second user-selected addressable HA devices among the addressable HA devices via a user input device.

21. A method for implementing a first desired scenario in a home automation HA system, the HA system comprising a plurality of addressable HA devices at a given location, the method include: Using HA device scene controller: obtaining, from a user, a first desired scenario including at least one first trigger action and at least one first response event for operating a corresponding addressable HA device of the plurality of addressable HA devices regardless of capabilities of the plurality of addressable HA devices, determining when the plurality of addressable HA devices at the given location cannot achieve the first desired scenario including at least one first trigger action and at least one first response event, and after obtaining the first desired scenario, presenting a purchase offer for additional addressable HA devices, determining when the plurality of addressable HA devices at the given location are capable of implementing the scenario, and presenting a first user-selectable list of corresponding ones of the addressable HA devices that are capable of implementing the first desired scenario, Determine a first user-selected addressable HA device among the addressable HA devices, and when the at least one first trigger action occurs, use the first user-selected addressable HA device among the addressable HA devices to execute the at least one first response event, thereby realizing a first desired scenario.

22. The method of claim 21, wherein the HA device scenario controller obtains the at least one first trigger action without obtaining a corresponding addressable HA device in the addressable HA devices for implementing the at least one first trigger action.

23. The method of claim 21, wherein the HA device scenario controller obtains the at least one first response event without obtaining a corresponding addressable HA device in the addressable HA devices for implementing the at least one first response event.

24. The method of claim 21, further comprising displaying the first user-selectable list on a display using a user interface device coupled to the HA device scene controller, and allowing a user to select a first user-selected addressable HA device among the addressable HA devices via a user input device.

Images