Multi-role device for automation environments

Abstract

According to one aspect of the invention, an automation control device, such as a host or hub, is provided with enhanced functionality such that it becomes a multi-role automation control device. The multi-role automation control device is capable of controlling an extended class of "smart" devices or other devices for which their native control protocols are generally incompatible. The multi-role automation control device may be implemented using a variety of underlying devices, including a host, hub, sound bar, "smart" thermostat, "smart" amplifier, audio speaker, or other device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 62 / 805,456 filed on February 14, 2019 by Madonna et al. for “MULTI-ROLE DEVICES FOR AUTOMATION ENVIRONMENTS,” which is hereby incorporated by reference.

[0003] Incorporated Patents and Applications

[0004] U.S. Patent No. 7,930,644, issued on April 19, 2011, and entitled "PROGRAMMING ENVIRONMENT AND METADATAMANAGEMENT FOR PROGRAMMABLE MULTIMEDIA CONTROLLER", is assigned to the common assignee of the present application and is hereby incorporated by reference. U.S. Patent No. 8,213,463, issued on July 3, 2012, and entitled "LOCATION-BASED SHARING OF MULTIMEDIA CONTROL RESOURCES", is assigned to the common assignee of the present application and is hereby incorporated by reference. U.S. Patent No. 8,296,669, issued on October 23, 2012, and entitled "VIRTUAL ROOM-BASED LIGHT FIXTURE AND DEVICE CONTROL", is assigned to the common assignee of the present application and is hereby incorporated by reference. U.S. Patent No. 8,659,704, issued on February 25, 2014, and entitled “APPARATUS AND METHOD FOR MIXING GRAPHICS WITH VIDEO IMAGES” is assigned to the common assignee of the present application and is hereby incorporated by reference. U.S. Patent No. 9,153,125, issued on October 6, 2015, and entitled “PROGRAMMABLE MULTIMEDIA CONTROLLER WITH PROGRAMMABLE SERVICES” is assigned to the common assignee of the present application and is hereby incorporated by reference. Co-pending application serial number 16 / 123,731, filed on September 6, 2018, and entitled “SMALL SCREEN VIRTUAL ROOM-BASED USER INTERFACE” is assigned to the common assignee of the present application and is hereby incorporated by reference. Co-pending application serial number 16 / 254,245, filed on January 22, 2019 and entitled “DEVICE MOUNTING SYSTEM,” is assigned to the common assignee of this application and is hereby incorporated by reference. Field of the Invention

[0005] The present invention relates generally to the field of automation, and more particularly to enhancing the capabilities of various devices so that they can perform multiple roles in an automation environment.

[0006] Background Information

[0007] The demand for automation of residential and commercial environments has led to a dramatic expansion of the market for "smart" doorbells, thermostats, lighting, security systems, audio, video, and many other devices. These devices are provided by a considerable number of manufacturers, which, for competitive or performance reasons, adopt different control protocols that are generally incompatible with each other. Therefore, a "smart" thermostat provided by one manufacturer does not operate according to widely adopted standards, and often will not interoperate with "smart" lighting provided by a different manufacturer. Among other disadvantages, this lack of interoperability limits the end user's choices in selecting devices for a specific automation environment. Another disadvantage is that it prevents the end user from effectively controlling all devices in an environment with a single control device (e.g., a remote control, a smart phone, or a tablet computer).

[0008] As a result, end users are often forced to either select one manufacturer - even if that manufacturer does not offer all of the "smart" devices that the end user is interested in - or select multiple manufacturers at increased cost and lack of interoperability. Summary of the invention

[0009] According to one aspect of the invention, an automation control device, such as a host or hub, is provided with enhanced functionality such that it becomes a multi-role automation control device. The multi-role automation control device is capable of controlling an extended class of "smart" devices or other devices for which their native control protocols are generally incompatible. The multi-role automation control device may be implemented using a variety of underlying devices, including a host, hub, sound bar, "smart" thermostat, "smart" amplifier, audio speaker, or other device.

[0010] According to another aspect of the present invention, an automation control hub is provided with enhanced functionality, making it a multi-role automation control hub. Utilizing the high performance graphics processing available in the automation control hub, the multi-role automation control hub is provided with an enhanced screen display. The enhanced screen display provides a user interface for automating control of an expanded variety of devices than the automation control hub can control with its local functionality. The enhanced screen display can also provide advanced video display management, including windowing, tiling, and picture-in-picture, among others. The enhanced screen display can also support configuration of substantially all aspects of the automation environment as well as real-time control and status. More specifically, the enhanced screen display can provide access and control to services, preferences, scenes, automation, and scheduling.

[0011] In addition, multiple automation control hubs with enhanced screen displays can be brought together as a shared resource that distributes such screen displays to a large number of TVs or video monitors. Such distribution can be achieved by transmitting or streaming over IP dedicated to TV HDMI.

[0012] According to another aspect of the present invention, a ceiling or wall mountable speaker including wired network connectivity is provided with additional functionality including wireless network connectivity. As a result, each speaker becomes a multi-role speaker that acts both as an audio output device and as a wireless network access point. In addition, as mentioned above, the multi-role speaker can perform a third role, namely, an automation control device.

[0013] According to another aspect of the present invention, an automation control hub is provided with AVB network connectivity and becomes a multi-role automation control hub. Through the AVB network, the multi-role automation control hub distributes music to networked speakers, sound bars or other devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The following description of the invention refers to the accompanying drawings, in which:

[0015] Figure 1 is a block diagram of an automation environment including a multi-role automation host according to one aspect of the present invention;

[0016] Figure 2 Can be used as Figure 1 A block diagram of a sound bar based on a multi-role automation host as shown in;

[0017] Figure 3 It is a diagram of Figure 1 A message flow diagram of an example of higher level messaging performed by a multi-role automation host and other devices as shown in;

[0018] Figure 4 It is a diagram of Figure 1 A message flow diagram of an example of lower level messaging performed by a multi-role automation host and other devices as shown in;

[0019] Figure 5 is a block diagram of an automation environment according to one aspect of the present invention, the automation environment including a multi-role automation hub providing an enhanced screen display;

[0020] Figures 6A-6D Is Figure 5 An exemplary screenshot of an enhanced screen display generated by a multi-role automation hub as shown in;

[0021] Figure 7 is a diagram showing the Figure 5 An enhanced screen display shown in FIG. 1 shows a message flow diagram of an example of higher level messaging performed by a multi-role automation hub;

[0022] Fig. 7A It is a block diagram of an automation environment where multiple are aggregated into shared resources;

[0023] Figure 7B It's a picture. Fig. 7A Message flow diagram for communication between various devices;

[0024] Figure 8 is a block diagram of a multi-role speaker according to one aspect of the present invention; and

[0025] Fig. 9 is a block diagram of a multi-role amplifier according to one aspect of the present invention.

[0026] DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

[0027] Figure 1 An automation environment 100 is shown that includes a multi-role automation host 104. A first network cloud 102 is accessible by the multi-role automation host 104. The multi-role automation host 104 includes a system configuration (not shown), which in turn includes a component or device profile (not shown) associated with each simplified device 108a ... 108b controlled by the multi-role automation host 104. In general, a device profile is a file or other data structure that describes the inputs, outputs, and available command sets of an associated device. The multi-role automation host 104 must have access to the associated device profile, either through internal storage, the network cloud 102, or another external source, in order to control a given device.

[0028] Examples of system configurations, component profiles, and hardware and software suitable for use with the multi-role automation host 104 may be found in US Pat. No. 9,153,125, which is incorporated by reference above.

[0029] The user can interact with and effectively control the multi-role automation host 104 by operating a user device 116a running a host control application 120 compatible with the multi-role automation host 104. The user device 116a can be implemented with a smart phone, a tablet computer, or other device capable of running the host control application 120 and having appropriate network connectivity to communicate with the multi-role automation host 104. Alternatively, the user can interact with and effectively control the multi-role automation host 104 using a user remote control (remote) 118a compatible with the multi-role automation host 104. The multi-role automation host 104 can be based on, for example, a smart host model SHC-S2-00, a surround sound bar, a "smart" multi-stable thermostat, a "smart" audio speaker (as described below in conjunction with Figure 8 described above), "smart" amplifiers (such as those combined with Fig. 9 described) or other equipment provided by Savant Systems, LLC of Hyannis, Massachusetts.

[0030] The second network cloud 110 is accessible by the automation hub 112. Unlike the multi-role automation host 104, which is capable of controlling any profiled device, the automation hub 112 is conventionally only capable of controlling devices 114a-114c that have been certified as compliant with the control protocol (i.e., devices that have been certified as compliant with the control protocol specified by the manufacturer of the automation hub 112).

[0031] The user may interact with and effectively control the automation hub 112 by operating a user device 116b running a hub control application 122 compatible with the automation hub 112. The user device 116b may be implemented with a smart phone, tablet computer, or other device capable of running the hub control application 122 and having appropriate network connectivity to communicate with the automation hub 112. Alternatively, the user may interact with and effectively control the automation hub 112 using a user remote control 118b compatible with the automation hub 112. For example, the automation hub 112 may be based on an Apple TV®, HomePod™, or iPad® running HomeKit®, all of which are provided by Apple Inc. of Cupertino, California.

[0032] The multi-role automation host 104 includes an automation bridge 106. In one embodiment, an SDK available from Apple Inc. can be used to create the automation bridge 106 and embed it into the multi-role automation host 104. As described in more detail below, the automation bridge 106 functions in part to communicate with the automation hub 112 (bidirectionally) using the same manufacturer-specified control protocol used by the certified control protocol-compliant devices 114a-114c. The automation bridge 106 also functions to convert control protocol messages received from the automation hub 112 into an appropriate form for processing by the multi-role automation host 104 (and vice versa). Advantageously, through such communication and protocol conversion, and depending on the type of device, the profiled devices 108a...108b and the user remote control 118a are exposed to the automation hub 112. That is, the profiled devices 108a . . . 108b and the user remote control 118a become visible to and controllable by the automation hub 112 by virtue of the multi-role automation host 104 even if they are not authenticated as compliant with the control protocol used by the automation hub 112 .

[0033] Figure 2 shows a method for implementing a multi-role automation host 104 ( Figure 1). A microprocessor 202 is coupled to an audio digital signal processor (DSP) 204 for bidirectional communication. The audio DSP 204 is also coupled to a TosLink 214. The audio DSP 204 is coupled to a digital-to-analog converter (not shown), the output of which is coupled to an amplifier 206, the output of which is coupled to various audio speakers 208.

[0034] The microprocessor 202 is also bidirectionally communicatively coupled to the WISA module 210 and the WiFi dual band module 212. In addition, the microprocessor 202 is coupled to the LAN / AVB port 216, the microphone 218, and the control port 220, which may include an RS232 control port, an infrared (IR) control port, or other control ports. The AC / DC power supply 224 provides power to the sound bar 200.

[0035] Figure 3 A message flow diagram is shown which illustrates the process of enabling the profiling devices 108a ... 108b and the user remote control 118a ( Figure 1 ) is visible to and controllable by the automation hub 112. The host control application 120 running on the user device 116a initiates a discovery protocol 300 with the multi-role automation host 104. The host control application 120 then sends a host provisioning message 302. In response, the multi-role automation host 104 proceeds to query 304 its configuration to identify any (profiled) device(s) of the appropriate type to be supported / controlled by the automation hub 112. Assuming at least one profiled device of the appropriate type is identified, the multi-role automation host 104 transmits an add bridged device to hub message 306 to the automation hub 112 via the bridge 106.

[0036] Subsequently, the host control application 120 issues a control device message 308 to the multi-role automation host 104, which responds by exerting control 310 on the simplified device to which the message 308 is directed. In contrast, when the hub control application 122 issues a control device message 312 directed to the simplified device, the automation hub 112 responds by forwarding a control request message 314 to the multi-role automation host 104. In turn, the multi-role automation host 104 responds by exerting control 316 on the simplified device to which the message 312 is directed.

[0037] Figure 4A message flow diagram is shown that illustrates an example of lower-level messaging that occurs when a hub control application 122 and an automation hub 112 exert control over a simplified device 108a that has been made visible through a multi-role automation host 104 as described above. In this example, the automation hub 112 represents an Apple TV®, HomePod™, or iPad® running HomeKit®. The hub control application 122 issues an HMActionSet execute message 400 to the automation hub 112, which in turn issues a device characteristic write message 402 to the multi-role automation host 104. The device characteristic write message 402 may operate to, for example, dim a light.

[0038] In response, the multi-role automation host 104 locates a profile 404 corresponding to the device to which the message 402 was directed, and then sends a simplified command 406 to the device 108a. The simplified command 406 is substantially equivalent to the message 402, but is prepared by the multi-role automation host 104 in a form that the device 108a expects and understands. After receiving and executing the simplified command 406, the device 108a sends a command response 408 to the multi-role automation host 104, which in turn sends a feature update message 410 to the automation hub 112.

[0039] Now turn to Figure 5 , shows an automation environment 500. In general, Figure 5 Each component or device shown in Figure 1 The multi-role automation hub 512 in its role as a hub has the same structure and functionality as the automation hub 112 ( Figure 1 ) functions in a substantially similar manner. However, the multi-role automation hub 512 generates an enhanced on-screen display (EOSD) for the television 524 in an additional role.

[0040] In a preferred embodiment, as mentioned above, the automation hub 112 can be based on an Apple TV® running HomeKit®. While the Apple TV® includes a limited screen display that allows the user to select to play audio / video or turn on / off the attached TV, the limited screen display is not capable of controlling other devices. However, the Apple TV® does include high performance graphics processing capabilities.

[0041] According to one aspect of the present invention, an application running on Apple's tvOS® is provided and utilizes high-performance graphics processing built into Apple TV® to generate a video such as Figures 6A-6DBy placing the Apple TV® in lock mode, the Apple TV® will automatically launch the application that generated the EOSD 600.

[0042] like Fig. 6A As shown in FIG. 6 , EOSD 600 may be formatted as three horizontal strips of rotating icons, graphics, images, videos, or other content. For example, strip 602 displays a series of "scenes" corresponding to various rooms or areas of the home. When a user selects a particular "scene" (by using user device 516a or 516b or user remote control 518a or 518b ( Figure 5 ) interacting with EOSD600) to center and select a corresponding image, previously defined conditions or states of lighting, equipment, shades, media or any other resources under the control of the multi-role automation host 504 or multi-role automation hub 512 will be set in the selected room or area.

[0043] The ribbon 604 of the EOSD 600 displays icons corresponding to various systems and services. When the user selects the centered “Security” icon, the EOSD 600 transitions to Figure 6B , where band 608 displays live feeds from several closed-circuit security cameras. Figure 6C As shown in FIG. 1 , when the user selects the “Lighting” icon in the center, the EOSD 600 changes to Fig.6D , where band 610 displays a real image of the current lighting conditions of an individual room or lamp along with a vertical slider control. Exemplary techniques for generating and displaying such real images of current lighting conditions are described in U.S. Patent No. 8,296,669 and co-pending application Ser. No. 16 / 123,731, both of which are incorporated by reference above.

[0044] EOSD 600 may also provide advanced video display management, including windowing, tiling, and picture-in-picture, among others. Details regarding the hardware and software used to generate EOSD 600 may be found in the patents and pending patent applications incorporated by reference above.

[0045] Figure 7 A message flow diagram is shown that illustrates an example of lower level messaging that occurs when a multi-role automation hub 512 with an EOSD 600 interacts with a multi-role automation host 504. The host control application 520 issues a control reduced device message 700 to the multi-role automation host 504. Because the message 700 is directed to a reduced device, the multi-role automation host 504 proceeds to execute the reduced device command 702.

[0046] Subsequently, the host control application 520 issues a control authentication device message 704. Because the message 704 is not directed to a profiled device, the multi-role automation host 504 issues a forward request message 706 to the multi-role automation hub 512 via the bridge 506. The multi-role automation hub 512 in turn issues a control request message 708 to the automation hub 112, which proceeds to execute the authentication device command 710.

[0047] According to one aspect of the present invention, an application running on Apple's tvOS® is provided, and the necessary functionality for converting Apple TV® into a multi-role automation host is provided. Other devices controllable by a dedicated remote control may also be suitable for use as a multi-role automation host. In general, the application embodies all necessary automation host functionality, including the ability to identify and control simplified devices. Due to the local functionality of Apple TV®, this type of multi-role automation host can effectively control a mix of simplified devices and authentication devices without an additional hub or other intermediate device. In addition, the capabilities of this type of multi-role automation host can be further enhanced by adding AVB network connectivity. In the case of an Apple TV® multi-role automation host, such connectivity permits music to be distributed directly to speakers, sound bars, or other devices supporting AVB in digital form.

[0048] According to another aspect of the present invention, multiple Apple TVs can be aggregated together to become a shared resource for multiple TVs. Fig. 7A As shown in FIG. 7 , the automation environment 700 includes a set of TVs 712a-712c, each of which is coupled to a video switch 714. A multi-role automation host 716 is coupled to the video switch 714, a plurality of Apple TVs 718a and 718b, and a plurality of hubs with enhanced OSDs 726a and 726b (which are substantially similar to the aforementioned hubs with enhanced OSD 512 ( Figure 5 )’s center).

[0049] In general, in response to commands from the multi-role automation host 716, the video switch 714 functions to establish and tear down video signal connections or paths between each TV in the TV 712a-712c and each Apple TV in the Apple TV 718a-718b or a hub with an enhanced OSD 726a-726b. The multi-role automation host 716 can also command the video switch 714 to establish a video signal connection between a single Apple TV 718a-718b and multiple TVs 712a-712c. Similarly, the multi-role automation host 716 can also command the video switch 714 to establish a video signal connection between a single hub with an enhanced OSD 726a-726b and multiple TVs 712a-712c.

[0050] Figure 7B is a diagram illustrating the enhanced OSD required to be displayed on one of the TVs 712a-712c. Fig. 7A 762 (i.e., the specific Apple TV associated with token Y). This is followed by a Start Stream 1 message 734 from the multi-role automation host 716 to the video transmitter 758 of menu resource Y, which in turn is followed by a Start Stream 1 message 736 with the resized video to the current transmitter 764 used by zone X.

[0051] Next, the multi-role automation host 716 sends a message 738 to the zone X video receiver 760, which sets up the picture-in-picture and allocates the appropriate transmitter. At this point, the enhanced OSD appears in the picture-in-picture 740. The user OSD navigation message 742 is then transmitted from the user remote control 724 to the menu resource Y 762. This is followed by a cancel or select message 744 from the user remote control 724. If the cancel message 744 is transmitted, an OSD deactivate message 746 is transmitted from the menu resource Y to the multi-role automation host 716. This is followed by the menu token Y being returned to the queue 748 because the OSD session has ended.

[0052] Subsequently, a message 750 is transmitted from the multi-role host 716 to restore the full screen video of the original transmitter. This is followed by a stop stream 1 message 752 to the current transmitter 764, a stop stream 1 message to the video transmitter 758 of the menu resource Y, and the restoration 756 of the full screen video.

[0053] Additionally, similar to the enhanced OSD example described above, other services available on Apple TV 718a-718b can be shared with multiple TVs 712a-712c. Thus, by placing the Apple TV in lock mode and streaming the selected video over IP, using the video switch 714 to establish the appropriate video connection, services such as Netflix, weather, or many other services can be shared with multiple TVs 712a-712c.

[0054] According to another aspect of the present invention, a ceiling or wall mountable speaker (such as disclosed in co-pending application serial number 16 / 254,245 and incorporated by reference above), or a bookshelf or other speaker is provided with enhanced functionality and becomes a multi-role speaker. The previously disclosed speakers include a LAN / AVB port through which both digital audio and power over Ethernet (PoE) are received. Figure 8 As shown in FIG. 8 , the multi-role speaker 800 includes a LAN / AVB port 802 coupled to a microprocessor 804. A microphone 806 is coupled to the microprocessor 804, as is a digital signal processor (DSP) 808. An amplifier 810 is coupled to the DSP 808 and to a speaker 812. The multi-role speaker 800 also includes a WISA module 814 and a WiFi dual-band module 816, both of which provide wireless connectivity. Through its WISA module 814 or LAN / AVB port 802, the multi-role speaker 800 can distribute digital audio in two forms: 16-bit, 44.1kHz form (used in Airplay 2) or a higher fidelity 24-bit, 96kHz form.

[0055] Through its WiFi dual-band module 816, the multi-role speaker 800 provides support for Airplay 2 and serves as a wireless access point. In order to provide full-house audio, speakers are usually distributed throughout most rooms or other areas of the house. By providing wireless network connectivity for such speakers, they become multi-role speakers, and robust wireless network coverage is achieved without the necessity of a separate wireless access point. Through its WISA module 814, the multi-role speaker 800 can be paired with a WISA speaker 820 to provide stereo. The paired WISA speaker 820 includes a microcontroller 822 coupled to a WISA module 830 and a DSP 824. The DSP 824 is coupled to an amplifier 826, and the amplifier 826 is then coupled to a speaker 828. In general, the paired WISA speaker 820 receives digital audio from the multi-role speaker 800 via the WISA module 830.

[0056] In the automation environment 100 ( Figure 1 ), the multi-role speaker 800 can serve in any combination of the following four roles: audio speaker; multi-role automation host 104 ( Figure 1 ); wireless access point; and pairing with WISA speaker 820.

[0057] Fig. 9 A multi-role amplifier 900 is shown, which includes a microprocessor 902 coupled to a DSP 904, which in turn is coupled to an amplifier 906. The microprocessor 902 is coupled to a LAN / AVB port 908, an analog input 910, a TosLink input 912, and a control port 914, respectively, which may include an RS232, IR, or other control port. An AC / DC power supply 916 provides power to the multi-role amplifier 900. The multi-role amplifier 900 provides an RCA preamp output 920 and can be used to drive an external passive speaker 918.

[0058] In the automation environment 100 ( Figure 1 ), the multi-role amplifier 900 can be used as both an audio amplifier and a multi-role automation host 104 ( Figure 1 ).

[0059] The foregoing description has been directed to specific embodiments of the present invention. However, it will be clear that other variations and modifications may be made to them while achieving some or all of the advantages of the described embodiments. For example, it is clearly contemplated that the teachings of the present invention may be implemented as software, including computer-readable media with program instructions executed on a computer, hardware, firmware, or a combination thereof. Therefore, this description will be understood only as an example, and does not otherwise limit the scope of the present invention. Therefore, the purpose of the appended claims is to cover all such variations and modifications as fall within the true spirit and scope of the present invention.

Claims

1. An automation control device (100), the device comprising a multi-role automation host (104), the multi-role automation host having connectivity for communicating with a first user device (116a, 118a), the first user device having a command set (300) to which the multi-role automation host responds, the device being characterized by The multi-role automation host comprises an automation bridge (106) that communicates bidirectionally with one or more devices including a first automation hub (112) using a predetermined control protocol, the automation bridge converting control protocol messages received from the first automation hub into an appropriate form for processing by the multi-role automation host, the multi-role automation host operating in a first role, wherein the multi-role automation host directly controls one or more reduced devices (108a, 108b) without the automation bridge (106), each of the one or more reduced devices having an associated profile accessible by the multi-role automation host, each of the associated profiles comprising information describing a set of device commands to which the associated reduced device responds; The multi-role automation host operates in a second role, wherein the multi-role automation host directly controls a first set (114a) of one or more devices through the automation bridge, wherein each device is authenticated as compliant with the predetermined control protocol, and indirectly controls a second set (114b, 114c) of one or more devices through the first automation hub (112) through the automation bridge, wherein each device is authenticated as compliant with the predetermined control protocol; as well as A second user device (116b, 118b) has a command set (312, 400) to which the first automation hub responds, the second user device in conjunction with the first automation hub and the multi-role automation host effectively controlling at least one of the one or more profiled devices.

2. A method for controlling a device in an automated environment (100), the method comprising the following steps: using a user device (116a) to discover (300) a multi-role automation host (104) including an automation bridge (106) and networked with an automation hub (112); determining whether a configuration (304) associated with the multi-role automation host includes one or more profiled devices (108a, 108b) of an appropriate type to be controlled by the automation hub via the automation bridge; sending a message (306) from the multi-role automation host to the automation hub through the automation bridge, the message (306) identifying at least one profiled device to be controlled by the automation hub; and The at least one simplified device is controlled (316) using the automation hub in conjunction with the multi-role automation host, wherein the multi-role automation host directly controls one or more simplified devices (108a, 108b) without an automation bridge (106).

Images