Application scalability for smart device control

Abstract

Features described herein relate to extending functionality of an application executing on an electronic device. Device feature specifications associated with the smart device may be used to generate a layout for defining a graphical user interface page of the application and bind at least one device feature of the set of device features to an application extension template of the application. Application extensions may be generated based on an application extension template populated with layout definition values and binding values. Application extensions may be provided to applications. The application extension may cause a graphical user interface page of the application to be displayed on a display of the electronic device in a defined layout and an element bound to the layout and configured to cause the smart device to be controlled based on device characteristics.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 444,128, filed on February 8, 2023, the entire disclosure of which is hereby incorporated by reference for all purposes as if fully set forth herein. Background Art

[0002] Users find it convenient to use smart devices (e.g., smart thermostats) to automate their homes. Control of these smart devices is typically performed using one or more applications installed on the user's mobile phone or other computerized device. Users typically need to install a different application for each smart device they want to control. In some cases, a device manufacturer that offers more than one type of smart device may provide a home automation application that can collectively control the smart devices manufactured by the manufacturer. Therefore, it may be desirable to integrate smart devices into applications for controlling the smart devices. Summary of the Invention

[0003] The embodiments described herein relate to smart devices, and more particularly, to systems and methods for extending the functionality of applications.

[0004] In some embodiments, a method for extending the functionality of an application executed on an electronic device includes: accessing an extension container, the extension container including an application extension template populated with layout definition values ​​and binding values, the layout definition values ​​being configured to define a layout of a graphical user interface page of the application, the binding values ​​being configured to link device traits to the graphical user interface page, wherein each of the device traits corresponds to a function of a smart device that wirelessly communicates with the electronic device; and providing an extension including the extension container to the electronic device.

[0005] In some embodiments, the extension, when implemented by the application, is configured to cause a graphical user interface page of the application to be displayed on a display of the electronic device in the layout, wherein the layout includes a plurality of graphical user interface elements configured to cause the smart device to be controlled based on device characteristics.

[0006] In some embodiments, the binding value is configured to link the device feature to a plurality of graphical user interface elements, each of the plurality of user interface elements being configured to cause a function of the smart device to be controlled based on a device feature included in the device features.

[0007] In some embodiments, before providing the extension to the electronic device, the method further includes: receiving a request for the application extension from the electronic device.

[0008] In some embodiments, the application extension template is populated with layout definition values ​​and binding values ​​based on input received by a user.

[0009] In some embodiments, the device characteristics are included in a device characteristics specification associated with the smart device.

[0010] In some embodiments, the extension container is JavaScript based.

[0011] Some embodiments include a system comprising a processing system and at least one computer-readable medium storing instructions that, when executed by the processing system, cause the system to perform some or all of the operations and / or methods disclosed herein.

[0012] Some embodiments include one or more non-transitory computer-readable media storing instructions that, when executed by at least one processing system, cause the system to perform some or all of the operations and / or methods disclosed herein.

[0013] The techniques described above and below can be implemented in a variety of ways and in a variety of contexts. Several example implementations and contexts are provided with reference to the following figures, as described in more detail below. However, the following implementations and contexts are only a few of the many implementations and contexts. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The nature and advantages of various embodiments may be further understood with reference to the following figures. In the accompanying figures, similar components or features may have the same reference numeral. Furthermore, various components of the same type may be distinguished by following the reference numeral with a dash and a second reference numeral that distinguishes the similar components. If only the first reference numeral is used in the specification, the description applies to any of the similar components having the same first reference numeral, regardless of the second reference numeral.

[0015] Figure 1 An example of a system for controlling a smart device according to some implementations of the present disclosure is illustrated.

[0016] Figure 2 An example of a system for extending an application according to some implementations of the present disclosure is illustrated.

[0017] Figure 3A An example of an application extension template according to some implementations of the present disclosure is illustrated.

[0018] Figure 3B Illustrated is an example of a populated application extension template according to some implementations of the present disclosure.

[0019] Figure 4 An example of a process for extending the functionality of an application is illustrated according to some implementations of the present disclosure.

[0020] Figure 5A An example of a process for extending the functionality of an application is illustrated according to some implementations of the present disclosure.

[0021] Figure 5B Another example of a process for extending the functionality of an application is illustrated according to some implementations of the present disclosure.

[0022] Figure 5C Another example of a process for extending the functionality of an application is illustrated according to some implementations of the present disclosure.

[0023] Figure 6 An example of a process for controlling a device using an application with expanded functionality is illustrated in accordance with some implementations of the present disclosure.

[0024] Figure 7 An example system incorporating an electronic device is illustrated according to some implementations of the present disclosure.

[0025] Figure 8 An exemplary embodiment of a smart home environment including various smart home devices according to some implementations of the present disclosure is illustrated. DETAILED DESCRIPTION

[0026] Smart devices have become ubiquitous. They are commonly used in a variety of residential and commercial environments, such as homes and office buildings, to automate tasks. For example, a semi-automated home may include several smart devices, such as smart thermostats, smart appliances, smart TVs, and the like. In some cases, different categories of smart devices may be offered under different brands and / or by different manufacturers. For example, a company or brand that specializes in providing smart home appliances, such as smart refrigerators and smart laundry centers, may not offer smart thermostats or smart lights. Setting up, controlling, and managing these smart devices is typically performed using one or more applications installed on one or more electronic devices (such as tablets, mobile phones, home assistant devices, display devices, and the like). Typically, to control all the smart devices in a given ecosystem (such as a user's home), users will ultimately rely on several different applications installed on their electronic devices (e.g., their mobile phones, tablets, and the like), each of which provides a different user experience. In this arrangement, interacting with their smart devices can often be confusing for both technical and non-technical users, and may limit the wider adoption of smart devices and home automation systems.

[0027] In some cases, smart device manufacturers may not have the resources, knowledge and / or interest to develop or provide applications for managing the smart devices they manufacture. In other cases, smart device manufacturers may not have the resources, knowledge or interest to integrate the management of their smart devices with existing smart device management applications. Therefore, these smart device manufacturers may lose the motivation to participate in the home automation ecosystem. Home automation control applications have been proven to be useful for using a single application to manage and control different smart devices. For example, home automation control applications (such as the Google Home application provided by Google LLC) can be used to set up and control several different smart devices. Although these home automation control applications often enable users to integrate new smart devices into their smart environment or smart home ecosystem, the smart devices that can be integrated into these applications are generally limited to specific smart devices and / or smart devices manufactured or provided by specific manufacturers / brands. In addition, even if a smart device provided by one provider is integrated with a home automation control application provided by another provider, the control of the smart device may also be limited to selected functions rather than all of its functions, and / or the ability to control the smart device may be limited in other aspects. Therefore, it may be desirable to provide means for integrating smart devices into applications for setting up, controlling, and managing the smart devices.

[0028] The technology described herein overcomes the aforementioned challenges and / or other challenges by providing systems and methods for extending smart home control applications. Using the technology described herein, device manufacturers / brands can integrate their smart home devices into smart home control applications with minimal software / hardware development effort on the part of the device manufacturer / brand and without modifying the native experience (e.g., look, feel, and operation) of the smart home control application. The cross-platform extensibility mechanism described herein enables the functionality of applications to be extended regardless of the platform on which the application is provided by dynamically providing extensions that enable applications to integrate the setup, control, and management of smart devices that they were not originally developed to setup, control, and manage. Utilizing the technology disclosed herein, both technical and non-technical users can setup, control, and manage their smart devices in an intuitive manner, even if the smart devices are manufactured / provided by different manufacturers / brands, and the device manufacturer / brand does not need to develop / provide applications for setting up, controlling, and managing their smart devices.

[0029] In some embodiments, a method for extending the functionality of an application executed on an electronic device includes: accessing a device feature specification associated with a smart device that wirelessly communicates with the electronic device, wherein the device feature specification includes a device feature set, each device feature in the device feature set corresponding to a function of the smart device; generating an application extension template based on the device feature specification, wherein the application extension template includes a layout definition section for defining a layout of a graphical user interface page of the application and a binding section for binding at least one device feature in the device feature set to the application; receiving an extension container, the extension container including the application extension template populated with a layout definition value and a binding value, the layout definition value being configured to define the layout of the graphical user interface page, the binding value being configured to link a subset of device features of the device feature set to the graphical user interface page; and providing an extension including the extension container to the electronic device.

[0030] In some embodiments, a request for an application extension may be received by a server from an electronic device, and in response, the server system may provide the application extension to the electronic device. The extension container within the application extension may be configured to cause a graphical user interface page of the application to be displayed on a display of the electronic device in a layout that may include elements configured to cause a smart device to be controlled based on device features in a device feature subset. The binding value may be configured to link the device feature subset to the elements, wherein each element may be configured to cause a function of the smart device to be controlled based on the device features included in the device feature subset.

[0031] Although the technology described herein is described with respect to smart devices and applications for setting up, controlling, and managing those devices, the technology described herein is not intended to be limited to such devices and applications and can be applied to any device and application for controlling the device. Additionally, the technology described herein and the aforementioned advantages are not intended to be exhaustive, and other advantages of the described technology will become apparent in the following description.

[0032] Figure 1 An example of a system for controlling a smart device is shown below. Figure 1 As shown, system 100 may include electronic device 102 , network 106 , smart device 108 , and smart device server 110 . Figure 1 The electronic device 102 shown is a smartphone (such as Google ® Pixel ® Phone), but this is not intended to be limiting, and the electronic device 102 may be a tablet computer (such as a Google ® Pixel ® Tablet) and / or Assistant devices (such as Google ®Nest ® Hub or Google ® Nest ® Hub Max). Figure 1 The smart device 108 shown is a smart thermostat (such as Google's Nest Learning Thermostat ® ), but this is not intended to be limiting, and the smart device 108 can be any smart device, such as a smart light, a smart garage door opener, a smart lock, etc. The electronic device 102 and the smart device 108 can be configured to communicate with each other via the network 106 and the smart device server 110.

[0033] The network 106 may be public and / or private and include any type of wireless and / or wired network. Example networks include Ethernet networks, the Internet, Wi-Fi networks, Bluetooth ® Networks, etc. Communications involving the network 106 can be implemented using any variety of networking technologies and / or communication protocols, such as Matter; ZigBee; IrDA; RFID; NFC; WLAN; Z-wave; Wi-Fi and Wi-Fi Direct; UWB; USB; ANT and ANT+; 3G; 4G; 5G; 6G, etc. The smart device server 110 can be a server hosted on a cloud infrastructure, such as Google Cloud provided by Google LLC. In some implementations, the smart device server 110 can be established and managed by the manufacturer / developer of the smart device 108 to facilitate communications with the smart device 108. In other implementations, the smart device server 110 can be established and managed by the developer of an application used to set up, control, and manage the smart device 108.

[0034] In some implementations, the electronic device 102 may execute an application for setting up, controlling, and managing the smart device 108. When executed, the application may present a graphical user interface on the display of the electronic device 102. For example, Figure 1 As shown, executing the application may cause a graphical user interface page 112 of a graphical user interface to be displayed on the display of the electronic device 102. The graphical user interface page 112 may enable a user of the electronic device 102 to interact with the graphical user interface page 112 (e.g., via touch input and / or other input mechanisms through the display) to set up, control, and / or manage the smart device 108. The graphical user interface page 112 may include, but is not limited to, one or more elements 114 for presenting information about the smart device 108 and one or more elements 116 for controlling the smart device 108. For example, Figure 1As shown, elements in element 114 may be used to present the current ambient temperature and the current ambient relative humidity, and elements in element 116 may be used to control the mode and fan speed of the smart device 108 .

[0035] The smart device server 110 can be configured to facilitate communication between the electronic device 102 and the smart device 108. For example, information about the smart device 108 (such as information to be presented in the graphical user interface page 112) can be sent from the smart device 108 to the smart device server 110 via the network 106, where the information can be received and sent to the electronic device 102 via the network 106. Similarly, information for setting up, controlling, and managing the smart device 108 can be sent from the electronic device 102 to the smart device server 110 via the network 106, where the information can be received and sent to the smart device 108 via the network 106. In some implementations, the electronic device 102 can communicate with the smart device 108 without involving the smart device server 110 (e.g., via a WiFi direct connection and / or a Bluetooth connection). Communication between the electronic device 102 and the smart device 108 can be facilitated using the Matter protocol, a unified Internet Protocol-based connectivity protocol for establishing and maintaining an IoT ecosystem.

[0036] Although about Figure 1 The described application can be used to set up, control, and manage a smart thermostat, but it may be desirable to extend the application's functionality to control different smart devices. For example, it may be desirable to extend the application's functionality to set up, control, and manage smart devices manufactured / supplied by different manufacturers / brands, as well as smart devices for which the application was not specifically developed / designed. Among other things, the techniques described herein provide an extensibility mechanism for extending the application's functionality for setting up, controlling, and managing smart devices.

[0037] Figure 2 An example of a system for extending the functionality of an application is illustrated. Figure 2 As shown, the system 200 may include an application server 202, one or more developer devices 218 (also referred to as “developer devices 218”), one or more client devices 222 (also referred to as “client devices 222”), one or more application stores 224 (also referred to as “application stores 224”), one or more smart devices 226 (also referred to as “smart devices 226”), and one or more smart device servers 228 (also referred to as “smart device servers 228”) that communicate with each other via a network 220.

[0038] The network 220 may be any type of network, wired or wireless, that can facilitate communication between components of the environment 200. For example, the network 220 may include one or more public networks, private networks, local area networks, wide area networks, any combination thereof, and the like. Examples of such networks include the Internet, Wi-Fi networks, Bluetooth ® Network 220 communications can be implemented using any networking technology and / or communication protocol. Examples of such technologies and protocols include: Matter; ZigBee; IrDA; RFID; NFC; WLAN; Z-wave; Wi-Fi and Wi-Fi Direct; UWB; USB; ANT and ANT+; 3G; 4G; 5G; 6G; and the like.

[0039] The application server 202 includes an extension console 204 and one or more applications 206 (also referred to as "applications 206"). The applications 206 can be downloaded from the application server 202 via a network 220 and installed on a client device 222. For example, the client device 222 can connect to the network 220, access the application store 224 and select the application 206 for download, and in response, the application server 202 can transmit the selected application 206 to the client device 222 via the network 220, where the selected application can be installed. Once installed, the application 206 can be executed and used to set up, control, and manage the smart device 226. For example, the client device 222 can use the network 220 to connect to the smart device server 228 and send a command to the smart device server 228 to control the smart device 226, and in response, the smart device server 228 can send the command to the smart device 226. The application server 202, the developer device 218, the client device 222, the smart device 226, and the smart device server 228 can use one or more electronic devices such as Figure 7 The system 200 may be implemented as an electronic device 702 (shown in FIG. 1 ) and may be configured with one or more operating systems and software programs that facilitate the techniques described herein. In some implementations, at least one device of the system 200 (such as the client device 222) may be implemented as a mobile phone and / or a computing device (such as a Google Assistant). ® Pixel ® Tablet and Google ® Pixel ® Phone) and / or Assistant devices (such as Google ® Nest ® Hub and Google ® Nest ® Hub Max).

[0040] When executed on a device such as client device 222, application 206 can be configured to provide a graphical user interface for display on a display of client device 222. The graphical user interface can be configured to set up, control, and / or manage smart devices (such as smart device 226). For example, the graphical user interface can include a first set of graphical user interface pages for setting up, controlling, and / or managing a first smart device 226 (such as a smart thermostat) and a second set of graphical user interface pages for setting up, controlling, and / or managing a second smart device 226 (such as a smart light fixture). To set up, control, and / or manage the smart devices, a user of client device 222 can interact with one or more graphical user interface pages, for example, via touch input and / or voice input. In response, application 206 can be configured to cause client device 222 to communicate with smart device server 228, which in turn can communicate with smart device 226.

[0041] In some implementations, the graphical user interface may include graphical user interface pages for installing, initializing, and / or setting up the smart device 226 and / or otherwise integrating the smart device 226 into the application 206 (e.g., to facilitate an out-of-box experience). The graphical user interface may also include graphical user interface pages for changing settings and / or preferences associated with the smart device 226. The graphical user interface may also include graphical user interface pages for managing the smart device 226 (such as creating routines and / or parameters under which the smart device 226 can operate individually, and / or controlling communications between smart devices in a smart device ecosystem).

[0042] like Figure 1 As shown, an example of a smart device is a smart thermostat. In the case of a smart thermostat as an example, one or more graphical user interface pages can be provided for facilitating an out-of-box experience of the smart thermostat (e.g., facilitating installation, initialization, setup, integration, etc.), controlling and / or operating the smart thermostat (e.g., raising the temperature and lowering the fan speed), and changing settings or preferences of the smart thermostat (e.g., displaying the temperature in degrees Celsius). Additionally, one or more graphical user interface pages can be provided for creating routines or parameters under which the smart thermostat can operate alone and / or in combination with other smart devices (e.g., a smart ceiling fan) that are also integrated with the application 206 (e.g., raising the temperature and lowering the fan speed between 11 PM and 6 AM).

[0043] In some implementations, application 206 can be configured to promote out-of-the-box experience, control and / or operate, manage, and / or change the settings or preferences of default smart devices. For example, application 206 can be configured to promote, control and / or operate, manage, and / or change smart devices made by a specific manufacturer, provided by a specific brand side and / or perform a specific function or function set (e.g., home automation type function). Additionally and / or alternatively, application 206 can be configured to promote, control and / or operate, manage, and change smart devices provided by different manufacturers / brand sides and / or perform different functions (e.g., home automation type function and entertainment type function). In some implementations, application 206 can be configured to promote, control and / or operate, manage, and change when the application is stored on application server 202 (e.g., accessed through a cloud portal). In some implementations, the application 206 can be configured to facilitate, control, and / or operate, manage, and change the application after it has been downloaded and installed on the client device 222 (eg, by downloading code, extensions, and / or other content).

[0044] Application 206 may be configured with a library 208, a dynamic layout engine 210, a settings module 212, a control module 214, and a management module 216. Library 208 may include interface elements that may be selected to define the layout of a graphical user interface page of application 206. The interface elements may include various graphical user interface elements, such as menus, buttons, icons, widgets, switches, indicators, and the like. Each graphical user interface page of application 206 may include one or more graphical user interface elements selected from library 208 and spatially positioned with the corresponding graphical user interface page. Additionally, each graphical user interface page may have a different number of graphical user interface elements spatially positioned at different locations within the corresponding graphical user interface page.

[0045] The dynamic layout engine 210 can be configured to generate graphical user interface pages for the application 206 using the user interface elements selected from the library 208. In some implementations, the dynamic layout engine 210 can be configured to generate the graphical user interface pages for the application 206 based on a set of instructions. For example, upon receiving the set of instructions, the dynamic layout engine 210 can be configured to generate one or more graphical user interface pages by executing the instructions, selecting user interface elements from the library 208 in accordance with the instructions, and arranging the selected user interface elements in the one or more graphical user interface pages in accordance with the instructions. In some implementations, the instructions can be provided by the application server 202. In some implementations, the instructions can be based on input received by the developer device 218 and provided to the application server 202.

[0046] In some implementations, the dynamic layout engine 210 can be configured to generate one or more graphical user interface pages based on one or more application extensions. For example, the application server 202 can generate one or more application extensions (where each application extension includes one or more extension containers) and provide those one or more application extensions to the application 206. Each application extension can represent an extension of the functionality of the application 206. Upon receiving the application extension, the dynamic layout engine 210 can decode, interpret, and / or implement the instructions included in the extension container of the application extension, select user interface elements from the library 208 according to the instructions, and arrange the selected user interface elements in one or more graphical user interface pages according to the instructions.

[0047] The graphical user interface pages generated by the dynamic layout engine 210 can be presented on a display of the client device 222. One or more users of the client device 222 can view and interact with the graphical user interface pages to facilitate setting up, controlling, and managing the smart device 226. In some implementations, the dynamic layout engine 210 can generate a set of graphical user interface pages for each smart device that is integrated with and / or will be integrated with the application 206. In some implementations, the dynamic layout engine 210 can generate one or more graphical user interface pages for setting up the smart device 226, one or more graphical user interface pages for controlling the smart device 226, and one or more graphical user interface pages for managing the smart device 226.

[0048] The setup module 212 can be configured to present one or more graphical user interface pages for setting up the smart device 226 on the display of the client device 222. The one or more graphical user interface pages for setting up the smart device 226 can facilitate the initial integration of the smart device 226 with the application 206. For example, a user of the client device 222 may wish to integrate a newly purchased smart device 226 with the application 206 currently executing on the client device 222. To this end, the setup module 212 can present one or more graphical user interface pages that facilitate the initial setup of the smart device 226 using the application 206 (e.g., installing the device, specifying the device type, the model number of the device, and the serial number of the device; connecting to the device; setting initial operating parameters, etc.). To facilitate the setup of the smart device 226, the user can interact with the one or more graphical user interface pages to make one or more selections. For example, the display of the client device 222 can be configured to receive touch input from the user, and the client device 222 can be configured to identify which graphical user interface elements of the graphical user interface pages the touch input corresponds to and generate corresponding commands for setting up the smart device 226. The commands may be packaged in a message and sent to the smart device server 228 of the smart device 226. In response, the smart device server 228 may communicate with the smart device 226 to facilitate setup of the smart device 226 based on the commands.

[0049] The control module 214 can be configured to present one or more graphical user interface pages on the display of the client device 222 for controlling and / or operating the smart device. The one or more graphical user interface pages for controlling and / or operating the smart device 226 can facilitate the control and / or operation of the smart device that has been integrated with the application 206 (e.g., via the settings module 212). For example, the user of the client device 222 may wish to control the previously integrated smart device 226 using the application 206 that is executing on the client device 222. To do so, the control module 214 can present one or more graphical user interface pages that facilitate the control and / or operation of the smart device 226 using the application 206 (e.g., turning on the fan in the case of a smart fan, changing the temperature in the case of a smart thermostat). To facilitate the control and / or operation of the smart device 226, the user can interact with the one or more graphical user interface pages to make one or more selections. For example, the display of the client device 222 may be configured to receive touch input from the user, and the client device 222 may be configured to identify which graphical user interface elements of the graphical user interface page the touch input corresponds to and generate corresponding commands for controlling and / or operating the smart device 226. The commands may be packaged in a message and sent to the smart device server 228 of the smart device 226. In response, the smart device server 228 may communicate with the smart device 226 to facilitate control and / or operation of the smart device 226 based on the commands.

[0050] The management module 216 can be configured to present one or more graphical user interface pages for managing smart devices on the display of the client device 222. The one or more graphical user interface pages for managing smart devices 226 can facilitate changing, modifying, and / or updating the operational settings, preferences, and / or parameters of smart devices that have been integrated with the application 206 (e.g., via the settings module 212). For example, a user of the client device 222 may wish to modify the settings or operational parameters of a previously integrated smart device 226 using the application 206 currently executing on the client device 222. To this end, the control module 214 can present one or more graphical user interface pages that facilitate management of the smart device 226 using the application 206 (e.g., changing the backlight display of a smart thermostat from bright to dark). To facilitate management of the smart device 226, the user can interact with the one or more graphical user interface pages to make one or more selections. For example, the display of the client device 222 can be configured to receive touch input from the user, and the client device 222 can be configured to identify which graphical user interface elements of the graphical user interface pages the touch input corresponds to and generate corresponding commands for managing the smart device 226. The commands may be packaged in a message and sent to the smart device server 228 of the smart device 226. In response, the smart device server 228 may communicate with the smart device 226 to facilitate management of the smart device 226 based on the commands.

[0051] Extension console 204 can be configured to generate one or more application extensions for extending the functionality of application 206. Each application extension can include an application extension container, and each application extension container can include an application extension template populated with layout definition values ​​and binding values. The layout definition values ​​of the application extension template can be configured to define the layout of one or more graphical user interface pages of application 206. The binding values ​​of the application extension template can be configured to link device features of smart device 226 to the one or more graphical user interface pages. As discussed above, each graphical user interface page can have a layout including one or more graphical user interface elements, one or more of which can be interacted with to control smart device 226. Smart device 226 can be controlled based on device features associated with smart device 226. In some implementations, the binding values ​​of the application extension template can be configured to link device features of smart device 226 to graphical user interface elements of one or more graphical user interface pages, such that at least one of the graphical user interface elements can be interacted with to control functionality of the smart device.

[0052] To generate an application extension, the extension console 204 can be configured to access one or more device feature specifications associated with one or more smart devices. The device feature specifications for a smart device can include a set of device features for the smart device, where each device feature in the set corresponds to a characteristic and / or functionality of the smart device. In some implementations, the device feature specifications can be received from the smart device 226 and / or stored on the application server 202. For example, the application server 202 can be configured to communicate with the smart device server 228 via the network 220 to retrieve the device feature specifications for the smart device 226 and store them on the application server 202 so that they are accessible. In some implementations, the device feature specifications can be received from the client device 222 and / or the developer device 218. For example, the application server 202 can be configured to communicate with the client device 222 and / or the developer device 218 via the network 220 to retrieve the device feature specifications for the smart device 226 and store them on the application server 202 so that they are accessible. In some implementations, the device feature specifications can be accessed upon receipt by the application server 202, and / or at a later time.

[0053] The extension console 204 may be further configured to generate one or more application extension templates based on the one or more device feature specifications. The application extension templates may be generated for the smart device 226 based on the device feature specifications of the smart device 226. In some implementations, the application extension templates may be implemented in a programming language such as JavaScript. Figure 3A An example of applying the extension template 302 is shown. Figure 3A As shown, the application extension template 302 may include a layout definition section 304 for facilitating definition of a layout of one or more graphical user interface pages of the application 206 and a feature binding section 306 for facilitating binding (linking or associating) at least one device feature in the device feature set of the smart device 226 to the application 206.

[0054] Layout definition section 304 may be configured to facilitate definition of a layout of one or more graphical user interface pages by identifying spatial locations in the one or more graphical user interface pages where user interface elements from library 208 may be placed. Figure 3AAs shown, for a given GUI page, the layout definition section 304 can identify a first GUI element, a second GUI element, and a third GUI element (e.g., Element 1, Element 2, Element 3) as well as the positioning (e.g., Position) and dimensions (e.g., Width, Height) of those GUI elements. In this way, an extension developer or user can facilitate defining one or more layouts for one or more GUI pages of the application 206.

[0055] The feature binding section 306 can be configured to facilitate binding at least one device feature in the device feature set of the smart device 226 to the application 206 by identifying the device feature and associating the device feature with the graphical user interface element identified in the layout definition section 304. For example, Figure 3A As shown, for a given set of device features, the feature binding section 306 can identify a first device feature, a second device feature, and a third device feature (e.g., Trait 1, Trait 2, and Trait 3) and bind those identified device features to the first, second, and third graphical user interface elements identified in the layout definition section 304. In this manner, an extension developer or user can facilitate linking, associating, or binding one or more device features of the smart device 226 to one or more of the one or more graphical user interface elements of the application 206. In some implementations, binding device features to graphical user interface elements of a graphical user interface page of the application 206 enables the application 206 to present information about the smart device 226 and facilitate control of the smart device 226. For example, in the case of a smart thermostat, the application 206 can be configured to present the heating temperature to which the smart thermostat is set in a first spatial location of the graphical user interface page and facilitate control of the smart thermostat (e.g., changing the temperature) in a second spatial location of the graphical user interface page as defined by the layout definition section 304.

[0056] In some implementations, the extension console 204 can be further configured to provide the one or more device feature specifications and the one or more application extension templates 302 to the developer device 218, where the one or more application extension templates can be populated with values. For example, the extension console 204 can be configured to package the one or more device feature specifications and the one or more application extension templates 302 in one or more application packages and transmit the one or more application packages to the developer device 218 via the network 220. The developer device 218 can then be configured to populate the one or more application extension templates 302 to form one or more populated application extension templates 308. Each application extension template 302 can be populated with layout definition values ​​and binding values. For example, Figure 3B As shown, the developer device 218 can populate the application extension template 302 with layout definition values ​​and binding values. In some implementations, the extension console 204 can be further configured to provide access to device feature specifications and one or more application extension templates, wherein the one or more application extension templates can be populated by accessing the application server 202 (e.g., based on input received through a user interface of the application server 202). The application extension template 302 can be provided to developers and / or users using a graphical user interface page of the extensibility console 204 and / or the developer device 218.

[0057] The developer and / or user can then populate the application extension template 302. The developer and / or user can populate the application extension template 302 by selecting the user interface elements that should be included in the graphical user interface page and the locations where those user interface elements should be spatially located within the graphical user interface page. For example, the developer and / or user can use the layout definition section 304 of the application extension template 302 to indicate that a dial icon should appear on the graphical user interface page at a specific size and in a specific column and row on the graphical user interface page. The developer and / or user can also populate the application extension template 302 by selecting binding properties and device features in the device feature specifications of the smart device 226 that should be linked, bound, and / or associated with the selected user interface elements. In some implementations, the developer and / or user can populate the application extension template 302 by manipulating the graphical user interface page of the extensibility console 104. The graphical user interface page of the extensibility console 104 can include menus, buttons, icons, drag-and-drop functionality, and other features and / or functionality that enable the developer and / or user to manipulate the graphical user interface page and populate the application extension template 302.

[0058] The layout definition value may be configured to define a graphical user interface element identified in the application extension template 302 (eg, Figure 3A208 . For example, Figure 3B As shown, a graphical user interface element representing a dial, a graphical user interface element representing a minimum amount (e.g., a minus sign), and a graphical user interface element representing a maximum amount (e.g., a plus sign) can be included in library 208 and can be identified as layout definition values ​​of the first graphical user interface element, the second graphical user interface element, and the third graphical user interface element of the application extension template 302.

[0059] The binding value may be configured to link the device feature of the smart device to one or more graphical user interface pages of the application 206, and more specifically, to link the device feature to one or more graphical user interface elements identified in the layout definition section 304. For example, Figure 3B As shown, a temperature device feature corresponding to a set heating or cooling temperature of smart device 226 can be linked to a dial GUI element, a minimum device feature corresponding to a minimum temperature of a temperature range, and a maximum device feature corresponding to a maximum temperature of the temperature range.

[0060] In some implementations, once one or more application extension templates are populated, the developer device 218 can be configured to package the one or more populated application extension templates into one or more extension containers and transmit the one or more extension containers to the application server 202 via the network 220. Upon receiving the extension container, the extension console 204 can be configured to store the extension container on the application server 202 where it can be accessed and / or generate an extension that includes the extension container. For example, the application server 202 can be configured to store the extension container and generate an extension that includes the selected extension container. The extension container and / or the extension can be based on a programming language such as JavaScript. The extension for the smart device 226 can be stored in the application server 202, where it can be retrieved later (described later).

[0061] In some implementations, the extension console 204 can be configured with one or more graphical user interface pages that enable developers and / or users to develop and manage application extensions. In some implementations, the extensibility console 204 can be an application programming interface of the application server 202. In other implementations, the extensibility console 204 can be configured to provide developers and / or users with a software development kit that enables them to develop and manage extensions on another device (i.e., a device other than the application server 202) (such as the developer device 218) and transfer (e.g., send or upload) the application extension to the application server 202 using the extensibility console 204. In some implementations, once an extension is generated or otherwise completed, the extension can be stored in the application server 202 in association with the application 206 for which the extension was developed. For example, the application server 202 can store a table or list that associates the developed application extension with the application 206. In this manner, developers and / or users interacting with the graphical user interface pages of extensibility console 204 can view application extensions for application 206 and manage them (eg, rename, delete, edit, update, etc.).

[0062] Once the application extension has been generated, developed, and / or otherwise completed, the application server 202 can validate the application extension. In some implementations, validating the application extension includes verifying that the application extension meets one or more criteria set by the developer, user, client, customer, application server 202, application store 224, combinations thereof, and the like. In other implementations, validating the application extension includes verifying that the application extension is complete and does not contain any errors that would cause the application server 202, application store 224, application 206, client device 222, and / or smart device 226 to malfunction and / or operate improperly. Additionally, validating the application extension includes verifying that the application extension does not include any malicious components (e.g., viruses, malware, spyware, etc.) and / or other content that would inhibit the functionality of the application server 202, application store 224, application 206, client device 222, smart device 226, network 220, combinations thereof, and the like. In some implementations, validating the app extension includes checking whether the app extension complies with the policies and / or standards of the app store 224 , privacy laws, those set by the manufacturer / provider of the smart device 226 , combinations thereof, and the like.

[0063] In some implementations, the application server 202 can be configured to receive one or more requests for application extensions for the application 206. In some implementations, the request for the application extension can be received from the client device 222 and / or the application 206 executing on the client device 222. In some implementations, the application 206 can generate the request and provide the request to the application server 202 over the network 220 in response to the client device 222 and / or the application 206 detecting that the smart device 226 is communicating with the client device 222 and / or the application 206 and / or new functionality and / or features have been added to the smart device 226.

[0064] For example, Figure 4 As shown, Figure 4 Illustrating an example of a process 400 for extending the functionality of an application, a client device 402 executing an application may connect to a smart device 404 and, in response, request an application extension for the application from a server 406. The application extension may be a JavaScript-based extension that dynamically extends the functionality of the application. The extended functionality may enable the application to facilitate the setup, control, and management of the smart device 404. The extension may be generated by a server 406, which may receive device characteristics from the smart device 404, generate an application extension template based on the device characteristics, and send the application extension template to a developer device 408. In response, the developer device 408 may populate the extension template and send the extension template to the server 406, where it may be used to generate the extension. Once the application on the client device 402 receives the extension container for the application extension, the application's dynamic layout engine, as described above, may generate a graphical user interface page based on the extension container. For example, as described above, the dynamic layout engine may decode and implement instructions included in the extension container and, based on the instructions, arrange selected user interface elements from the application's user interface library within the graphical user interface page.

[0065] In this way, smart devices can be integrated into applications by extending the functionality of the application with minimal software / hardware development effort by the device manufacturer / brand and without modifying the native experience (e.g., look, feel, and operation) of the application. Additionally, the features and techniques described herein are cross-platform and can extend the functionality of any application executed on any operating system and on any device. Utilizing the techniques disclosed herein, both technical and non-technical users can set up, control, and manage their smart devices in an intuitive manner and in a manner that violates any app store policies, privacy laws, and other regulations that may exist, even if the smart device is manufactured / provided by a different manufacturer / brand. Additionally, the features and techniques described herein enable smart device setup, control, and management without the need for manufacturers / brands to develop / provide specialized applications for setting up, controlling, and managing their smart devices.

[0066] The foregoing features have been described with respect to smart home applications and controlling smart thermostats; however, this is not intended to be limiting, and the techniques described throughout can be applied to other applications and other smart devices. In some implementations, the techniques throughout can be applied to operating systems of electronic devices and components of electronic devices. Examples of smart devices include, but are not limited to, assistant devices (e.g., Google Assistant, etc.). ® Nest ® Hub; Google ® Nest ® Hub Max); home automation controllers (e.g., controllers for alarm systems, thermostats, lighting systems, door locks, electric gates, etc.); home appliances (e.g., smart speakers, smart TVs, streaming sticks (such as Google Assistant, etc.); ® Chromecast ®) , home theater systems, smart refrigerators, smart dishwashers, smart washing machines, smart dryers, smart air purifiers, smart vacuum cleaners, smart lamps, etc.). The foregoing implementations are not intended to be limiting, and the smart devices may be implemented as any kind of device that can be configured to be controlled by another device using part or all of the methods disclosed herein.

[0067] Figure 5A An example process 500A for extending the functionality of an application is illustrated. The process 500A may be performed by, for example, Figure 2 The application server 202 shown and / or a server such as Figure 7 The electronic device 702 is shown as being implemented in the electronic device. Additionally, the process 500A can be implemented in software or hardware or any combination thereof.

[0068] At block 502, a device feature specification is accessed. In some implementations, the device feature specification is associated with a smart device that wirelessly communicates with the electronic device. In some implementations, the device feature specification includes a device feature set, where each device feature in the device feature set corresponds to a characteristic and / or functionality of the smart device. In some implementations, the device feature specification is received from the smart device by a server, such as application server 202, and / or is stored on the server, where the device feature specification can be accessed. In some implementations, the device feature specification is received from a developer device by application server 202, and / or is stored on the server, where the device feature specification can be accessed.

[0069] At block 504, an application extension template is generated. In some implementations, the application extension template is generated based on a device feature specification. For example, a stored device feature specification can be accessed and used to generate the application extension template. In some implementations, the application extension template includes a layout definition section for defining the layout of a graphical user interface page of the application and a binding section for binding at least one device feature from a device feature set to the application. In some implementations, the application extension template is populated with layout definition values ​​and binding values ​​based on input received through a user interface of the developer's device. In some implementations, the application extension template is generated in a programming language such as JavaScript.

[0070] At block 506 , the application extension template is provided to an electronic device, such as a developer device.

[0071] At block 508, an extension container is received. In some implementations, the extension container is received from a developer device. In some implementations, the extension container includes an application extension template populated with layout definition values ​​and binding values. The layout definition values ​​can be configured to define a layout of a graphical user interface page, and the binding values ​​can be configured to link a subset of device features in a device feature set to the graphical user interface page. In some implementations, the binding values ​​are configured to link the subset of device features to a plurality of graphical user interface elements, wherein each of the plurality of graphical user interface elements is configured to cause a function of the smart device to be controlled based on the device features included in the subset of device features. In some implementations, the extension container is generated using and / or is based on a programming language such as JavaScript.

[0072] At block 510, the expansion container is stored. In some implementations, the expansion container is stored upon receipt. In some implementations, the stored expansion container is accessed.

[0073] Figure 5BAnother example process 500B for extending the functionality of an application is illustrated. The process 500B may be performed by, for example, Figure 2 The application server 202 shown and / or a server such as Figure 7 The process 500B is implemented in the electronic device of the illustrated electronic device 702. Additionally, the process 500B can be implemented in software or hardware or any combination thereof.

[0074] At block 512, an extension container is accessed. In some implementations, the extension container is accessed from a storage device of a server, such as a cloud-based server and / or an application server. In some implementations, the extension container is accessed from an electronic device, such as a developer device. In some implementations, the extension container is accessed in response to receiving a request for an application extension. In some implementations, the extension container is accessed upon receiving the extension container from an electronic device, such as a developer device.

[0075] At block 514, an application extension is generated. In some implementations, the application extension is generated in response to receiving a request for the application extension. In some implementations, the application extension is generated when an extension container is accessed. In some implementations, the application extension is generated by selecting an extension container, accessing the selected extension container, and forming the application extension using the accessed extension container. In some implementations, the application extension can be based on a programming language such as JavaScript.

[0076] At block 516, the application extension is stored. In some implementations, the extension container is stored upon generation. In some implementations, the stored application extension can be accessed and provided to an electronic device, such as a client device.

[0077] Figure 5C Another example process 500C for extending the functionality of an application is illustrated. The process 500C may be performed by, for example, Figure 2 The application server 202 shown and / or a server such as Figure 7 The process 500C is implemented in the electronic device of the illustrated electronic device 702. Additionally, the process 500C can be implemented in software or hardware or any combination thereof.

[0078] At block 518, a request for an application extension is received. In some implementations, the request for the application extension is received from an electronic device, such as a client device. In some implementations, the request for the application extension is received from an application executing on the electronic device. In some implementations, the application generates the request and provides it to the application server in response to the electronic device and / or application detecting that a smart device is in communication with the electronic device and / or application and / or new functionality and / or features have been added to the smart device.

[0079] At block 512, the application extension is provided. In some implementations, the application extension is provided in response to receiving a request for the application extension. In some implementations, the application extension is provided to an electronic device. In some implementations, the application extension and / or extension container is configured to, when implemented, cause a graphical user interface page of the application to be displayed on a display of the electronic device in a layout. In some implementations, the layout includes a plurality of graphical user interface elements that can be configured to cause the smart device to be controlled based on device features from a subset of device features.

[0080] Figure 6 An example process 600 is illustrated for controlling a device using an application with extended functionality according to some implementations of the present disclosure. The process 600 may be performed by a client device such as the client device 222 and / or a client device such as Figure 7 The electronic device 600 is shown as being implemented in the electronic device 702. Additionally, the process 600 can be implemented in software or hardware or any combination thereof.

[0081] At block 602, a request for an extension is sent. In some implementations, the request for the extension is sent in response to receiving a request to facilitate an out-of-box experience, control and / or operation, and change settings or preferences of a smart device. In some implementations, the request is generated by the smart device and / or a user of the smart device.

[0082] At block 604, an extension is received. In some implementations, the extension is received in response to a request for the extension. The extension may be received from a server. The extension may include an application extension template having a layout definition section of the application extension template populated with at least one value and a binding section of the application extension template populated with at least one value. The application extension template may be generated by the server based on a device feature specification. The layout of the graphical user interface page may be defined by the layout definition section, and at least one feature in the device feature specification may be bound by the binding section. The device feature specification may be associated with a smart device and received by the server from the smart device. The device feature specification may include device features, each device feature corresponding to and / or characterizing one or more functions of the smart device. The extension container may be populated with at least one value in the layout definition section and at least one value in the binding section based on input received by the developer device from the extension developer at the server. The extension container may be based on JavaScript.

[0083] At block 606, the graphical user interface page is updated. The graphical user interface page may be updated based on at least one value in the layout definition section. At least one value in the binding section may enable control of the smart device.

[0084] At block 608, input for controlling the smart device is received. The input may be received from the smart device and / or a user interacting with the graphical user interface page. The input may control the smart device based on one or more functions of the smart device as characterized by the device feature specification.

[0085] At block 610, the input is associated with at least one device feature in the device feature specification.The input may be associated with the at least one device feature based on at least one device being bound by a binding section.

[0086] At block 612, the smart device is controlled. The smart device may be controlled based on at least one device characteristic.

[0087] Figure 7 An example system 700 incorporating an electronic device 702 is illustrated according to some aspects of the present disclosure. The system 700 may include the electronic device 702, a network 720, and a cloud server system 722. The electronic device 702 may include a processing subsystem 704, a sensor subsystem 712, a display subsystem 714, an audio subsystem 716, and a communication subsystem 718. In some implementations, the electronic device 702 may be a portable computing device and / or a mobile phone (such as a Google Assistant). ® Pixel ® Tablet and Google ® Pixel ® Phone) and / or Assistant devices (such as Google ® Nest ® Hub and Google ® Nest ® Hub Max).

[0088] The processing subsystem 704 includes RAM 706, one or more processors 708, and one or more memories 710. The one or more processors 708 can read one or more programs from the one or more memories 710 and execute them using the RAM 706. Each processor in the one or more processors 708 can be of any type, including but not limited to a microprocessor, a microcontroller, a graphics processing unit, a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any combination thereof. In some implementations, the one or more processors 708 may include multiple cores, multiple arrays, one or more coprocessors, and / or one or more local cache memory layers. The one or more processors 708 can execute one or more programs stored in the one or more memories 710 to perform the operations and / or methods (including portions thereof) described herein.

[0089] Each of the one or more memories 710 can be non-volatile and can include any type of memory device that retains stored information when power is removed. Non-limiting examples of memory include electrically erasable and programmable read-only memory (EEPROM), flash memory, or any other type of non-volatile memory. At least one of the one or more memories 710 can include a non-transitory computer-readable storage medium from which the one or more processors 708 can read instructions. A computer-readable storage medium can include an electronic storage device, an optical storage device, a magnetic storage device, or other storage device capable of providing computer-readable instructions or other program code to the one or more processors 708. Non-limiting examples of computer-readable storage media include a disk, a memory chip, a read-only memory (ROM), a RAM, an ASIC, a configured processor, an optical storage device, or any other medium from which a computer processor can read instructions.

[0090] The sensor subsystem 712 may be configured to sense information about the environment in which the electronic device 702 is located. Examples of sensors included in the sensor subsystem 712 include image sensors, ultrasonic sensors, radar sensors, ranging sensors, LiDAR sensors, and the like.

[0091] The display subsystem 714 may be configured to display color images, grayscale images, video frames, depth images, three-dimensional (3D) images, and other content on a display screen of the electronic device 702, and to receive input from a user of the electronic device 702. Examples of display screens included in the display subsystem 714 include liquid crystal displays, light-emitting diode displays, organic light-emitting diode displays, projector displays, touch screen displays, and the like.

[0092] The audio subsystem 716 may be configured to record sounds from the environment surrounding the electronic device 702 and output sounds to the environment surrounding the electronic device 702. Examples of audio devices included in the audio subsystem 716 include microphones, speakers, and other audio / sound transducers for receiving and outputting audio signals and other sounds.

[0093] The communication subsystem 718 can be configured to enable the electronic device 702 to communicate with various wired or wireless networks and other systems and devices. For example, the communication subsystem 718 can enable the electronic device 702 to communicate with a network 720. The network 720 can include one or more private and / or public networks, such as the Internet. The network 720 can be used to enable the electronic device 702 to communicate with a cloud server system 722. In some implementations, the cloud server system 722 can perform some of the processing functions performed by the processing subsystem 704. Additionally or alternatively, the cloud server system 722 can be used to relay notifications generated by the electronic device 702 and / or store data generated by the electronic device 702. For example, alerts and / or notifications generated by the electronic device 702 can be recorded by the cloud server system 722 in association with a user account. Examples of communication devices included in the communication subsystem 718 include wireless communication modules and chips, wired communication modules and chips, chips for communicating over local area networks, wide area networks, cellular networks, satellite networks, fiber optic networks, etc., systems on chips, and other circuit systems that enable the electronic device 702 to send and receive data.

[0094] Although not shown, the electronic device 702 may further include other components that provide various functionalities to the electronic device 702. The other components may include storage devices, power generation / storage devices, input / output (I / O) components, etc. The foregoing configuration of the electronic device 702 is not intended to be limiting, and the electronic device 702 may include other subsystems, devices, and components.

[0095] Figure 8 An embodiment of a smart home environment 800 is illustrated that includes various smart home devices that can be set up, controlled, and managed using applications that have been expanded according to the techniques described herein. The smart home environment 800 includes a structure 850 (e.g., a house, a nursery, an office building, an apartment, a condominium, a garage, or a mobile home) with various integrated devices. It should be understood that devices can also be integrated into a smart home environment 800 that does not include the entire structure 850 (such as an apartment or condominium). Further, the smart home environment 800 can control and / or couple to devices that are outside the actual structure 850. In fact, some of the devices in the smart home environment 800 do not need to be physically within the structure 850.

[0096] It should be understood that a "smart home environment" may refer to a smart environment in a home, such as a single-family home, but the scope of the present teachings is not limited thereto. The present teachings are also applicable, but not limited to, duplexes, townhouses, multi-unit apartment buildings, hotels, retail stores, office buildings, industrial buildings, and more generally any living or working space.

[0097] It should also be understood that while the terms user, customer, installer, homeowner, resident, guest, tenant, landlord, maintenance personnel, etc. may be used to refer to one or more persons taking actions in the context of some specific situations described herein, these references do not limit the scope of the present teachings with respect to the one or more persons performing such actions. Thus, for example, in the context of a single-family residential home, the terms user, customer, purchaser, installer, subscriber, and homeowner may generally refer to the same person, as the homeowner is typically the person who makes the purchasing decision, purchases the unit, and installs and configures the unit, and is also one of the users of the unit. However, in other scenarios, such as a landlord-tenant environment, the customer may be the landlord with respect to the purchase of the unit, the installer may be the local apartment manager, the first user may be the tenant, and the second user may again be the landlord with respect to remote control functionality. Importantly, while the identity of the person performing an action may be closely related to the specific advantages provided by one or more of the implementations, such identity should not be interpreted in the subsequent description as necessarily limiting the scope of the present teachings to specific individuals with those specific identities.

[0098] The depicted structure 850 includes a plurality of rooms 852 at least partially separated from one another by walls 854. The walls 854 may include interior walls or exterior walls. Each room may further include a floor 856 and a ceiling 858. Equipment may be mounted on, integrated with, and / or supported by the walls 854, floor 856, or ceiling 858.

[0099] In some implementations, the integrated devices of the smart home environment 800 include intelligent, multi-sensing, network-connected devices that seamlessly integrate with each other and / or with a central server or cloud computing system in a smart home network to provide a variety of useful smart home functions. The smart home environment 800 can include one or more intelligent, multi-sensing, network-connected thermostats 802 (hereinafter referred to as "smart thermostats 802"), one or more intelligent, network-connected, multi-sensing hazard detection units 804 (hereinafter referred to as "smart hazard detectors 804"), one or more intelligent, multi-sensing, network-connected entryway interface devices 806 and 820, and one or more intelligent, multi-sensing, network-connected alarm systems 822 (hereinafter referred to as "smart alarm systems 822"). Each of these devices can incorporate the functionality of the electronic device 110 / 202.

[0100] In some implementations, one or more intelligent thermostats 802 detect ambient climate characteristics (eg, temperature and / or humidity) and accordingly control the HVAC system 803. For example, a respective intelligent thermostat 802 includes an ambient temperature sensor.

[0101] Smart hazard detectors can detect smoke, carbon monoxide, and / or some other hazard present in the environment. One or more smart hazard detectors 804 can include a thermal radiation sensor for a corresponding heat source (e.g., a stove, oven, other appliance, fireplace, etc.). For example, a smart hazard detector 804 in kitchen 853 includes a thermal radiation sensor for a network-connected appliance 812. The thermal radiation sensor can determine the temperature of the corresponding heat source (or a portion thereof) it is targeting and can provide corresponding blackbody radiation data as output.

[0102] The smart doorbell 806 and / or smart door lock 820 can detect a person approaching or leaving a location (e.g., an exterior door), control doorbell / door lock functionality (e.g., receiving user input from the portable electronic device 866 to actuate the bolts of the smart door lock 820), notify the person of the approach or departure via audio or visual means, and / or control settings on the security system (e.g., activating or deactivating the security system when a resident enters or exits). In some implementations, the smart doorbell 806 includes a camera and is therefore also referred to as a "doorbell camera 806" in this document.

[0103] The smart alarm system 822 can detect the presence of an individual in close proximity (e.g., using a built-in IR sensor), sound an alarm (e.g., through a built-in speaker or by sending a command to one or more external speakers), and send notifications to entities or users inside / outside the smart home environment 800. In some implementations, the smart alarm system 822 also includes one or more input devices or sensors for verifying the identity of the user (e.g., a keypad, a biometric scanner, an NFC transceiver, a microphone), and one or more output devices (e.g., a display, a speaker). In some implementations, the smart alarm system 822 can also be set to an armed mode so that detection of a trigger condition or event causes an alarm to be sounded unless a disarming action is performed. Each of these devices can incorporate the functionality of the electronic device 110 / 202.

[0104] In some implementations, the smart home environment 800 includes one or more smart, multi-sensing, network-connected wall switches 808 (hereinafter referred to as "smart wall switches 808") and one or more smart, multi-sensing, network-connected wall outlet interfaces 810 (hereinafter referred to as "smart wall outlets 810"). The smart wall switch 808 can detect ambient lighting conditions, detect room occupancy status, and control the power and / or dimming state of one or more light fixtures. In some cases, the smart wall switch 808 can also control the power state or speed of a fan, such as a ceiling fan. The smart wall outlet 810 can detect occupancy of a room or enclosure and control power to one or more wall outlets (e.g., so that the outlets are not powered when no one is home). Each of these types of devices can incorporate the functionality of the electronic device 110 / 202.

[0105] In some implementations, Figure 8 The smart home environment 800 includes a plurality of intelligent, multi-sensing, network-connected appliances 812 (hereinafter referred to as "smart appliances 812"), such as refrigerators, stoves, ovens, televisions, washers, dryers, lamps, stereos, intercom systems, wall clocks, garage door openers, floor fans, ceiling fans, wall-mounted air conditioners, pool heaters, irrigation systems, security systems, space heaters, window AC units, electric duct vents, and the like. Each of these devices can incorporate the functionality of electronic device 110 / 202. In some implementations, when plugged into an outlet, an appliance can announce itself to the smart home network, such as by indicating what type of appliance it is, and it can automatically integrate with the smart home's controls. This communication by the appliance to the smart home can be facilitated by wired or wireless communication protocols. The smart home can also include a variety of non-communicating legacy appliances 840 (such as an old, conventional washer / dryer, refrigerator, etc.), which can be controlled via the smart wall outlet 810. The smart home environment 800 may further include a variety of partially communicating legacy appliances 842 (such as infrared (“IR”) controlled wall-mounted air conditioners or other IR controlled devices) that may be controlled via IR signals provided by the smart hazard detector 804 or the smart wall switch 808.

[0106] In some implementations, the smart home environment 800 includes one or more network-connected cameras 818 that are configured to provide video monitoring and security in the smart home environment 800. The camera 818 can be installed in a location, such as indoors and mounted to a wall, or can be movable and placed on a surface. Various embodiments of the camera 818 can be installed indoors or outdoors. Each of these types of devices can be combined with the functionality of the electronic device 110 / 202. The camera 818 can be used to determine the occupancy of the structure 850 and / or a specific room 852 in the structure 850, and thus can act as an occupancy sensor. For example, the video captured by the camera 818 can be processed to identify the presence of a resident in the structure 850 (e.g., in a specific room 852). A specific individual can be identified based on, for example, the specific individual's appearance (e.g., height, face) and / or movement (e.g., their walking / gait). The cameras 818 may additionally include one or more sensors (e.g., IR sensors, motion detectors), input devices (e.g., microphones for capturing audio), and output devices (e.g., speakers for outputting audio). In some implementations, the cameras 818 are each configured to operate in a day mode and a low-light mode (e.g., night mode). In some implementations, the cameras 818 each include one or more IR illuminators for providing illumination when the camera is operating in low-light mode. In some implementations, the cameras 818 include one or more outdoor cameras. In some embodiments, the outdoor cameras include additional features and / or components, such as weather resistance and / or solar light compensation.

[0107] The smart home environment 800 may additionally or alternatively include one or more other occupancy sensors (e.g., a smart doorbell 806, a smart door lock 820, a touch screen, an IR sensor, a microphone, an ambient light sensor, a motion detector, a smart night light 870, etc.). In some implementations, the smart home environment 800 includes a radio frequency identification (RFID) reader (e.g., in each room 852 or a portion thereof) that determines occupancy based on an RFID tag located on or embedded in a resident. For example, the RFID reader may be integrated into the smart hazard detector 804.

[0108] The smart home assistant 819 may have one or more microphones that continuously listen to the surrounding environment. The smart home assistant 819 may be able to respond to verbal queries posed by the user that may begin with a trigger phrase. If a camera is integrated as part of the device, the smart home assistant 819 may stream audio and possibly video to a cloud-based server system 864 (which represents Figure 7The smart home assistant 819 may be a smart device through which non-auditory discomfort alerts may be output and / or an audio stream from a streaming video camera may be output. As previously noted, the smart home assistant 819 may incorporate the functionality of the electronic device 110 / 202.

[0109] With network connectivity, one or more of the smart home devices can further allow the user to interact with the device even if the user is not near the device. For example, the user can use a computer (e.g., a desktop computer, laptop computer, or tablet computer) or another portable electronic device 866 (e.g., a mobile phone, such as a smart phone) to communicate with the device. The web page or application can be configured to receive communications from the user and control the device based on the communication, and / or present information about the operation of the device to the user. For example, the user can view the current set point temperature of the device (e.g., a stove) and use the computer to adjust the current set point temperature. The user may be in the structure during this remote communication, or outside the structure.

[0110] As discussed above, users can use network-connected computers or portable electronic devices 866 to control smart devices in the smart home environment 800. In some examples, some or all of the residents (e.g., individuals living in the home) can register their portable electronic devices 866 with the smart home environment 800. This registration can be performed at a central server to authenticate the residents and / or devices as being associated with the home and grant residents permission to use the devices to control smart devices in the home. Residents can use their registered portable electronic devices 866 to remotely control the home's smart devices, such as when the residents are at work or on vacation. Residents can also use their registered devices to control smart devices when they are physically at home, such as when they are sitting on the couch at home. It should be understood that, instead of or in addition to registering portable electronic devices 866, the smart home environment 800 can infer which individuals live in the home and are therefore residents, and which portable electronic devices 866 are associated with those individuals. Thus, the smart home environment can "learn" who the residents are and grant portable electronic devices 866 associated with those individuals permission to control the home's smart devices.

[0111] In some implementations, in addition to including processing and sensing capabilities, the smart thermostat 802, smart hazard detector 804, smart doorbell 806, smart wall switch 808, smart wall outlet 810, network-connected appliances 812, camera 818, smart home assistant 819, smart door lock 820, and / or smart alarm system 822 (collectively, "smart home devices") are capable of communicating data and sharing information with other smart devices, a central server or cloud computing system, and / or other network-connected devices. Data communications can be performed using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, Matter, ZigBee, 3LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, etc.), and / or any of a variety of custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), or any other suitable communication protocol (including communication protocols not yet developed as of the filing date of this document).

[0112] In some implementations, smart devices are used as wireless or wired repeaters. In some implementations, the first smart device in the smart device communicates with the second smart device in the smart device via a wireless router. Smart devices can further communicate with each other via a connection to a network such as the Internet (e.g., network interface 860). Through the Internet, smart devices can communicate with a cloud-based server system 864 (also referred to herein as a cloud-based server system, a central server system and / or a cloud computing system). Cloud-based server system 864 can be associated with a manufacturer, a support entity or a service provider associated with the smart device. In some embodiments, the user can use the smart device itself to contact customer support, rather than needing to use other communication means, such as a phone or an Internet-connected computer. In some implementations, software updates are automatically sent to the smart device (e.g., when available, when purchased or at regular intervals) from the cloud-based server system 864.

[0113] In some implementations, the network interface 860 includes a conventional network device (e.g., a router), and Figure 8The smart-home environment 800 includes a hub device 880 that is communicatively coupled to a network 862, either directly or via a network interface 860. The hub device 880 is further communicatively coupled to one or more of the aforementioned intelligent, multi-sensing, network-connected devices (e.g., the smart devices of the smart-home environment 800). Each of these smart devices optionally communicates with the hub device 880 using at least one radio communication network (e.g., Matter, ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi, and other radio communication networks) available in the smart-home environment 800. In some implementations, the hub device 880 and devices coupled to / to the hub device can be controlled and / or interacted with via an application running on a smartphone, home controller, laptop, tablet, game console, or similar electronic device. In some implementations, users of such a controller application can view the status of the hub device or coupled smart devices, configure the hub device to interoperate with smart devices newly introduced to the home network, debug new smart devices, and adjust or view settings of connected smart devices, etc. In some implementations, the hub device extends the capabilities of low-capability smart devices to match the capabilities of highly capable smart devices of the same type, integrates functionality of multiple different device types (even across different communication protocols), and is configured to streamline the addition of new devices and the debugging of the hub device. In some implementations, the hub device 880 further includes a local storage device for storing data related to or output by the smart devices of the smart home environment 800. In some implementations, the data includes one or more of the following: video data output by a camera device, metadata output by a smart device, setting information for the smart device, usage logs of the smart device, etc.

[0114] In some implementations, the smart home environment 800 includes a local storage device 890 for storing data associated with or output by smart devices in the smart home environment 800. In some implementations, the data includes one or more of the following: video data output by a camera device (e.g., camera 818 or smart doorbell 806), metadata output by the smart device, smart device settings information, smart device usage logs, etc. In some implementations, the local storage device 890 is communicatively coupled to one or more smart devices via a smart home network. In some implementations, the local storage device 890 is selectively coupled to one or more smart devices via a wired and / or wireless communication network. In some implementations, the local storage device 890 is used to store video data when external network conditions are poor. For example, the local storage device 890 is used when the encoding bit rate of the camera 818 exceeds the available bandwidth of the external network (e.g., network 862). In some implementations, local storage device 890 temporarily stores video data from one or more cameras (eg, camera 818 ) before transmitting the video data to a server system (eg, cloud-based server system 864 ).

[0115] exist Figure 8 The exemplary smart home environment 800 further includes and illustrates service robots 868, each of which is configured to perform any of a variety of household tasks in an autonomous manner. For some embodiments, the service robots 868 can be configured to perform floor sweeping, floor washing, etc.

[0116] In some embodiments, the service robot can follow a person from room to room and position itself so that the person can be monitored while in the room. The service robot can be parked in a position within the room where it may not be in the way but still has a relatively clear view of the room. The service robot 868 can be combined with the functionality of the electronic device 110 / 202. This arrangement can have the advantage of allowing one service robot combined with the functionality of the electronic device 110 / 202 to travel throughout multiple rooms or detect and track people in a large space.

[0117] The systems and methods of the present disclosure may be implemented using hardware, software, firmware, or a combination thereof, and may be implemented in one or more computer systems or other processing systems. Some embodiments of the present disclosure include a system comprising a processing system comprising one or more processors. In some embodiments, the system comprises a non-transitory computer-readable storage medium comprising instructions that, when executed on one or more processors, cause the system and / or one or more processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause the system and / or one or more processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes.

[0118] The terms and expressions that have been employed are used as terms of description and not of limitation, and the use of such terms and expressions is not intended to exclude any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Therefore, it should be understood that although the invention as claimed has been specifically disclosed by way of embodiments and optional features, modifications and variations of the concepts disclosed herein may be made by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined in the appended claims.

[0119] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. For example, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as not to obscure the embodiments.

[0120] The foregoing description of certain examples (including the illustrated examples) is presented for illustration and description purposes only and is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Modifications, adaptations, and uses thereof will be apparent to those skilled in the art without departing from the scope of the present disclosure. For example, any example described herein may be combined with any other example.

Claims

1. A method for extending the functionality of an application executed on an electronic device, the method comprising: accessing an extension container, the extension container comprising an application extension template populated with layout definition values ​​and binding values, the layout definition values ​​being configured to define a layout of a graphical user interface page of the application, the binding values ​​being configured to link device features to the graphical user interface page, wherein each of the device features corresponds to a function of a smart device that wirelessly communicates with the electronic device; and An extension including the extension container is provided to the electronic device.

2. The method of claim 1 , wherein the extension, when implemented by the application, is configured to cause the graphical user interface page of the application to be displayed on the display of the electronic device in the layout, wherein the layout includes a plurality of graphical user interface elements configured to cause the smart device to be controlled based on the device characteristics.

3. The method of claim 2, wherein the binding value is configured to link the device feature to the plurality of graphical user interface elements, each of the plurality of user interface elements being configured to cause a function of the smart device to be controlled based on a device feature included in the device feature.

4. The method of claim 1 , wherein before providing the extension to the electronic device, the method further comprises: A request for an application extension is received from the electronic device. 5 . The method of claim 1 , wherein the application extension template is populated with the layout definition values ​​and the binding values ​​based on input received by a user. The method of claim 1 , wherein the device characteristics are included in a device characteristics specification associated with the smart device. The method of claim 1 , wherein the extension container is JavaScript-based.

8. A system comprising: processing systems; as well as at least one computer-readable medium storing instructions that, when executed by the processing system, cause the system to perform operations comprising: accessing an extension container, the extension container including an application extension template populated with layout definition values ​​and binding values, the layout definition values ​​being configured to define a layout of a graphical user interface page of an application executed on an electronic device, the binding values ​​being configured to link device features to the graphical user interface page, wherein each of the device features corresponds to a function of a smart device in wireless communication with the electronic device; and An extension including the extension container is provided to the electronic device.

9. The system of claim 8 , wherein the extension, when implemented by the application, is configured to cause the graphical user interface page of the application to be displayed on the display of the electronic device in the layout, wherein the layout includes a plurality of graphical user interface elements configured to cause the smart device to be controlled based on the device characteristics.

10. The system of claim 9, wherein the binding value is configured to link the device feature to the plurality of graphical user interface elements, each of the plurality of user interface elements being configured to cause a function of the smart device to be controlled based on a device feature included in the device feature.

11. The system of claim 8, wherein before providing the extension to the electronic device, the operations further comprise: A request for an application extension is received from the electronic device.

12. The system of claim 8, wherein the application extension template is populated with the layout definition values ​​and the binding values ​​based on input received by a user.

13. The system of claim 8, wherein the device characteristics are included in a device characteristics specification associated with the smart device.

14. The system of claim 8, wherein the extension container is JavaScript-based.

15. One or more non-transitory computer-readable media storing instructions that, when executed by at least one processing system, cause the system to perform operations comprising: accessing an extension container, the extension container including an application extension template populated with layout definition values ​​and binding values, the layout definition values ​​being configured to define a layout of a graphical user interface page of an application executed on an electronic device, the binding values ​​being configured to link device features to the graphical user interface page, wherein each of the device features corresponds to a function of a smart device in wireless communication with the electronic device; and An extension including the extension container is provided to the electronic device.

16. One or more non-transitory computer-readable media as described in claim 15, wherein the extension, when implemented by the application, is configured to cause the graphical user interface page of the application to be displayed on the display of the electronic device in the layout, wherein the layout includes a plurality of graphical user interface elements configured to cause the smart device to be controlled based on the device characteristics.

17. One or more non-transitory computer-readable media as described in claim 16, wherein the binding value is configured to link the device feature to the plurality of graphical user interface elements, each of the plurality of user interface elements being configured to cause a function of the smart device to be controlled based on a device feature included in the device features.

18. The one or more non-transitory computer-readable media of claim 15, wherein prior to providing the extension to the electronic device, the operations further comprise: A request for an application extension is received from the electronic device.

19. The one or more non-transitory computer-readable media of claim 15, wherein the application extension template is populated with the layout definition values ​​and the binding values ​​based on input received by a user.

20. The one or more non-transitory computer-readable media of claim 15, wherein the device characteristics are included in a device feature specification associated with the smart device.

Images