3D Virtual Room-Based User Interface for Home Automation Systems

Abstract

Providing a user-navigable 3D virtual room-based user interface for home automation systems. [Solution] A user interface based on user-navigable 3D virtual rooms, each of which displays a substantially photorealistic representation of a corresponding physical room in a building, including the physical room's boundaries, furniture present in the physical room, and a substantially photorealistic representation of devices present in the physical room that are under the control of a home automation system. A user can move through the 3D virtual rooms using explicit navigation commands or implicit actions and move a virtual camera in 3D space to view the virtual room from various perspectives. By interacting with the substantially photorealistic representations of devices in the 3D virtual rooms, a user can direct changes to the state of the corresponding devices in the physical rooms.

Description

[Technical Field]

[0001] [Related Applications] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 898,941, entitled "Three Dimensional Virtual Room-Based User Interface for a Home Automation System," filed September 11, 2019, by Robert P. Madonna et al. [Background technology]

[0002] [Technical field] The present disclosure relates generally to device control, and more particularly to a user interface for controlling devices in a home automation system.

[0003] [Background information] As homes and other buildings become larger and filled with more devices, controlling the devices becomes increasingly difficult. Traditionally, many devices have been controlled by mechanical switches. While mechanical switches are reliable and cost-effective, they have many limitations, especially when many devices are located in the same room of a building. For example, a large room may contain numerous lighting devices, displays, electric blinds, heating, ventilation, and air conditioning (HVAC) units, etc. Controlling all of these devices may require numerous mechanical switches. As the number of mechanical switches in a room increases, usability decreases. Mechanical switches are often unlabeled, or when labeled, they are labeled with only cryptic descriptions (e.g., "Lamp 1," "Lamp 2," etc.). Users may be forced to remember which of the many mechanical switches available in a room controls which device. Users who cannot remember this correspondence typically must rely on trial and error, flicking switches until they accidentally achieve the desired result.

[0004] Various types of home automation systems have been developed that attempt to overcome the shortcomings of mechanical switches. Such systems typically include one or more controllers that manage the operation of various devices. Interaction with the controller (also called "dialogue" or "interaction") can occur through a user interface device, such as a dedicated touchscreen unit, that provides a user interface for controlling the various devices. The user interface may include an array of touch-sensitive buttons or sliders, each of which may control one device or a group of devices.

[0005] However, these button-centric, on-screen user interfaces suffer from many of the same drawbacks as mechanical switches. While the buttons and sliders appear on the screen rather than being physically present like mechanical switches, they function very similarly. Looking at a button or slider on the screen may not tell you what it does. While labels may be provided, given screen space constraints, such labels are often short and unclear. While labels are meaningful to the installer setting up the system, they may have little intrinsic meaning to the user. As with mechanical switches, users must touch each button or slide each slider on the screen and, by trial and error, figure out which button or slider achieves the desired result.

[0006] Recently, a device control solution has been developed that addresses many of the shortcomings of on-screen user interfaces that are dominated by mechanical switches and buttons. This solution provides a user interface that includes one or more fixed-viewpoint, two-dimensional (2D) virtual rooms displayed on a touchscreen. Each virtual room represents a 2D representation of a corresponding physical room in a building. By touching the representation of a device in the fixed-viewpoint, 2D virtual room displayed on the screen, the user can direct a state change of the device to be implemented by the home automation system in the physical room. When the state of the device in the physical room is changed, the appearance of the fixed-viewpoint, 2D virtual room is updated to show the changed state.

[0007] While this type of solution represents a notable advancement, addressing many of the shortcomings of on-screen user interfaces centered around mechanical switches and buttons, it can still be improved. One problem with interfaces based on fixed-perspective 2D virtual rooms is that they require users to be familiar with the building's floor plan and the specific names of each room. The interface may present the user with various fixed-perspective 2D virtual rooms, from which the user must select. This selection must be repeated as the user moves around the building and wishes to control devices in various physical rooms. If the user is not familiar with the building's floor plan and the names of each room, the user must use trial and error to select the correct virtual room to use to perform the desired changes.

[0008] Another problem with interfaces based on fixed-viewpoint 2D virtual rooms is that some types of state changes may not be easily represented or displayed in a "natural" way. According to prior art, a fixed-viewpoint 2D virtual room is generated by taking multiple 2D images (e.g., photographs) of a physical room in various states from the same preselected viewpoint. This includes an all-off 2D image and device-specific images with one device activated and all other devices deactivated. Various images from the multiple images are combined with filters to generate a fixed-viewpoint 2D virtual room containing various devices in various states. While such filtering works well for some types of state changes (e.g., some lighting state changes), other types of state changes (e.g., color lighting state, media content state, motorized blind position, gas fireplace flame setting, etc.) are not realistic, and their appearance cannot be well reproduced by combining a small number of 2D images. For example, some color lighting devices may be capable of generating multiple colors (e.g., 32-bit color). The appearance of all those colors in a physical room may not be easily simulated by simply combining a few 2D images with filters, as there are simply too many possibilities. Similarly, the state of a television's media content (channels, sources, files, etc.) may not be easily simulated by combining a few 2D images with filters, as the necessary information is simply not there. Similarly, filtering does not always perfectly reproduce the subtle differences in the effects that occur when multiple devices interact to affect the appearance of a physical room. For example, the interplay of natural sunlight caused by various positions of motorized blinds, artificial light from various lighting fixtures, ambient light from a gas fireplace flame, etc. may not be well reproduced by simply combining a few 2D images with filters.

[0009] Another problem with interfaces based on fixed-viewpoint 2D virtual rooms is that it may be difficult to preselect a desirable viewpoint depending on the shape of the room and the arrangement of devices. Typically, installers preselect a small number of viewpoints to capture 2D images that display a large number of devices at an appropriate size while minimizing overlap of objects. For a particular room with a complex shape or unusual arrangement of devices, it may be difficult or impossible to preselect a small number of viewpoints that meet these objectives. User interfaces based on fixed-viewpoint 2D virtual rooms may still be generated, and their usability may be reduced.

[0010] Therefore, there is a need for an improved virtual room-based user interface for controlling home automation systems that can address some or all of these issues. Summary of the Invention [Means for solving the problem]

[0011] [overview] In one embodiment, a user interface based on user-navigable three-dimensional (3D) virtual rooms for a home automation system is provided. Each user-navigable 3D virtual room depicts a substantially photorealistic representation of a corresponding physical room in a building, including the physical room's boundaries (walls, ceiling, floor, etc.), furniture present in the physical room (e.g., sofas, chairs, beds, wall hangings, etc.), and devices present in the physical room that are under the control of the home automation system (e.g., lighting devices, displays, motorized blinds, HVAC devices, and / or other types of devices). A user navigates within the user-navigable 3D virtual rooms using explicit navigation commands (e.g., movement commands or node selection) or implicit actions (e.g., moving devices detected using positioning beacons and / or orientation sensors) and can move a virtual camera in 3D space to view the virtual room from various perspectives. By interacting with (e.g., touching, clicking, etc.) the substantially photorealistic representations of devices in the user-navigable 3D virtual rooms, the user can direct changes to the state of the corresponding devices in the physical room. As the state of devices in the physical room changes, the 3D graphics engine dynamically updates the appearance of the user-navigable 3D virtual room to reflect the change, so that what the user sees in the virtual room mimics the user's experience in the corresponding physical room. Because users can move through the 3D space and observe the relationships between rooms, they can operate interfaces based on 3D virtual rooms more easily than traditional interfaces. Additionally, thanks to the 3D graphics engine, 3D virtual rooms may be able to display a variety of states and appearance effects that were previously difficult to represent. Furthermore, 3D virtual rooms may be better able to adapt to various room shapes and device arrangements.

[0012] A user-navigable 3D virtual room-based user interface can be generated using a 3D mesh model and 2D images of a physical room. In one example process, an installer places 3D cameras at multiple locations in the physical room and captures multiple sets of overlapping 2D images (e.g., 2D panoramic images) and 3D spatial models (e.g., 3D meshes). The overlapping 2D images (e.g., 2D panoramic images) and 3D spatial models (e.g., 3D meshes) are imported into a stitching application, which links (i.e., stitches) the image data to corresponding locations in the 3D spatial model. The stitched 2D images (e.g., 2D panoramic images) and 3D spatial models (e.g., 3D meshes) are imported into a 3D modeling application. The installer uses the 3D modeling application to correct visual artifacts and tag depictions of various devices with hit regions. The hit regions are mapped to various characteristics of those devices and recognized control commands for changing the device states. The installer then uses the 3D modeling application to assign appearance changes to the device representations that match those device characteristics and control commands. The assigned appearance changes define how the appearance must update to match changes made in the physical room when a control command is issued. The stitched, artifact-corrected, tagged, and appearance-assigned 2D images and 3D space model (called a virtual room) are then exported to a control app. The user can then use the control app to control the home automation system and its devices.

[0013] When a virtual camera in the virtual room is in a position corresponding to the position where one of the 2D images (e.g., a 2D panoramic image) was taken, the controlling app's 3D graphics engine can display the data from that 2D image (e.g., a 2D panoramic image), adding appearance modifications as needed. When the virtual camera is moved through a position that does not correspond to any of the 2D images (e.g., a 2D panoramic image), the controlling app's 3D graphics engine blends the various available 2D images (e.g., modifying the alpha channel and rendering layers of those 2D images) using a 3D space model (e.g., a 3D mesh) and displays the blended data, adding appearance modifications as needed.

[0014] It should be understood that various additional features and alternative embodiments may be implemented. This summary is intended merely to provide a brief introduction to the reader and does not represent or imply that the examples referred to herein are exhaustive or essential aspects of the invention. [Brief explanation of the drawings]

[0015] In the following description, reference is made to the accompanying drawings.

[0016] FIG. 1 is a block diagram illustrating an example architecture of a home automation system operable to control devices in rooms of a building (e.g., a residential or commercial building).

[0017] FIG. 2A is a screenshot of an exemplary user-navigable 3D virtual room that can be displayed by a controlling app.

[0018] 2B-2C are screenshots illustrating the exemplary user-navigable 3D virtual room of FIG. 2A, showing free movement of the virtual camera in 3D space to view the virtual room from different perspectives.

[0019] 2D-2E are screenshots showing the exemplary user-navigable 3D virtual room of FIG. 2A, illustrating the movement of the virtual camera in 3D space using navigation nodes to view the virtual room from different perspectives.

[0020] 2F-2G are screenshots illustrating the exemplary user-navigable 3D virtual room of FIG. 2A, showing changes in lighting device brightness in response to user interaction with the depictions of the lighting devices.

[0021] 2H-2I are screenshots illustrating the exemplary user-navigable 3D virtual room of FIG. 2A, showing changes in the state of the display devices in response to user interaction with the representation of the display devices.

[0022] 2J-2L are screenshots illustrating the exemplary user-navigable 3D virtual room of FIG. 2A, showing changes in the state of lighting devices in response to selections on the menu.

[0023] FIG. 2M is a screenshot of an exemplary user-navigable 3D virtual room without visual artifacts and with relatively high resolution, which may be relatively close to a commercial implementation.

[0024] FIG. 3 is a flow diagram illustrating an exemplary series of steps for operating a user-navigable 3D virtual room-based user interface for controlling devices in a home automation system.

[0025] FIG. 4 is a flow diagram illustrating an exemplary series of steps for generating a user-navigable 3D virtual room-based user interface for controlling devices in a home automation system. DETAILED DESCRIPTION OF THE INVENTION

[0026] [Definition] As used herein, the term "home automation system" should be interpreted broadly to encompass various types of home control systems, "smart home" systems, and / or device control systems that can control various devices (e.g., lighting devices, displays, motorized blinds, HVAC devices, and / or other types of devices) within a building, such as a residence or commercial building.

[0027] As used herein, the term "physical room" refers to the interior of a physical building or an external space associated with a physical building where one or more devices can provide services.

[0028] As used herein, the term "virtual room" refers to a digital twin (a virtual digital replica) of a physical room, represented by a representation of the interior of a physical building or an exterior space associated with a physical building.

[0029] As used herein, the term "mobile device" refers to an electronic device suitable for being carried by a person that runs a general-purpose operating system. Devices such as smartphones should be considered mobile devices. Desktop computers, servers, or other primarily stationary computing devices should generally not be considered mobile devices.

[0030] Home automation system example FIG. 1 is a block diagram illustrating an example architecture of a home automation system 100 operable to control devices in rooms of a building (e.g., a residential or commercial building). At the core of the system is a host controller 110 coupled to a home local area network (LAN) 150 (e.g., a wired network such as Ethernet and / or a wireless network such as Wi-Fi). The host controller may include hardware components such as a processor, memory, and storage devices, which collectively store and execute host software 111. The host software 111 is configured to monitor and control the operation of devices 112-124, utilize beacons 125, provide UI interpretation, system management and monitoring, and synchronization with cloud services 180, remote controllers 140, mobile devices 160, and other electronic devices 165 used to control the system, provide activity logging services, provide activity prediction services, and / or other types of functionality. The host controller 110 may maintain a home database 130 within its own storage device. The home database 130 stores configuration information, including information about the devices 112-124 controlled by the home automation system and the services they can provide, as well as information about the controllers 140, mobile devices 160, and other electronic devices 165 used to control the system.

[0031] The devices 112-124 in a home automation system may include lighting devices 112, such as light fixtures and dimming modules; interface devices 113, such as keypads, switches, and touchscreens; security devices 114, such as home monitors / cameras, motion sensors, home health sensors, and associated controllers; audio and video devices 116 and 118 (collectively, A / V devices), such as display devices (e.g., televisions, monitors, etc.), A / V device controllers, media servers, audio amplifiers, and cable boxes; electronic door locks 120; motorized blinds 121 and other types of motorized devices that create movement within a room (e.g., television lifts, automatic doors, etc.); HVAC devices 122, such as thermostatically controlled heating and cooling systems, gas fireplaces, and whole-room fans; interconnection devices 124, such as IR blasters, matrix switchers, and signal extenders; and other types of home automation system devices. Each of the devices 112-124 may be associated with (i.e., configured for use in conjunction with) a physical room in a building and may therefore be said to be "in" the room. It should be understood that when used in this context, the word "in" should be interpreted to encompass equipment that is physically present in the room or that is present elsewhere (e.g., in a remote equipment rack) and that provides various services to the room from such a remote location.

[0032] The communication capabilities of the devices 112-124 in the home automation system may vary depending on the embodiment. For example, at least some devices may include a LAN interface (e.g., an Ethernet or Wi-Fi adapter) and / or a wireless personal area network (WPAN) interface (e.g., a Bluetooth or Bluetooth Low Energy (BLE) adapter) that allows them to communicate with devices such as the host controller 110. Similarly, some devices may only have ports or transceivers for wired communication or point-to-point wireless communication (e.g., RS-232, RS-485, general-purpose input / output (GPIO), infrared (IR), etc.) and may use those technologies to communicate with the host controller 110 or other devices. Some devices (e.g., interconnection devices such as an IR blaster) may be able to bridge different types of communication, for example, they may include and bridge both a WPAN interface (e.g., a Bluetooth or BLE adapter) and a point-to-point wireless transceiver (e.g., an IR transceiver). Additionally, some devices may include a LAN interface (e.g., an Ethernet or Wi-Fi interface) but may not be configured to communicate directly with the host controller 110 or other devices in the home automation system over the home LAN 150. Instead, these devices may have access to the Internet 170 and cloud services 180 and / or third-party infrastructure 190, which may then communicate with the host controller 110. It should be understood that some of the HVAC devices 122 shown in FIG. 1 may communicate in this manner. Additionally or alternatively, other types of devices 112-124 may communicate in this manner.

[0033] A home automation system may include multiple positioning beacons that transmit and receive WLAN, WPAN, or other wireless signals (e.g., Bluetooth, BLE, Wi-Fi, Ultra-Wideband (UWB), Radio Frequency Identification (RFID), etc.) to determine the location of a remote control 140, mobile device 160, or other electronic device 165 within a building. Location determination may be performed by using received signal strength (RSS) to select the nearest beacon location, performing trilateration based on multiple beacon locations and their associated signal strengths, and / or using other techniques. These beacons may be standalone devices, such as standalone beacon 125, or may be integrated into one or more of devices 112-124 that provide other functionality. In one embodiment, the beacons are integrated into lighting devices 112 and keypads, with lighting devices 112 providing both lighting and positioning functionality and keypads providing both user interface and positioning functionality.

[0034] A user can control the devices 112-124 of the home automation system using the remote controller 140. The remote controller 140 may include a touch-sensitive display screen, physical buttons, a communication interface (e.g., IR, WPAN, etc.), a processor, memory, and a storage device for storing and executing a control app configured to interface with the host controller 110 and the cloud service 180. The remote controller may further include an orientation sensor, which, in conjunction with a positioning beacon, enables determining the remote controller's location and orientation within a building's vicinity. The control app on the remote controller 140 can display, among other functions, a user-navigable 3D virtual room-based user interface for controlling the devices 112-124 of the home automation system 100.

[0035] A user can also control the devices 112-124 of the home automation system using the mobile device 160. The mobile device 160 may include a touch-sensitive display screen, a communications interface (e.g., Wi-Fi, WPAN, etc.), a processor, memory, and storage devices that store and execute a control app 162 configured to connect with the host controller 110 and / or cloud service 180. The mobile device 160 may further include an orientation sensor that, in conjunction with a positioning beacon, enables determining the location and orientation of the mobile device 160 within a building vicinity. The control app on the mobile device 160 can display, among other functions, a user-navigable 3D virtual room-based user interface for controlling the devices 112-124 of the home automation system 100.

[0036] Additionally, a user may control the devices 112-124 of the home automation system using another electronic device 165, such as a tablet computer, a head-mounted display (HMD) like a GOOGLE GLASS® HUD, a dedicated touchscreen unit, a television and remote control, a computer and mouse, or other types of technology. The electronic device 165 may include a display screen (e.g., touch-sensitive, non-touch-sensitive, HMD, etc.), input devices, a communications interface (e.g., Wi-Fi®, WPAN, etc.), a processor, memory, and storage devices that store and execute software configured to interface with the host controller 110 and / or cloud service 180.

[0037] The electronic device 165 may further include an orientation sensor, which, in conjunction with the positioning sensor, allows the electronic device 165 to determine its position and orientation in the vicinity of the building. For example, in an embodiment in which the electronic device 165 is an HMD and the beacon is a BLE beacon, the position determination may be performed by BLE trilateration and the orientation determination may be performed by head movement. The control app may display a 3D virtual room-based user interface on the HMD for controlling the devices 112-124 of the home automation system, and the user may make selections using the input devices of the HMD.

[0038] It should also be understood that electronic device 165 may include multiple individual devices operating together. For example, in an embodiment in which electronic device 165 is a television and a remote control, the control app may display a 3D virtual room-based user interface on the television screen, and selections may be received on the remote controller (e.g., by moving a cursor on the screen to select various items).

[0039] A remote control 140, a mobile device 160, or an electronic device 165 can communicate with the host controller 110 to perform device control. Some devices (e.g., mobile device 160, electronic device 165, etc.) can communicate with a cloud service 180 and its host application program interface (API) 182 and mobile API 184. The cloud service 180 can provide, among other functions, remote access to the home automation controller, persistent backup of the home database 130 (storing data in a configuration database 186), connectivity to third-party infrastructure (via a third-party adapter 188), user profile and usage tracking (storing data in a user database 189), as well as mechanisms for over-the-air updates, host crash reporting, and license management.

[0040] User interface operation based on a user-navigable 3D virtual room A control app on the remote controller 140, mobile device 160, or other electronic device 165 can display a 3D virtual room-based user interface for controlling the devices 112-124 of the home automation system 100. The interface may include multiple user-navigable 3D virtual rooms, each showing a substantially photorealistic depiction of a corresponding physical room in a building. Each user-navigable 3D virtual room may include a substantially photorealistic depiction of the physical room's boundaries (e.g., walls, ceiling, floor, etc.), furniture present in the physical room (e.g., sofas, chairs, beds, wall hangings, etc.), and devices 112-124 (e.g., lighting devices, displays, motorized blinds, and / or other types of devices) present in the physical room. Each of the devices 112-124 may have several possible states. Devices 112-124 may have a set of binary possible states (e.g., an inactive "off" state and an active "on" state), or may have a set of more numerous possible states (e.g., multiple illumination levels, colors (e.g., 32-bit color), color temperatures (e.g., 3000K, 5000K, etc.), media content (e.g., television channels, sources, individual media files, etc.), locations, temperatures, etc.).

[0041] 2A-2M are screenshots showing an exemplary user-navigable 3D virtual room that can be displayed by a control app, corresponding to an open-plan kitchen and living room in a physical building. It should be understood that while the screenshots in FIGS. 2A-2L are low resolution and contain visual artifacts, commercial embodiments are preferably high resolution and have those artifacts corrected, making them appear substantially photorealistic. FIG. 2M shows an example that more closely resembles the preferred appearance of a commercial embodiment.

[0042] 2A is a screenshot showing an exemplary user-navigable 3D virtual room that may be displayed by a control app. The user-navigable 3D virtual room includes substantially photorealistic depictions of the physical room's boundaries, such as floor 210 and walls 212; the physical room's furniture, such as sofa 220, table and chairs 222, and stove 224; and the devices in the physical room, such as chandelier 230, recessed can light fixtures 232-238, and television 239, that are under the control of home automation system 100.

[0043] A user can navigate the virtual room using explicit navigation commands or implicit actions to view the virtual room from different perspectives by moving the virtual camera in 3D space. Explicit navigation commands can take a variety of forms. In one embodiment, explicit navigation commands may take the form of movement commands (e.g., touch gestures such as scrolling or swiping on a touch-sensitive display screen, cursor movement, etc.). Navigation may include free movement where the virtual camera is freely moved horizontally or vertically in 3D space and freely rotated in various orientations within 3D space.

[0044] 2B-2C are screenshots illustrating the exemplary user-navigable 3D virtual room of FIG. 2A, illustrating free movement of a virtual camera in 3D space to view the virtual room from various perspectives. A movement icon 240 may be displayed and may be moved upon receiving a movement command (e.g., a scroll gesture, cursor movement, etc.). In this example, the virtual camera is moved horizontally forward between FIGS. 2B and 2C.

[0045] In another embodiment, the explicit navigation command may take the form of a node selection. A number of predetermined nodes may be positioned at predetermined locations within the virtual room and represented as icons. When a node is selected in response to a user selection of the node (e.g., touching the node on a touch-sensitive display screen, selecting with a cursor, etc.), the virtual camera is moved (e.g., a "snap move") to that location. Such movement may be displayed in a "smooth" manner, moving the virtual camera through space and continuously updating the virtual room to indicate the movement. Each node may be associated with a predetermined starting orientation. Once the node is reached, the virtual camera may be freely rotated to various orientations within 3D space in response to a navigation command.

[0046] 2D-2E are screenshots of the exemplary user-navigable 3D virtual room of FIG. 2A, illustrating the use of navigation nodes to move the virtual camera in 3D space to view the virtual room from different perspectives. Multiple selection nodes 245, 247 are displayed and can be selected (e.g., touched, clicked, etc.). In this example, selection node 245 is selected in FIG. 2D, which moves the virtual camera to the perspective shown in FIG. 2C.

[0047] Implicit actions can also take various forms. In one embodiment, implicit actions may be based on the position and orientation of the remote controller 140, mobile device 160, or other electronic device 165 determined using positioning beacons (and, for example, their Bluetooth, BLE, Wi-Fi, UWB, RFID, and other signals) and orientation sensors. A user can freely move the virtual camera by walking around a physical room while holding the remote controller 140, mobile device 160, or other electronic device 165. A user can freely rotate the virtual room by rotating the remote controller 140, mobile device 160, or other electronic device 165. If the electronic device is an HMD, the position and orientation of the user's head may be directly translated into a position and orientation in the virtual room.

[0048] By interacting (e.g., touching, clicking, etc.) with the substantially photorealistic representations of devices in the user-navigable 3D virtual room, the user can direct a change in the state of the corresponding device in the physical room. The state change may cycle through various available states of the device (e.g., dual states, multiple possible states, etc.). As the state of the device changes, the controlling app's 3D graphics engine (e.g., the Unity® or Unreal® graphics engine) reflects those changes by dynamically updating the appearance of the user-navigable 3D virtual room so that what the user sees in the virtual room mimics the user's experience in the corresponding physical room. The dynamic update may include changing the appearance (e.g., light level, color, color temperature, media content, position, or other visual characteristics) of the photorealistic representation of each device whose state has changed. The dynamic update may also include changing the appearance (e.g., shadows and reflections) of the substantially photorealistic representations of boundaries, furniture, and other devices whose state has not yet changed to depict the effect of the state change on those items. In this way, the 3D graphics engine mimics in the virtual room the experience that the user would observe in the physical room as the states are changed.

[0049] 2F-2G are screenshots of the exemplary user-navigable 3D virtual room of FIG. 2A , illustrating changes in the brightness of lighting devices in response to user interaction with a substantially photorealistic representation of the lighting devices. In FIG. 2F , the user interacts (e.g., touches, clicks, etc.) with a representation of a lighting device, specifically a recessed can lighting fixture 232. In response to such interaction, the control app causes the home automation system 100 to activate the recessed can lighting fixture 232 in the physical room. The control app's graphics engine also dynamically updates the appearance of the representation of the recessed can lighting fixture 232 in the virtual room to make the lighting fixture 232 appear to be lit (e.g., by placing a virtual light source at its location), and based on the changes, dynamically updates the appearance of boundary representations (e.g., shadows and reflections 250 on the wall), furniture representations (e.g., shadows and reflections 254 on the sofa), and other device representations (e.g., shadows and reflections 252 on the chandelier), as shown in FIG. 2G .

[0050] 2H-2I are screenshots of the exemplary user-navigable 3D virtual room of FIG. 2A, illustrating changes in the state of the display device in response to user interaction with the substantially photorealistic representation of the display device. Here, the state is a media content state, i.e., a channel (e.g., a television channel). However, it should be understood that the media content state may take other forms, such as a source (e.g., a signal from a DVD, cable box, etc.), a media file (e.g., a video file, a TV program file, etc.). In FIG. 2H, the user interacts (e.g., touches, clicks, etc.) with the display device, specifically the representation of a television 239. In response to such interaction, the control app causes the home automation system 100 to change the channel of the television in the physical room (here, from channel 6 to channel 1). Such changes may include cycling through several intermediate channels (e.g., in response to repeated touches, clicks, etc.). The control app's graphics engine further dynamically updates the appearance of the representation of the television 239 so that the actual media content for that channel is displayed within the television 239 in a user-navigable 3D virtual room, as shown in Figure 2I.

[0051] It should be noted that in addition to changing the brightness of lighting devices or changing the state of display devices, a wide variety of other types of state changes may be made to other types of devices 112-124. For example, in response to a user interaction with a substantially photorealistic representation of motorized blinds, the control app may cause the home automation system 100 to activate the motors of the blinds and change the position of the blinds (e.g., opening or closing the blinds). The graphics engine of the control app may also dynamically update the appearance of the representation of the motorized blinds in the virtual room to reflect the new position of the blinds, and may also dynamically update the appearance of the representations of boundaries, furniture, and other devices based on the change (e.g., changing shadows and reflections in response to more or less natural light entering the room through the windows).

[0052] Similarly, in response to user interaction with the substantially photorealistic representation of the gas fireplace, the control app can cause the home automation system 100 to send a signal to the electronic ignition gas supply system to adjust the flame. The control app's graphics engine also dynamically updates the appearance of the representation of the gas fireplace in the virtual room to reflect the changed flame state, and also dynamically changes the appearance of the representation of boundaries, furniture, and other devices based on the changed flame state (e.g., changing shadows and reflections depending on the amount of flame in the fireplace).

[0053] With a large number of devices, it may be difficult to find a substantially photorealistic depiction of a desired device in the virtual room. Similarly, with a large number of device states, cycling through the states may be inefficient or impractical. In such cases, the user-navigable 3D virtual room may be configured to display a menu in response to a user interacting with an interface element. The menu may list various controllable devices and device states. The user may select (e.g., by touching, clicking, etc.) the desired device or state. The controlling app may cause the home automation system to make the desired state change, and the 3D graphics engine dynamically updates the appearance of the user-navigable 3D virtual room to reflect the change, such that what the user sees in the virtual room mimics the user's experience in the corresponding physical room.

[0054] 2J-2L are screenshots of the example user-navigable 3D virtual room of FIG. 2A, illustrating changes in the state of lighting devices in response to menu selections. In FIG. 2J, the user interacts (e.g., touches, clicks, etc.) with menu interface element 260. In response to such interaction, the control app displays menu 270, e.g., superimposed on the virtual room, as shown in FIG. 2K. In this example, menu 270 includes various lighting devices in the physical room and their possible states, e.g., intensity level, color, and color temperature. In this example, the user selects a lighting device (in this example, a recessed can lighting fixture 238), its intensity level, and its color. The control app then causes the home automation system to illuminate recessed can lighting fixture 238 to the desired level in the desired color. A large number of different colors can be supported (e.g., 32-bit color). The control app's graphics engine also dynamically updates the appearance of the representation of the recessed can lighting fixture 238 so that it appears to be lit at a desired color and level (e.g., by placing a virtual light source in its location), and also dynamically updates the appearance (e.g., shadows and reflections) of the representation of boundaries, furniture, and other devices in the room, as shown in Figure 2L. Supporting a large number of colors (e.g., 32-bit color) can greatly simplify control, as users can see what a room looks like by observing the virtual room with various colors.

[0055] 2M is a screenshot of an exemplary user-navigable 3D virtual room with a relatively high resolution and no visual artifacts, which may be relatively close to a commercial implementation. As can be seen, the depictions of controlled devices (such as television 239), boundaries (such as walls), furniture (such as sofas), and other devices appear substantially photorealistic. It should be understood that the virtual rooms shown in FIGS. 2A-2L may also appear this way.

[0056] 3 is a flow diagram illustrating an exemplary series of steps for operating a user interface based on a user-navigable 3D virtual room to control devices 112-124 of home automation system 100. The various steps in FIG. 3 summarize the operations described in detail above. In step 310, a control app on remote controller 140, mobile device 160, or other electronic device 165 uses a graphics engine to render a user-navigable 3D virtual room from a first perspective defined by a virtual camera. The user-navigable 3D virtual room may include a substantially photorealistic depiction of the physical room's boundaries (e.g., walls, ceiling, floor, etc.), furniture in the physical room (e.g., sofas, chairs, beds, wall hangings, etc.), and devices present in the physical room that are under the control of home automation system 100 (e.g., lighting devices, displays, motorized blinds, HVAC equipment, and / or other types of devices). In step 320, the controlling app displays the user-navigable 3D virtual room rendered by the controlling app on a display screen (e.g., a touch-sensitive display screen) of the remote controller 140, mobile device 160, or other electronic device 165.

[0057] At step 330, the controlling app determines whether it has received an explicit navigation command (e.g., a movement command or node selection) or an implicit action (e.g., a change in the position or orientation of the remote controller 140, mobile device 160, or other electronic device 165). If so, at step 340, the controlling app responsively changes the viewpoint by changing the position and / or orientation of the virtual camera, and execution loops back to step 310, where the graphics engine re-renders the virtual room from this new viewpoint. If not, execution proceeds to step 350, where the controlling app determines whether the user has interacted (e.g., touched, clicked, etc.) with the substantially photorealistic representation of a device in the user-navigable 3D virtual room. If so, at step 360, the controlling app causes the home automation system 100 to change the state of that device in the physical room. Additionally, in step 370, the controlling app dynamically updates the appearance (e.g., light level, color, color temperature, media content, position, or other visual characteristics) of the substantially photorealistic representation of the device in the virtual room, as well as the appearance (e.g., shadows and reflections) of the substantially photorealistic representation of the boundaries, furniture, and other devices. Execution then loops back to step 310, where the controlling app's graphics engine re-renders the virtual room with these new appearances.

[0058] If not, execution proceeds to step 380, where the controlling app determines whether the user has interacted (e.g., touched, clicked, etc.) with a menu interface element. If so, in step 390, a menu is displayed superimposed on the user-navigable 3D virtual room. In step 395, the controlling app determines whether a device and state have been selected on the menu. If so, execution loops back to step 360, where the controlling app causes the home automation system 100 to change the state of that device in the physical room. Next, in step 370, the controlling app dynamically updates the appearance of the substantially photorealistic representation of the selected device in the virtual room, as well as the appearance of the substantially photorealistic representations of boundaries, furniture, and other devices, based on the selected state. Thereafter, execution loops back to step 310, where the controlling app's graphics engine re-renders the virtual room based on those new appearances. If not, the controlling app waits for further user input, and execution loops back to step 330.

[0059] Generating a user interface based on a user-navigable 3D virtual room A 3D virtual room-based user interface is typically generated by a combination of data collection and configuration operations performed by a configuration application running on a local computing device or in the cloud, and rendering operations performed by a graphics engine of a control app running on a remote controller 140, mobile device 160, or other electronic device 165. Figure 4 is a flow diagram illustrating a series of steps for generating a user-navigable 3D virtual room-based user interface for controlling devices 112-124 of a home automation system. Steps 410-480 represent data collection and configuration operations, and steps 485-495 represent rendering operations.

[0060] In step 410, the installer places 3D cameras at multiple locations in the physical room and captures overlapping sets of 2D images (e.g., 2D panoramic images) and a 3D spatial model (e.g., a 3D mesh). The 3D cameras may use any of a variety of imaging scanning techniques (e.g., single-point laser scanning, line-profile laser scanning, structured light (non-laser) detection, stereo vision, etc.) to generate the 3D spatial model. Preferably, all devices are deactivated or "off" during capture to simplify the creation of the visual effect.

[0061] In step 420, the 2D image (e.g., a 2D panoramic image) and the 3D space model (e.g., a 3D mesh) are imported from the 3D camera into a stitching application. The stitching application may run in the cloud or on a local computing device. In one embodiment, the stitching application may be a cloud-based software package such as Matterport®. In step 430, the installer uses the stitching application to stitch the 2D image (e.g., a 2D panoramic image) and the 3D space model (e.g., a 3D mesh) together and link (stitch) the image data to corresponding locations in the 3D space model.

[0062] At step 440, the stitched 2D images and 3D space model are imported into a 3D modeling application. The 3D modeling application may run in the cloud or on a local computing device. In one embodiment, the 3D modeling application may be a 3D development platform such as Unity® or Unreal®. At step 450, the installer uses the 3D modeling application to correct visual artifacts. Visual artifacts can be caused by various factors during the photography and stitching process. For example, reflective surfaces such as display screens and window glass typically do not photograph well and can create visual artifacts that must be corrected. At step 460, the installer uses the 3D modeling application to tag the depictions of various devices with hit regions (i.e., 3D hit boxes) and map these hit regions to various device characteristics and home automation system 100 control commands for changing the device states. For example, a lighting device may be tagged with a hit region encompassing its outline and mapped to various light emitting characteristics of the lighting load controlled by specific lighting control commands (e.g., to change illuminance level, color, color temperature, etc.). Similarly, a display device may be tagged with a hit region encompassing its screen and mapped to various display characteristics of the display device controlled by specific control commands affecting the state of media content (e.g., channel, source, file, etc.). Similarly, motorized shades may be tagged with a hit region encompassing its outline and mapped to various movement characteristics of the motorized shades controlled by specific position control commands.

[0063] In step 470, the installer uses a 3D modeling application to assign appearance changes to the device representations that match the device's characteristics and control commands. The assigned appearance changes define how the controlling app's graphics engine must update the device representation to match changes made in the physical room when a control command is issued, and how the appearance changes must affect the appearance of room boundaries, furniture, and other devices. The assigned appearance changes may include types and boundaries based on the device's characteristics. In step 480, the stitched, artifact-corrected, tagged, and appearance-assigned 2D images and 3D space model (called a virtual room) are exported to the controlling app for inclusion in a user-navigable 3D virtual room-based user interface.

[0064] The virtual room is rendered by the graphics engine of the controlling app. In step 485, the controlling app determines whether the virtual camera showing the user's desired viewpoint is in a position corresponding to the position where one of the 2D images (e.g., a 2D panoramic image) was taken. If so, in step 485, the graphics engine of the controlling app renders the virtual room using the data of the 2D image (e.g., a 2D panoramic image) taken from that position. If not, in step 495, the graphics engine of the controlling app blends (e.g., modifies the alpha channel and renders layers of those 2D images) the various available 2D images (e.g., 2D panoramic images) according to a 3D space model (e.g., a 3D mesh) and renders the virtual room using the blended data.

[0065] In summary, a user-navigable 3D virtual room-based user interface is provided for controlling various devices in a home automation system. While the above description uses specific examples, it should be apparent that numerous modifications and / or additions can be made thereto. For example, while it has been described above that remote controller 140, mobile device 160, or other electronic device 165 may each have a touch-sensitive display screen and that user input in the user-navigable 3D virtual room-based user interface may be made by gesture or touch, it should be understood that the interface may be adapted for a non-touch-sensitive screen and that user input may be received via a pointing device or cursor (e.g., clicking an item to make a selection) or via other types of input devices.

[0066] Similarly, while the user interface based on a user-navigable 3D virtual room is described above as being usable for controlling a configured home automation system 100 in a building, the user interface based on a user-navigable 3D virtual room may also be adapted for use in previewing or pre-configuring a home automation system for sales or installation purposes. For example, a user-navigable 3D virtual room may be used to show a user effects beyond those that can be generated in a building prior to purchase. Alternatively, the user may be shown potential effects that can be generated during the pre-configuration process when the system is initially installed or set up. In such cases, it may not be possible to actually generate those effects in the physical room at the time of the demonstration.

[0067] Additionally, while the above discussion describes a user-navigable 3D virtual room mimicking the appearance of a physical room and describes various types of visual appearances, it should be understood that appearances may also include non-visual aspects of the physical room experience, such as sound. In such cases, the controlling app may play audio being played in the physical room and / or sound effects that mimic the ambient sounds in the physical room from speakers on the remote controller 140, mobile device 160, and other electronic devices 165. For example, when a user turns on the television 239 and changes to a channel, the actual audio for that channel may be played from speakers on the remote controller 140, mobile device 160, and other electronic devices 165, along with the visual representation of the user-navigable 3D virtual room. Similarly, when a user changes the position of motorized blinds, sound effects that mimic the blinds raising or lowering may be played from speakers on the remote controller 140, mobile device 160, and other electronic devices 165, along with the visual representation of the user-navigable 3D virtual room.

[0068] Additionally, while it has been described above that changes in the state of a device in a physical room may occur in response to user interaction with a substantially photorealistic representation of that device in a user-navigable 3D virtual room (e.g., the user touching or clicking on the representation of the device), it should be understood that some state changes may be configured to occur at a predetermined time or in response to a predetermined condition being met. In one embodiment, a user may interact with the system to provide circadian lighting by configuring lighting device illuminance levels, colors, and / or color temperatures, or other conditions to dynamically change throughout the day. These state changes may be based, at least in part, on outdoor sensors that capture current lighting data for the outdoor environment. The appearance of the representations of lighting devices, boundaries, and furniture in the user-navigable 3D virtual room is updated to reflect the changed states implemented by the circadian lighting.

[0069] Finally, it should be understood that the above steps may be implemented in hardware, software (embodied as a non-transitory electronic device readable medium containing software), firmware, or a combination thereof. The non-transitory electronic device readable medium may take the form of a memory such as a random access memory (RAM), a disk such as a hard drive or flash drive, or other tangible storage device. In general, it should be understood that the above description is intended to be taken by way of example only.

Claims

1. 1. A method for controlling a home automation system using a user interface including a three-dimensional (3D) virtual room, comprising: a control application (control app) running on an electronic device renders a 3D virtual room from a viewpoint defined by a virtual camera in 3D space and displays the 3D virtual room on a display screen of the electronic device, the control app rendering the 3D virtual room based on data of a plurality of two-dimensional (2D) images of the physical room taken from different positions within the physical room, the 3D virtual room including representations of one or more devices present in the physical room under the control of the home automation system, representations of one or more boundaries of the physical room, and representations of one or more pieces of furniture present in the physical room; receiving navigation commands from a user via the display screen or signals from a positioning beacon or orientation sensor indicating a change in the position or orientation of the electronic device; and in response to the navigation command or the signal indicating a change in position or orientation of the electronic device, the controlling app changes at least a position or orientation of the virtual camera in the 3D space and re-renders and displays the 3D virtual room from a new viewpoint, the new viewpoint not corresponding to a position in the physical room at which any of the 2D images were taken, and the 3D virtual room is re-rendered by blending 2D images taken from different positions to display the 3D virtual room from the new viewpoint. receiving a touch or click on the display screen; In response to the touch or the click, causing the home automation system to change an operational state of a device in the physical room; and dynamically updating, by the controlling app, the appearance of the representation of the device, the representation of the one or more boundaries, and the representation of the one or more pieces of furniture in the 3D virtual room on the display screen to reflect the changed operating state of the device. A method comprising:

2. receiving the navigation command or receiving a signal indicative of a change in position or orientation of the electronic device is receiving a navigation command; The method of claim 1 , wherein the navigation command comprises a touch gesture or cursor movement on a touch-sensitive display screen.

3. receiving the navigation command or receiving a signal indicative of a change in position or orientation of the electronic device comprises receiving a signal indicative of a change in position or orientation of the electronic device; The method of claim 1 , wherein the signal indicative of a change in position or orientation of the electronic device comprises a signal indicative of a change in position or orientation of the electronic device detected using one or more positioning beacons or orientation sensors.

4. changing at least the position or orientation of the virtual camera includes free movement; The method of claim 1 , wherein the virtual camera is freely moved horizontally or vertically through the 3D space to the changed position or freely rotated within the 3D space to the changed orientation.

5. displaying an icon corresponding to a predetermined position in the 3D virtual room on the display screen by the control application; moving the virtual camera to the predetermined position in response to a user selecting the icon. further comprising a predetermined orientation of the virtual camera is associated with the predetermined position; The method of claim 1 , wherein the virtual camera is moved through the 3D space to the predetermined position or rotated within the 3D space to the predetermined orientation.

6. The method of claim 1 , wherein receiving the contact or click comprises receiving a contact or click on a representation of the device on the display screen.

7. displaying, by the controlling app, menu interface elements on the display screen; further comprising The method of claim 1 , wherein receiving the contact or click comprises receiving a contact or click on the menu interface element on the display screen.

8. the device is a lighting device; the operating state of the device includes at least one of an illumination level, a color, or a color temperature; dynamically updating the appearance of the device representation includes displaying the lighting device at an illuminance level, color, or color temperature that reflects the change; 2. The method of claim 1, wherein dynamically updating the appearance of the one or more boundary representations and the one or more furniture representations includes changing shadows and reflections on the one or more boundary representations and the one or more furniture representations caused by the lighting device.

9. the device is an electric blind; the operating state of the device includes a blind position; dynamically updating the appearance of the device representation includes displaying the motorized blinds in a blind position that reflects the change; 2. The method of claim 1, wherein dynamically updating the appearance of the one or more boundary representations and the one or more furniture representations includes changing shadows and reflections on the one or more boundary representations and the one or more furniture representations caused by the position of the blinds.

10. the device is a gas fireplace; the operating conditions of the device include a flame condition; dynamically updating the appearance of the representation of the device includes displaying the gas fireplace with a flame state that reflects the change; 2. The method of claim 1, wherein dynamically updating the appearance of the one or more boundary depictions and the one or more furniture depictions includes changing shadows and reflections on the one or more boundary depictions and the one or more furniture depictions caused by the state of the flame.

11. A non-transitory storage medium readable by an electronic device, encoded with software, said software, when executed on one or more processors of one or more electronic devices, Rendering a three-dimensional (3D) virtual room from a viewpoint defined by a virtual camera in 3D space and displaying the 3D virtual room on a display screen of the electronic device, the rendering being based on data of a plurality of two-dimensional (2D) images of the physical room taken from different positions within the physical room, the 3D virtual room including a representation of one or more devices present in the physical room that are under the control of a home automation system; receiving navigation commands from a user via the display screen or signals from a positioning beacon or orientation sensor indicating a change in the position or orientation of the electronic device; in response to the navigation command or the signal indicating a change in position or orientation of the electronic device, changing at least a position or orientation of the virtual camera in the 3D space and re-rendering and displaying the 3D virtual room from a new viewpoint, wherein the new viewpoint does not correspond to a position in the physical room at which any of the 2D images were taken, and wherein the re-rendering blends together 2D images taken from different positions to display the 3D virtual room from the new viewpoint; receiving a touch or click on the display screen; In response to the touch or the click, causing the home automation system to change an operational state of a device in the physical room; Dynamically updating the appearance of a representation of the device in the 3D virtual room on the display screen to reflect the changed operating state of the device. A non-transitory storage medium readable by an electronic device, operable to:

12. the 3D virtual room further includes a representation of one or more boundaries of the physical room and a representation of one or more pieces of furniture present in the physical room; 12. The non-transitory storage medium readable by an electronic device of claim 11, wherein the software operable to dynamically update comprises software operable to update an appearance of the one or more boundary representations or the one or more furniture representations in the 3D virtual room on the display screen to reflect the changed operating state of the device.

13. changing at least the position or orientation of the virtual camera includes free movement; 12. The non-transitory storage medium readable by an electronic device of claim 11, wherein the virtual camera is freely moved horizontally or vertically through the 3D space to the changed position or freely rotated within the 3D space to the changed orientation.

14. The software comprises: displaying an icon on the display screen corresponding to a predetermined position within the 3D virtual room; In response to a user selecting the icon, the virtual camera is moved to the predetermined position. and further operable as follows: a predetermined orientation of the virtual camera is associated with the predetermined position; The non-transitory storage medium readable by an electronic device of claim 11 , wherein the virtual camera is moved through the 3D space to the predetermined position or rotated within the 3D space to the predetermined orientation.

15. the device is a lighting device; the operating state of the device includes at least one of an illumination level, a color, or a color temperature; 12. The non-transitory storage medium readable by an electronic device of claim 11, wherein dynamically updating the appearance of the device depiction includes displaying the lighting device at an illumination level, color, or color temperature that reflects the change.

16. the device is a gas fireplace; the operating conditions of the device include a flame condition; 12. The non-transitory storage medium readable by an electronic device of claim 11, wherein dynamically updating the appearance of the device depiction includes displaying the gas fireplace with a flame state that reflects the change.

17. the device is an electric blind; the operating state of the device includes a blind position; 12. The non-transitory storage medium readable by an electronic device of claim 11, wherein dynamically updating the appearance of the depiction of the device includes displaying the motorized blinds in a blind position that reflects the change.

18. the device is a display device, the operating state of the device includes a channel, source, or file of media content displayed on the display device; 12. The non-transitory storage medium readable by an electronic device of claim 11, wherein dynamically updating the appearance of the device depiction includes displaying the display device with media content corresponding to the channel, source, or file to which the change is reflected.

19. 1. A home automation system controlled by a user interface including a three-dimensional (3D) virtual room, comprising: a remote controller, mobile device, or head-mounted display (HMD) including a display screen and a control application (control app) configured to render a 3D virtual room from a viewpoint of a virtual camera and display the 3D virtual room on the display screen of the remote controller, mobile device, or HMD, wherein the control app renders the 3D virtual room based on data of a plurality of two-dimensional (2D) images of the physical room taken from different positions within the physical room, the 3D virtual room including representations of one or more devices present in a physical room under control of the home automation system, one or more boundaries of the physical room, and one or more pieces of furniture present in the physical room; and the control app receives from a user the data of a plurality of two-dimensional (2D) images of the physical room taken from different positions within the physical room, the 3D virtual room including representations of one or more devices present in the physical room under control of the home automation system, one or more boundaries of the physical room, and one or more pieces of furniture present in the physical room, the controlling app is further configured to receive a navigation command via a display screen or a signal indicating a change in position or orientation of the remote controller, mobile device, or HMD from a positioning beacon or an orientation sensor, and in response to the navigation command or the signal indicating the change in position or orientation of the remote controller, mobile device, or HMD, change at least a position or orientation of the virtual camera in 3D space to re-render and display the 3D virtual room from a new viewpoint, wherein the new viewpoint does not correspond to a position in the physical room at which any of the 2D images were taken, and the 3D virtual room is re-rendered by blending 2D images taken from different positions to display the 3D virtual room from the new viewpoint, and the controlling app is further configured to receive a touch or a click on the display screen; a controller in communication with the remote controller, mobile device, or head-mounted display (HMD) configured to control the one or more devices present in the physical room, the controller configured to change an operating state of the device in response to the touch or the click received by the control app; further comprising the control app is further configured to, in response to the changed operating state of the device in the physical room, dynamically update the appearance of a representation of the device, the one or more boundary representations, and the one or more furniture representations in the 3D virtual room on the display screen to reflect the changed operating state of the device.

Images