Three-dimensional virtual room based user interface for a home automation system

Abstract

In one embodiment, a user interface for a home automation system based on a user- navigable three-dimensional (3-D) virtual room is provided. Each user-navigable 3-D virtual room shows a substantially photo-realistic depiction of a corresponding physical room of the premises, including the boundaries of the physical room, furniture present in the physical room, and a substantially photo-realistic depiction of devices present in the physical room that are under control of the home automation system. A user can use explicit navigation commands or implicit actions to navigate within the user-navigable 3-D virtual room, moving a virtual camera in 3-D space to view the virtual room from different perspectives. By interacting with (e.g., touching, clicking, etc.) the substantially photo-realistic depictions of devices within the user-navigable 3-D virtual room, a user can indicate a change in state of the corresponding devices in the physical room.

Description

[0001] Related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 898,941, filed September 11, 2019, by Robert P. Madonna et al., entitled “Three Dimensional Virtual Room-Based User Interface for a Home Automation System,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates generally to device control, and more specifically to user interfaces for controlling devices in a home automation system. Background Technology

[0004] As homes and other buildings become larger and filled with more equipment, equipment control becomes an increasing challenge. Traditionally, many devices have been controlled by mechanical switches. While mechanical switches are reliable and cost-effective, they have many limitations, especially when there are many devices located in the same room of a building. For example, a large room can include a large number of lighting fixtures, display devices, electronic curtains, heating, ventilation, and air conditioning (HVAC) equipment, etc. Controlling all of these devices may require a large number of mechanical switches. As the number of mechanical switches in a room increases, availability decreases. Mechanical switches are often unlabeled, or when labeled, only with obscure descriptions (e.g., "Light 1," "Light 2," etc.). Users may be forced to remember which of the many mechanical switches available in a room controls which device. Users who don't remember this relationship often have to rely on trial and error, flipping switches until they produce the desired result.

[0005] Various types of home automation systems have been developed that attempt to overcome the shortcomings of mechanical switches. These systems typically include one or more controllers that manage the operation of the devices. The controllers can be interacted with via a user interface device, such as a dedicated touchscreen unit, which provides a user interface for controlling the devices. The user interface may include an array of touch-sensitive buttons or sliders, where each button or slider controls a device or a group of devices.

[0006] However, such button-centric on-screen user interfaces share many of the same drawbacks as mechanical switches. While buttons and sliders are shown on the screen, rather than physically present like mechanical switches, they operate very similarly. What a button or slider does may not be immediately obvious by looking at it on the screen. Although labels may be available, such labels are often short and obscure due to screen space constraints. While the label may be meaningful to the installer configuring the system, it may have little intrinsic meaning for the user. Similar to the situation with mechanical switches, the user may have to touch every button or slide every slider on the screen to discover, through trial and error, which button or slider achieves the desired result.

[0007] Recently, a device control solution has been developed that addresses many shortcomings of mechanical switches and button-centric on-screen user interfaces. This solution provides a user interface comprising one or more fixed-viewpoint 2D virtual rooms displayed on a touchscreen. Each virtual room shows a 2D depiction of a corresponding physical room within the building. By touching the depiction of a device within the fixed-viewpoint 2D virtual room shown on the screen, the user can instruct the device to change its state, a change performed by a home automation system in that physical room. As the device's state changes within the physical room, the appearance of the fixed-viewpoint 2D virtual room is updated to reflect the changed state.

[0008] While this type of solution addresses many shortcomings of mechanical switches and button-centric on-screen user interfaces and represents significant progress, it can still be improved. One problem with interfaces based on fixed-view 2D virtual rooms is that it requires users to be familiar with the building's layout and the specific names of each room. Multiple different fixed-view 2D virtual rooms can be presented to the user in the interface, and the user needs to choose from them. This selection needs to be repeated as the user moves around the building and wants to control devices in different physical rooms. If users are unfamiliar with the building's layout and what each room is called, they may need to resort to trial and error to select the correct virtual room to use to achieve their desired changes.

[0009] Another problem with interfaces based on fixed-viewpoint 2D virtual rooms is that some types of state changes may be either difficult to represent or appear unnatural. According to existing techniques, fixed-viewpoint 2D virtual rooms are generated by capturing multiple 2D images (e.g., photographs) of a physical room in different states from the same pre-selected perspective. These images include all-off 2D images and device-specific images, where one device is active and all others are deactivated. Different portions of these multiple images are then filtered together to generate a fixed-viewpoint 2D virtual room with devices in different combinations of states. While such filtering works well for some types of state changes (e.g., some lighting state changes), it is impractical for other types of state changes (e.g., colored lighting states, media content states, electronic curtain positions, gas fireplace flame settings, etc.), for which the appearance cannot be well reproduced by combining a small number of 2D images. For example, some colored lighting devices are capable of producing a large number of colors (e.g., 32-bit color). The appearance of all such colors in a physical room may not be easily simulated by filtering a small number of 2D images together; there are simply too many possibilities. Similarly, the state of media content on a television (e.g., channel, source, file, etc.) may not be easily simulated by filtering a small number of 2-D images together; the necessary information simply isn't there. Likewise, filtering isn't always a complete re-creation of the nuances of how multiple devices interact to affect the physical appearance of a room. For example, the interaction of natural daylight caused by the position of electronic curtains, artificial light from lighting fixtures, and ambient light from a gas fireplace flame may not be well reproduced by simply filtering a small number of 2-D images together.

[0010] Another problem with fixed-viewpoint 2D virtual room interfaces is that it can be difficult to pre-select satisfactory viewpoints for certain room shapes or equipment arrangements. Typically, installers will pre-select a small number of viewpoints from which to capture 2D images that show multiple devices at a reasonable size with minimal occlusion. For some rooms with complex shapes, unusual equipment arrangements, etc., it may be difficult or impossible to pre-select a small number of viewpoints that well meet these objectives. While fixed-viewpoint 2D virtual room user interfaces can still be generated, their usability may be reduced.

[0011] Therefore, there is a need for an improved virtual room-based user interface for controlling home automation systems that can solve some or all of these problems. Summary of the Invention

[0012] In one embodiment, a user interface for a home automation system is provided based on a user-navigable 3D virtual room. Each user-navigable 3D virtual room displays a substantially photo-realistic depiction of a corresponding physical room in the building, including the boundaries of the physical room (e.g., walls, ceilings, floors, etc.), the furniture present in the physical room (e.g., sofas, chairs, beds, wall coverings, etc.), and the substantially photo-realistic depiction of the devices present in the physical room that are under the control of the home automation system (e.g., lighting fixtures, display devices, electronic curtains, HVAC equipment, and / or other types of equipment). Users can navigate within the user-navigable 3D virtual room using explicit navigation commands (e.g., move commands or node selection) or implicit actions (e.g., movement of devices detected by positioning beacons and / or orientation sensors), thereby moving a virtual camera in the 3D space to view the virtual room from different perspectives. By interacting with the substantially photo-realistic depiction of the devices within the user-navigable 3D virtual room (e.g., touch, click, etc.), users can indicate changes to the state of the corresponding devices in the physical room. As the state of the devices in the physical room changes, the 3D graphics engine can dynamically update the appearance of the user-navigable 3D virtual room to reflect the changes, so that what the user sees in the virtual room will mimic their experience in the corresponding physical room. Compared to existing interfaces, users can navigate this 3D virtual room-based interface more easily because they can move through 3D space and observe the relationships between rooms. Furthermore, utilizing the 3D graphics engine, the 3D virtual room can display states and appearances that were previously difficult to represent. Even further, the 3D virtual room can be more adaptable to various room shapes and device arrangements.

[0013] A user interface for the user-navigable 3D virtual room can be generated using a 3D mesh model and 2D images of the physical room. In the example, the installer places 3D cameras at multiple locations within the physical room and captures a collection of multiple overlapping 2D images (e.g., 2D panoramic images) and 3D spatial models (e.g., 3D meshes). The overlapping 2D images (e.g., 2D panoramic images) and 3D spatial models (e.g., 3D meshes) are imported into a stitching application, which links (i.e., stitches) the image data to corresponding locations in the 3D spatial model. The stitched 2D images (e.g., 2D panoramic images) and 3D spatial models (e.g., 3D meshes) are then imported into a 3D modeling application. The installer uses the 3D modeling application to correct visual artifacts and uses hit regions to mark depictions of the device, which are mapped to device attributes and identified control commands used to change the device's state. The installer further utilizes 3D modeling applications to assign appearance changes to the depiction of devices that align with the attribute and control commands. The assigned appearance change defines how the appearance should be updated to match the changes in the physical room when a control command is issued. Subsequently, the stitched, artifact-corrected, tagged, appearance-assigned 2D images and 3D spatial model (now referred to as the virtual room) are exported to the control app, which the user can use to control the home automation system and its devices.

[0014] When a virtual camera in a virtual room is positioned relative to a location in a 2-D image (e.g., a 2-D panoramic image) corresponding to its captured position, the control app's 3-D graphics engine can display data from the 2-D image (e.g., the 2-D panoramic image) to add as needed in appearance changes. When the virtual camera moves through a location that does not correspond to any position in the 2-D image (e.g., the 2-D panoramic image), the control app's 3-D graphics engine blends the available 2-D image with a 3-D spatial model (e.g., a 3-D mesh) (e.g., changing its alpha channel and rendering layer) and displays the blended data to add as needed in appearance changes.

[0015] It should be understood that various additional features and alternative embodiments can be implemented. This overview is intended merely as a brief introduction for the reader and does not indicate or imply that the examples mentioned herein cover all aspects of the invention, or any necessary or essential aspects of the invention. Attached Figure Description

[0016] The following description refers to the accompanying drawings, in which:

[0017] Figure 1This is a block diagram of an example architecture for a home automation system that can be operated to control devices in rooms of a building (e.g., a residential or commercial building);

[0018] Figure 2A These are screenshots of an example user-navigable 3D virtual room that can be displayed by the control app;

[0019] Figure 2B-2C yes Figure 2A The example screenshot shows a user navigating a 3-D virtual room, illustrating the free movement of a virtual camera in the 3-D space to view the virtual room from different perspectives.

[0020] Figure 2D-2E yes Figure 2A The example screenshot shows a user-navigable 3-D virtual room, illustrating the movement of a virtual camera in 3-D space to view the virtual room from different perspectives when using navigation nodes.

[0021] Figure 2F-2G yes Figure 2A The example screenshot shows a user-navigable 3D virtual room, illustrating how the lighting of the lighting fixtures changes in response to user interaction with the depiction of the lighting fixtures.

[0022] Figure 2H-2I yes Figure 2A A screenshot of an example user-navigable 3D virtual room illustrates how the state of the display device changes in response to user interaction with the depicted scene.

[0023] Figure 2J-2L yes Figure 2A The example screenshot shows a user-navigable 3D virtual room, illustrating how the state of lighting equipment changes in response to selections in the menu.

[0024] Figure 2M These are screenshots of an example user-navigable 3D virtual room at a higher resolution and without visual artifacts, which more closely approximates commercial implementations.

[0025] Figure 3 This is a flowchart illustrating an example sequence of steps for operating a user interface based on a user-navigable 3D virtual room to control a home automation system; and

[0026] Figure 4 This is a flowchart of an example sequence of steps for generating a user interface based on a user-navigable 3D virtual room to control a home automation system. Detailed Implementation

[0027] definition

[0028] As used herein, the term “home automation system” should be broadly interpreted to encompass all types of home controls, “smart homes,” and / or device control systems that can control devices (e.g., lighting, display devices, electronic curtains, HVAC equipment, and / or other types of equipment) within buildings such as residential or commercial buildings.

[0029] As used herein, the term “physical room” refers to the interior portion of a physical building or the external space associated with a physical building, in which one or more devices can provide services.

[0030] As used herein, the term “virtual room” refers to a digital twin of a physical room represented by a depiction of the interior portion of a physical building or the external space associated with the physical building.

[0031] As used herein, the term "mobile device" refers to an electronic device that runs a general-purpose operating system and is suitable for transport on a person. Devices such as smartphones should be considered mobile devices. Desktop computers, servers, or other primarily stationary computing devices should generally not be considered mobile devices.

[0032] Example Home Automation System

[0033] Figure 1 This is a block diagram of an example architecture 100 for a home automation system that operates to control devices in rooms of a building (e.g., a residential or commercial building). At the heart of the system is a host controller 110 coupled to a local area network (LAN) 150 in the home (e.g., a wired network such as Ethernet and / or a wireless network such as Wi-Fi). The host controller may include hardware components such as a processor, memory, and storage devices that collectively store and execute host software 111, which is configured to monitor and control the operation of devices 112-124, provide a UI for interpretation, system administration and monitoring using beacon 125, synchronize with cloud services 180, remote controls 140, mobile devices 160, and other electronic devices 165 used to control the system, provide activity logging services, provide activity prediction services, and / or other types of functions. The host controller 110 may maintain a home database 130 in its storage device, which stores configuration information, including information about devices 112-124 controlled by the home automation system and the services that the devices can provide, as well as information about control 140, mobile devices 160, and other electronic devices 165 used to control the system.

[0034] The devices 112-124 of the home automation system may include: lighting devices 112, such as lamps, dimmer modules, etc.; interface devices 113, such as keypads, switches, touchscreens, etc.; security devices 114, such as home monitors / cameras, motion sensors, home health sensors, related controllers, etc.; audio devices 116 and video devices 118 (collectively referred to as A / V devices), such as display devices (e.g., televisions, monitors, etc.), A / V device controllers, media servers, audio amplifiers, cable boxes, etc.; electronic door locks 120; electronic curtains 121 and other types of motor-operated devices that generate movement in the room (e.g., TV lifts, automatic doors, etc.); HVAC devices 122, such as thermostat-controlled heating and cooling systems, gas fireplaces, whole-house fans, etc.; interconnection devices 124, such as infrared enhancers, matrix switches, signal extenders, etc.; and other types of home automation system devices. Each of the devices 112-124 may be associated with (i.e., configured to be used in conjunction with) a physical room in the building, and is thus referred to as "in that room". It should be understood that, when used in this context, the term “in” should be interpreted to include equipment that is physically located in a room or located elsewhere (e.g., a remote equipment rack) and provides services to the room from such remote locations.

[0035] Depending on the implementation, the communication capabilities of devices 112-124 in the home automation system can vary. For example, at least some of the devices may include LAN interfaces (e.g., Ethernet or Wi-Fi adapters) and / or Wireless Personal Area Network (WPAN) interfaces (e.g., Bluetooth or Bluetooth Low Energy (BLE) adapters) to enable them to communicate with host controller 110 and other devices. Similarly, some devices may only have ports or transceivers for wired or point-to-point wireless communication (e.g., RS-232, RS-485, General Purpose Input / Output (GPIO), Infrared (IR), etc.) and use such technologies to communicate with host controller 110 and other devices. Some of the devices (e.g., interconnecting devices such as infrared enhancers) may bridge different types of communication, such as including both WPAN interfaces (e.g., Bluetooth or BLE adapters) and point-to-point wireless transceivers (e.g., IR transceivers), as well as bridging between them. Furthermore, some devices may include LAN interfaces (e.g., Ethernet or Wi-Fi interfaces), but are not configured to communicate directly with the host controller 110 or other devices in the home automation system via LAN 150 in the home. Instead, they may access the Internet 170 and cloud services 180 and / or third-party infrastructure 190, which in turn can communicate with the host controller 110. It should be understood that... Figure 1Some of the HVAC devices 122 shown can communicate in this manner. Additionally or alternatively, other types of devices 112-124 can communicate in this manner.

[0036] Home automation systems may include multiple location beacons that transmit and receive WLAN, WPAN, or other wireless signals (e.g., Bluetooth, BLE, Wi-Fi, Ultra-Wideband (UWB), Radio Frequency Identification (RFID), or other signals) that can be used to determine the location of a remote control 140, mobile device 160, or other electronic device 165 within the building. Location can be determined using Received Signal Strength (RSS) to select the nearest beacon location, or by performing trilateration and / or other techniques based on multiple beacon locations and their associated signal strengths. These beacons may be standalone devices, such as a standalone beacon 125, or integrated into one or more of devices 112-124 that provide additional functionality. In one implementation, the beacons are integrated into a lighting device 112 and a keypad, with the lighting device 112 providing both lighting and location functionality, and the keypad providing both a user interface and location functionality.

[0037] Users can use remote control 140 to control devices 112-124 of the home automation system. Remote control 140 may include a touch-sensitive display, physical buttons, communication interfaces (e.g., IR, WPAN, etc.), a processor, memory, and a storage device that stores and executes a control app configured to interface with host controller 110 and cloud service 180. The remote control may also include an orientation sensor, which, together with a positioning beacon, allows determination of the remote control 140's location and orientation relative to the building. The control app on remote control 140 may present a user interface based on a user-navigable 3D virtual room, which, among other functions, is used to control devices 112-124 of the home automation system 100.

[0038] Users can also use mobile device 160 to control devices 112-124 of the home automation system. Mobile device 160 may include a touch-sensitive display, a communication interface (e.g., Wi-Fi, WPAN, etc.), a processor, memory, and a storage device that stores and executes a control app 162 configured to interface with host controller 110 and / or cloud service 180. Mobile device 160 may also include an orientation sensor that, together with a positioning beacon, allows determination of the mobile device 160's position and orientation relative to the building. The control app on mobile device 160 may present a user interface based on a user-navigable 3D virtual room, which, among other functions, is used to control devices 112-124 of the home automation system 100.

[0039] Furthermore, the user can use another electronic device 165 to control devices 112-124 of the home automation system. This other electronic device 165 may be a tablet computer, a head-mounted display (HMD) such as Google Glass® HUD, a dedicated touchscreen unit, a television and remote control, a computer and mouse, or other types of technology. Electronic device 165 may include a display screen (e.g., touch-sensitive, non-touch-sensitive, HMD, etc.), input devices, communication interfaces (e.g., Wi-Fi, WPAN, etc.), a processor, memory, and a storage device that stores and executes software configured to interface with host controller 110 and / or cloud service 180.

[0040] Electronic device 165 may also include an orientation sensor, which, together with a positioning beacon, allows the determination of the electronic device 165's position and orientation relative to the building. For example, in an implementation where electronic device 165 is an HMD and the beacon is a BLE beacon, position can be determined by BLE trilateration, and orientation can be determined by head movement. A control app can present a 3-D virtual room-based user interface on the HMD for controlling devices 112-124 of the home automation system, and the user can make selections using the HMD's input devices.

[0041] It should be understood that electronic device 165 may also include multiple individual devices that operate together. For example, in an implementation where electronic device 165 is a television and a remote control, the control app can present a user interface based on a 3-D virtual room on the television screen and can receive selections on the remote control (e.g., selecting items by moving the cursor on the screen).

[0042] Remote controller 140, mobile device 160, or electronic device 165 can communicate with host controller 110 to control devices. Some devices (e.g., mobile device 160, electronic device 165, etc.) can communicate with cloud service 180 and its host application programming interface (API) 182 and mobile API 184. Among other functions, cloud service 180 can provide remote access to home automation control, persistent backup of home database 130 (data stored in configuration database 186), interface to third-party infrastructure (via third-party adapter 188), user profiles and usage tracking (data stored in user database 189), mechanisms for over-the-air updates, host crash reporting, and license management.

[0043] Operation based on the user interface of a user-navigable 3D virtual room

[0044] A control app on remote control 140, mobile device 160, or other electronic device 165 can present a 3-D virtual room-based user interface for controlling devices 112-124 of the home automation system 100. This interface can include multiple user-navigable 3-D virtual rooms, each showing a substantially photorealistic depiction of a corresponding physical room in the building. Each user-navigable 3-D virtual room can include substantially photorealistic depictions of the boundaries of the physical room (e.g., walls, ceiling, floor, etc.), the furniture present in the physical room (e.g., sofas, chairs, beds, wall coverings, etc.), and the devices 112-124 present in the physical room (e.g., lighting fixtures, display devices, electronic curtains, and / or other types of devices). Each of the devices 112-124 can have a number of possible states. Depending on devices 112-124, there may be a binary set of possible states (e.g., an invalid "off" state and an effective "on" state) or a larger set of many states (e.g., multiple lighting levels, colors (e.g., 32-bit color), color temperatures (e.g., 3000K, 5000K, etc.), media content (e.g., TV channel, source, individual media file, etc.), location, temperature, etc.).

[0045] Figure 2A-2M These are screenshots of a sample user-navigable 3D virtual room that can be displayed by the control app, corresponding to the open-plan layout of a physical building's kitchen and living room. Although Figure 2A-2L The screenshots are low-resolution and contain visual artifacts, but it should be understood that commercial implementations will preferably have high resolution, where such artifacts are corrected to make them look essentially photographic. Figure 2M An example is shown that more closely approximates the preferred appearance of a commercial implementation.

[0046] Figure 2A This is a screenshot of an example user-navigable 3D virtual room that can be displayed by a control app. The user-navigable 3D virtual room includes a substantially photorealistic depiction of the boundaries of the physical room (such as floor 210 and walls 212), the furniture in the physical room (such as sofa 220, table and chairs 222 and stove 224), and the devices in the physical room under the control of the home automation system 100 (such as chandelier 230, recessed canister lights 232-238 and television 239).

[0047] Users can navigate within a virtual room using explicit navigation commands or implicit actions to move a virtual camera in 3D space to view the virtual room from different perspectives. Explicit navigation commands can take various forms. In one implementation, explicit navigation commands can take the form of movement commands (e.g., touch gestures, such as scrolling, swiping, etc. on a touch-sensitive display, cursor movement, etc.). Navigation can include free movement, where the virtual camera can freely translate horizontally or vertically through 3D space and freely rotate to different orientations within 3D space.

[0048] Figure 2B-2C yes Figure 2A A screenshot of an example user-navigable 3D virtual room illustrates the free movement of a virtual camera in 3D space to view the virtual room from different perspectives. A movement icon 240 can be displayed, which is shifted when a movement command (e.g., scroll gesture, cursor movement, etc.) is received. In this example, the virtual camera is... Figure 2B and Figure 2C It moves horizontally forward between them.

[0049] In another implementation, explicit navigation commands can take the form of node selection. Multiple predefined nodes can be positioned at predetermined locations and represented as icons within the virtual room. In response to a user selection of a node (e.g., a touch on a node on a touch-sensitive display, selection using a cursor, etc.), the node is selected, and the virtual camera is moved (e.g., "snap") to its position. Such movement can be depicted in a "smooth" manner, where the virtual camera translates spatially, and the virtual room is continuously updated to represent the movement. Each node can be associated with a predetermined starting orientation. Once at a node, the virtual camera can freely rotate in 3D space to different orientations in response to navigation commands.

[0050] Figure 2D-2E yes Figure 2A A screenshot of an example user-navigable 3D virtual room illustrates the movement of a virtual camera in 3D space to view the virtual room from different perspectives when using navigation nodes. Multiple selectable nodes 245, 247 are shown, which can be selected (e.g., touched, clicked, etc.). In this example, in Figure 2D Selecting node 245 caused the virtual camera to be panned to... Figure 2C The perspective shown.

[0051] Implicit actions can take various forms. In one implementation, implicit actions can be based on the position and orientation of the remote control 140, mobile device 160, or other electronic device 165, which is determined using positioning beacons (e.g., and their Bluetooth, BLE, Wi-Fi, UWB, RFID, or other signaling) and orientation sensors. A user can freely pan the virtual camera by walking around the physical room while holding the remote control 140, mobile device 160, or other electronic device 165. A user can freely rotate the virtual room by rotating the remote control 140, mobile device 160, or other electronic device 165. In the case where the electronic device is an HMD, the user's head position and orientation can be directly panned to the position and orientation within the virtual room.

[0052] By interacting with substantially photorealistic depictions of devices within a user-navigable 3D virtual room (e.g., touching, clicking, etc.), users can indicate changes in the state of corresponding devices in the physical room. State changes can cycle through the available states of a device (e.g., between binary states, between a large number of possible states, etc.). When a device's state is changed, the 3D graphics engine controlling the app (e.g., Unity® or Unreal® graphics engine) dynamically updates the appearance of the user-navigable 3D virtual room to reflect the change, so that what the user sees in the virtual room mimics their experience in the corresponding physical room. Dynamic updates can involve altering the appearance of the photorealistic depiction of each device whose state is changed (e.g., lighting level, color, color temperature, media content, position, or other visual attributes). Dynamic updates can also involve altering the appearance of borders, furniture, and substantially photorealistic depictions of other devices that are not currently changing their state (e.g., shadows and reflections) to depict the effect of the state change on these items. In this way, when a state is changed, the 3D graphics engine in the virtual room mimics the experience the user would observe in the physical room.

[0053] Figure 2F-2G yes Figure 2A The example screenshot shows a user-navigable 3D virtual room, illustrating how the lighting changes in response to user interaction with the lighting fixtures in a largely photorealistic depiction. Figure 2FIn this system, the user interacts with a depiction of a lighting device, specifically a recessed can light fixture 232 (e.g., by touching, clicking, etc.). In response to such interaction, the control app causes the home automation system 100 to activate the recessed can light fixture 232 in the physical room. The control app's graphics engine further dynamically updates the appearance of the depiction of the recessed can light fixture 232 in the virtual room, making it appear illuminated (e.g., by imposing a virtual light source at its location), and based on this change, dynamically updates the appearance of depictions of boundaries (e.g., shadows and reflections 250 on walls), furniture (e.g., shadows and reflections 254 on a sofa), and other devices (e.g., shadows and reflections 252 on a chandelier), such as... Figure 2G As shown in the image.

[0054] Figure 2H-2I yes Figure 2A The screenshot illustrates an example of a user-navigable 3D virtual room, showing the change in the state of the display device in response to user interaction with a substantially photorealistic depiction of the device. Here, the state is the media content state, i.e., the channel (e.g., a television channel). However, it should be understood that the media content state can take other forms, such as the source (e.g., a signal from a DVD, cable box, etc.), media files (e.g., movie files, TV program files, etc.). Figure 2H In this context, the user interacts with the display device, specifically the depiction of television 239 (e.g., touch, click, etc.). In response to such interaction, the control app causes the home automation system 100 to change the channel of the television in the physical room, here from channel 6 to channel 1. Such changes may involve cycling through multiple intermediate channels (e.g., in response to repeated touches, clicks, etc.). The control app's graphics engine further dynamically updates the appearance of the depiction of television 239, so that the actual media content of that channel is displayed within television 239 in the user-navigable 3-D virtual room, such as... Figure 1 As shown in the image.

[0055] In addition to changes in lighting from lighting fixtures and changes in the state of display devices, it should be remembered that a wide variety of other types of state changes can be made to other types of devices 112-124. For example, in response to user interaction with a substantially photorealistic depiction of electronic curtains, the control app can cause the home automation system 100 to activate the curtain motors to change the curtain position (e.g., open or close the curtains). The control app's graphics engine further dynamically updates the appearance of the depiction of the electronic curtains in the virtual room so that it reflects the new curtain position, and dynamically updates the appearance of the depictions of boundaries, furniture, and other devices based on this change (e.g., changes in shadows and reflections due to more or less natural light entering the room through the windows).

[0056] Similarly, in response to user interaction with a virtually photorealistic depiction of the gas fireplace, the control app enables the home automation system 100 to signal the electronic ignition and gas supply systems to adjust the flame. The control app's graphics engine further dynamically updates the appearance of the depiction of the gas fireplace in the virtual room, making it reflect the changing flame state, and dynamically updates the appearance of boundaries, furniture, and other equipment based on the changing flame state (e.g., changing shadows and reflections due to the amount of flame in the fireplace).

[0057] When a large number of devices are present, it can be difficult to locate a substantially photorealistic depiction of the desired device in the virtual room. Similarly, when there are numerous states for a device, looping through states may be inefficient or impractical. In such cases, the user-navigable 3D virtual room can be configured to display a menu in response to user interaction with interface elements. This menu can list various controllable devices and their states. The user can select (e.g., by touch, click, etc.) the desired device and state. The control app can cause the home automation system to make the desired state changes, and the 3D graphics engine can dynamically update the appearance of the user-navigable 3D virtual room to reflect these changes, so that what the user sees in the virtual room will mimic their experience in the corresponding physical room.

[0058] Figure 2J-2L yes Figure 2A The example screenshot shows a user navigating a 3-D virtual room, illustrating how the state of lighting equipment changes in response to selections in a menu. Figure 2J In this context, the user interacts with the menu interface element 260 (e.g., touch, click, etc.). In response to such interaction, the control app causes the menu 270 to be displayed, for example, overlaid on the virtual room. Figure 2K As shown in the example. In this example, menu 270 includes lighting fixtures in the physical room, and possible states of such fixtures, such as lighting level, color, color temperature, etc. The user selects a lighting fixture, in this example, an inset can luminaire 238, along with the lighting level and color. The control app then instructs the home automation system to illuminate the inset can luminaire 238 to the desired level with the desired color. A wide variety of colors can be supported (e.g., 32-bit color). The control app's graphics engine further dynamically updates the appearance of the depiction of the inset can luminaire 238 so that it appears to be illuminated to the desired level with the desired color (e.g., by imposing a virtual light source at its location), and dynamically updates the appearance of the depiction of boundaries, furniture, and other fixtures in the room (e.g., shadows and reflections on the depiction), such as... Figure 2LAs shown in the diagram. When a large number of supported colors (e.g., 32-bit color) exist, the ability to see what a virtual room would look like by observing virtual rooms with different colors can greatly simplify control.

[0059] Figure 2M These are screenshots of an example user-navigable 3D virtual room at a higher resolution and without visual artifacts, which more closely approximates commercial implementations. As can be seen, the depiction of controlled devices (such as a TV 239), boundaries (such as walls), furniture (such as a sofa), and other equipment appears substantially photorealistic. It should be understood that... Figure 2A-2L The virtual room shown can appear in this way.

[0060] Figure 3 This is a flowchart of an example sequence of steps for operating a user interface based on a user-navigable 3-D virtual room to control devices 112-124 of a home automation system 100. Figure 3 The steps below summarize the operations discussed in more detail above. At step 310, the control app on the remote control 140, mobile device 160, or other electronic device 165 uses a graphics engine to present a user-navigable 3D virtual room from a first-person perspective defined by a virtual camera. This user-navigable 3D virtual room may include a substantially photorealistic depiction of the boundaries of the physical room (e.g., walls, ceiling, floor, etc.), the furniture present in the physical room (e.g., sofas, chairs, beds, wall coverings, etc.), and the devices present in the physical room that are controlled by the home automation system 100 (e.g., lighting fixtures, display devices, electronic curtains, HVAC equipment, and / or other types of equipment). At step 320, the control app displays the presented user-navigable 3D virtual room on the display screen (e.g., a touch-sensitive display) of the remote control 140, mobile device 160, or other electronic device 165, within the control app.

[0061] At step 330, the control app determines whether any explicit navigation commands (e.g., move commands or node selections) or implicit actions (e.g., changes in the position or orientation of remote control 140, mobile device 160, or other electronic device 165) have been received. If so, at step 340, the control app responds by changing the viewpoint by altering the position and / or orientation of the virtual camera and loops back to step 310, where the graphics engine re-renders the virtual room from this new viewpoint. If not, execution proceeds to step 350, where the control app determines whether the user has interacted with a substantially photorealistic depiction of the devices within the user-navigable 3-D virtual room (e.g., touch, click, etc.). If so, at step 360, the control app causes the home automation system 100 to change the state of the devices in the physical room. Further, at step 370, the control app dynamically updates the substantially photorealistic appearance of the device (e.g., lighting levels, colors, color temperature, media, media content, location, or other visual attributes) and the substantially photorealistic appearance of the boundaries, furniture, and other equipment in the virtual room (e.g., shadows and reflections). Execution then loops back to step 310, where the control app's graphics engine uses these new appearances to re-render the virtual room.

[0062] If not, execution proceeds to step 380, where the control app determines whether the user has interacted with the menu interface elements (e.g., touched, clicked, etc.). If yes, at step 390, a menu overlaid on the user-navigable 3-D virtual room is displayed. At step 395, the control app determines whether a device and status have been selected in the menu. If yes, execution loops to step 360, where the control app causes the home automation system 100 to change the status of devices in the physical room. Then, at step 370, the control app dynamically updates the substantially photorealistic depiction of the selected device, as well as the substantially photorealistic depiction of the boundaries, furniture, and other devices in the virtual room, based on the selected status. Execution then loops back to step 310, where the control app's graphics engine re-renders the virtual room from these new appearances. If not, the control app waits for further user input and execution loops back to step 330.

[0063] Generation of User Interface for User-Navigable 3D Virtual Rooms

[0064] The user interface for this 3-D virtual room is typically generated by a combination of data collection and configuration operations performed by a configuration application running on a local computing device and / or in the cloud, and rendering operations performed by the graphics engine of a control app running on a remote control 140, mobile device 160, or other electronic device 165. Figure 4 This is a flowchart of an example sequence of steps for devices 112-124 used to generate a user interface based on a user-navigable 3D virtual room to control a home automation system. Steps 410-480 represent data collection and configuration operations, while steps 485-495 represent presentation operations.

[0065] At step 410, the installer places the 3-D camera at multiple locations within the physical room and captures a collection of multiple overlapping 2-D images (e.g., 2-D panoramic images) and 3-D spatial models (e.g., 3-D meshes). The 3-D camera can use any of a variety of imaging and scanning techniques, such as single-point laser scanning, line-contour laser scanning, structured light (non-laser) detection, stereo vision, etc., to generate the 3-D spatial model. Preferably, during capture, the device is entirely in a deactivated or "off" state to simplify the later generation of the appearance effect.

[0066] At step 420, 2-D images (e.g., 2-D panoramic images) and 3-D spatial models (e.g., 3-D meshes) are imported from a 3-D camera into a stitching application, which can be executed in the cloud or on a local computing device. In one implementation, the stitching application may be a Matterport® cloud-based software package. At step 430, the installer utilizes the stitching application to stitch the 2-D images (e.g., 2-D panoramic images) and 3-D spatial models (e.g., 3-D meshes) together to link (i.e., stitch) the image data to corresponding locations within the 3-D spatial model.

[0067] At step 440, the stitched 2D image and 3D spatial model are imported into a 3D modeling application, which can be executed in the cloud or on a local computing device. In one implementation, the 3D modeling application can be a Unity® or Unreal® 3D development platform. At step 450, the installer uses the 3D modeling application to correct visual artifacts. Visual artifacts can be caused by various factors during the capture and stitching process. For example, reflective surfaces such as displays or window glass are often not captured well and may introduce visual artifacts that need correction. At step 460, the installer uses the 3D modeling application to mark the device depiction using hit areas (i.e., 3D hit boxes) and maps these hit areas to device attributes and control commands of the home automation system 100 for changing the device's state. For example, lighting devices can be marked using hit areas surrounding their outer perimeter and mapped to lighting attributes of the lighting load controlled by certain lighting control commands (e.g., to change lighting level, color, color temperature, etc.). Similarly, a display device can be marked with a hit area around its screen and mapped to display attributes of the display device controlled by certain control commands that affect the state of the media content (e.g., channel, source, file, etc.). Likewise, an electronic curtain can be marked with a hit area around its outer perimeter and mapped to movement attributes of the electronic curtain controlled by certain position control commands.

[0068] At step 470, the installer uses the 3D modeling application to assign appearance changes to the depiction of devices consistent with their attributes and control commands. The assigned appearance change defines how the control app's graphics engine should update the device depiction to match changes occurring in the physical room when a control command is issued, and how the appearance change should affect the appearance of boundaries, furniture, and other equipment in the room. The assigned appearance change can have types and boundaries based on device attributes. At step 480, the artifact-corrected, tagged, appearance-assigned, stitched 2D image and 3D spatial model (now referred to as the virtual room) are exported to the control app for inclusion in the user interface of the user-navigable 3D virtual room.

[0069] The virtual room is rendered by the graphics engine of the control app. At step 485, the control app determines whether the virtual camera indicating the user's desired viewpoint is located at a position corresponding to one of the positions captured from it in a 2-D image (e.g., a 2-D panoramic image). If yes, then at step 485, the control app's graphics engine renders the virtual room using data from the 2-D image (e.g., the 2-D panoramic image) captured from that position. If no, then at step 495, the control app's graphics engine blends the available 2-D images (e.g., the 2-D panoramic image) according to a 3-D spatial model (e.g., a 3-D mesh) (e.g., changing its alpha channel and rendering layer), and uses the blended data to render the virtual room.

[0070] In summary, a user interface for a user-navigable 3-D virtual room based on devices for controlling a home automation system is provided. While certain specific examples have been used in the description above, it should be apparent that numerous modifications and / or additions can be made thereto. For example, while it has been discussed above that each of the remote control 140, mobile device 160, or other electronic device 165 can have a touch-sensitive display and can utilize gestures and touch to make user input in the user interface for a user-navigable 3-D virtual room, it should be understood that the interface can be adapted to a non-touch-sensitive display and can receive user input via a pointing device and cursor (e.g., with selection made by clicking on an item) or other types of input devices.

[0071] Similarly, while the user interface based on a user-navigable 3D virtual room described above can be used to control a configured home automation system 100 in a building, such a user interface can also be adapted for use in a sales or setup role in previewing or pre-configuring a home automation system. For example, the effects that can be produced in a building using a user-navigable 3D virtual room can be shown to the user before purchase. Alternatively, during the pre-configuration process when the system is first installed or set up, possible effects can be shown to the user. In such cases, the effects may not actually occur in the physical room at the time of display.

[0072] Furthermore, while the foregoing has discussed user-navigable 3D virtual rooms mimicking the appearance of physical rooms and various types of visual appearances, it should be understood that appearance can also include non-visual aspects of the experience within the physical room, such as sound. In such cases, the control app can play audio playing in the physical room and / or sound effects mimicking ambient sounds in the physical room through the speakers of the remote control 140, mobile device 160, and other electronic devices 165. For example, when a user activates the TV 239 and changes it to a channel, the actual audio of that channel can be played by the speakers of the remote control 140, mobile device 160, and other electronic devices 165 accompanied by the visual display of the user-navigable 3D virtual room. Similarly, when a user changes the position of electronic curtains, sound effects mimicking the curtains rising or falling can be played by the speakers of the remote control 140, mobile device 160, and other electronic devices 165 accompanied by the visual display of the user-navigable 3D virtual room.

[0073] Furthermore, while it has been discussed above that the state of the equipment in the physical room can change in response to user interaction with a substantially photorealistic depiction of the equipment (e.g., user touch, click, etc. on the depiction of the equipment in the user-navigable 3-D virtual room), it should be understood that some changes in state can be configured to be triggered at predefined times or in response to the fulfillment of predetermined conditions. In one embodiment, a user can interact with the system to configure the lighting level, color and / or color temperature and / or other states of the lighting equipment to change dynamically throughout the day, thereby providing circadian lighting. Such changes in state can be based at least in part on outdoor sensors that capture current lighting data of the outdoor environment. The appearance of the depictions of lighting equipment, boundaries, and furniture in the user-navigable 3-D virtual room is updated to reflect the changed state achieved by the circadian lighting.

[0074] Finally, it should be understood that the steps described above can be implemented in hardware, software (specifically, a non-transitory electronic device readable medium including software), firmware, or a combination thereof. A non-transitory electronic device readable medium can take the form of memory such as random access memory (RAM), a disk such as a hard disk drive or flash memory device, or other tangible storage media. Generally, it should be understood that the above description is intended to be illustrative only.

Claims

1. A method for controlling a home automation system using a user interface based on a user-navigable three-dimensional (3-D) virtual room, comprising: A user-navigable 3D virtual room is presented and displayed by executing a control application (app) with a 3D graphics engine on an electronic device. The user-navigable 3D virtual room includes data from 2D images of the physical room captured from various locations within the physical room and linked to positions in a 3D spatial model. The 3D graphics engine is used to present and display the user-navigable 3D virtual room from a perspective defined by a virtual camera in the 3D space. This perspective coincides with a position associated with a 2D image by accessing image data of a 2D image linked to a corresponding position in the 3D spatial model. The user-navigable 3D virtual room includes a depiction of one or more devices present in the physical room under the control of the home automation system, a depiction of one or more boundaries of the physical room, and a depiction of one or more pieces of furniture present in the physical room. Receive explicit navigation commands or implicit actions from the user; In response to the explicit navigation command or implicit action, the virtual camera can be translated or rotated via the control app to change its position or orientation in the 3D space; The control app uses a 3D graphics engine to re-represent and display a user-navigable 3D virtual room from a new perspective defined by a changed location or orientation. This new perspective does not coincide with any location associated with any 2D image. The 3D graphics engine re-represents the user-navigable 3D virtual room by accessing image data from multiple 2D images captured from different locations and linked to different locations in the 3D spatial model, and by fusing the image data from the multiple 2D images, in order to display the user-navigable 3D virtual room from a new perspective. Receive user interaction; In response to the user interaction, the home automation system changes the state of the devices in the physical room; The 3D graphics engine of the control app dynamically updates the appearance of the depictions of devices, one or more boundaries, and one or more pieces of furniture in the user-navigable 3D virtual room to reflect the changing state; and The control app re-presents the user-navigable 3D virtual room with an updated look.

2. The method of claim 1, wherein receiving an explicit navigation command or an implicit action is receiving an explicit navigation command, and the explicit navigation command includes a touch gesture on a touch-sensitive display screen or movement of a cursor.

3. The method of claim 1, wherein receiving an explicit navigation command or an implicit action is receiving an implicit action, and the implicit action includes a change in the position or orientation of the electronic device detected using one or more positioning beacons or orientation sensors.

4. The method of claim 1, wherein translating or rotating the virtual camera involves free movement, wherein the virtual camera is freely translated or rotated in the 3-D space to the position or orientation.

5. The method of claim 1, wherein translating or rotating the virtual camera involves moving the virtual camera to a predefined node, wherein the virtual camera is rotated in the 3-D space to a predetermined orientation of the predefined node.

6. The method of claim 1, wherein the user interaction is an interaction with a depiction of devices within the user-navigable 3-D virtual room.

7. The method of claim 1, further comprising: The menu is displayed via the control app, and The user interaction mentioned above refers to the interaction with the menu interface element of the menu.

8. The method of claim 1, wherein the device is a lighting device, the state of the device includes at least one of lighting level, color, or color temperature, and dynamically updating the appearance of the depiction of the device includes showing a lighting device having the lighting level, color, or color temperature, and dynamically updating the appearance of the depiction of one or more boundaries and the depiction of one or more pieces of furniture includes changing the shadows and reflections caused by the lighting device on the depiction of one or more boundaries and the depiction of one or more pieces of furniture.

9. The method of claim 1, wherein the device is an electronic curtain, the state of the device includes a curtain position, and dynamically updating the appearance of the depiction of the device includes showing a curtain with the curtain position, and dynamically updating the appearance of the depiction of one or more boundaries and the depiction of one or more pieces of furniture includes changing the shadows and reflections on the depiction of one or more boundaries and the depiction of one or more pieces of furniture caused by the curtain position.

10. The method of claim 1, wherein the device is a gas fireplace, the state of the device includes a flame state, and dynamically updating the appearance of the depiction of the device includes showing a gas fireplace with the flame state, and dynamically updating the appearance of the depiction of one or more boundaries and the depiction of one or more pieces of furniture includes changing the shadows and reflections on the depiction of one or more boundaries and the depiction of one or more pieces of furniture caused by the flame state.

11. The method of claim 1, wherein the depiction of one or more devices, the depiction of one or more boundaries, and the depiction of one or more pieces of furniture are each substantially photorealistic.

12. A non-transitory electronic device readable medium having coded software thereon, said software being operable when executed on one or more processors of one or more electronic devices for: A 3D graphics engine is used to render and display a user-navigable 3D virtual room. The user-navigable 3D virtual room contains data from 2D images of the physical room captured from various locations within the physical room and linked to positions in a 3D spatial model. The 3D graphics engine is used to render and display the user-navigable 3D virtual room from a perspective defined by a virtual camera in the 3D space to show depictions of one or more devices in the physical room under the control of a home automation system, depictions of one or more boundaries of the physical room, and depictions of one or more pieces of furniture in the physical room. The perspective coincides with the position associated with the 2D image by accessing image data of the 2D image linked to the corresponding position in the 3D spatial model of the user-navigable 3D virtual room. Receive explicit navigation commands or implicit actions from the user; In response to the explicit navigation command or implicit action, the virtual camera is translated or rotated to change its position or orientation in the 3D space; The 3D graphics engine is used to re-present and display the 3D virtual room from a new perspective defined by the changed position or orientation, wherein the new perspective does not coincide with the position associated with any 2D image, and the 3D graphics engine re-presents the user-navigable 3D virtual room by accessing image data from multiple 2D images captured from different locations and linked to different locations in the 3D spatial model and fusing the image data from the multiple 2D images, so as to display the user-navigable 3D virtual room from a new perspective; Receive user interaction; In response to the user interaction, the home automation system changes the state of the devices in the physical room; The appearance of the device's depiction, as well as the appearance of one or more boundaries or one or more pieces of furniture in the user-navigable 3D virtual room, is dynamically updated using a 3D graphics engine to reflect the changing state; and The user-navigable 3D virtual room is re-presented with an updated look.

13. The non-transitory electronic device readable medium of claim 12, wherein translating or rotating the virtual camera involves free movement, wherein the virtual camera is freely translated or rotated in the 3-D space to the position or orientation.

14. The non-transitory electronic device readable medium of claim 12, wherein translating or rotating the virtual camera involves moving the virtual camera to a predefined node, wherein the virtual camera is rotated in the 3-D space to a predetermined orientation of the predefined node.

15. The non-transitory electronic device readable medium of claim 12, wherein the device is a lighting device, the state of the device includes at least one of lighting level, color, or color temperature, and dynamically updating the depicted appearance of the device includes showing a lighting device having said lighting level, color, or color temperature.

16. The non-transitory electronic device readable medium of claim 12, wherein the device is a gas fireplace, the state of the device includes a flame state, and dynamically updating the appearance of the depicted device includes showing a gas fireplace having the flame state.

17. The non-transitory electronic device readable medium of claim 12, wherein the device is an electronic curtain, the state of the device includes the curtain position, and dynamically updating the depicted appearance of the device includes showing the curtain having the curtain position.

18. The non-transitory electronic device readable medium of claim 12, wherein the device is a display device, the state of the device includes a media content state, and dynamically updating the depicted appearance of the device includes displaying a display device having media content corresponding to the media content state on its display screen.

19. A home automation system controlled by a user interface based on a user-navigable three-dimensional (3-D) virtual room, comprising: A remote control, mobile device, or head-mounted display (HMD), including a display screen and a control application (app) with a 3D graphics engine configured to render and display a user-navigable 3D virtual room on the display screen. The user-navigable 3D virtual room contains data from 2D images captured from various locations within a physical room and linked to locations in a 3D spatial model. The 3D graphics engine is used to render and display the user-navigable 3D virtual room from the perspective of a virtual camera in the 3D space, in order to display a depiction of one or more devices present in the physical room under the control of the home automation system, a depiction of one or more boundaries of the physical room, and one or more devices present in the physical room. The depiction of furniture, wherein the perspective coincides with the location associated with the 2D image by accessing image data of a 2D image linked to a corresponding location in a user-navigable 3D virtual room within a 3D spatial model, the perspective of the 3D virtual camera being translated or rotated in 3D space in response to explicit navigation commands or implicit actions from the user, wherein at least some perspective changes cause the 3D graphics engine to re-represent the user-navigable 3D virtual room by accessing image data from multiple 2D images captured from different locations and linked to different locations in the 3D spatial model and fusing image data from the multiple 2D images associated with different locations, so as to display the 3D virtual room from a new perspective that does not coincide with the location associated with any 2D image; and A controller that communicates with the remote control, mobile device, or HMD, the controller being configured to control one or more devices present in the physical room, the controller being configured to change the state of the devices in response to user interaction with the user-navigable 3D virtual room. The 3D graphics engine of the control app is further configured to dynamically update the appearance of the depiction of the devices, one or more boundaries, and one or more pieces of furniture in the user-navigable 3D virtual room to reflect the changed state and to re-represent the user-navigable 3D virtual room with the updated appearance.

Images