Lighting control UI for controlling a physical lighting device and a virtual lighting device

Abstract

A system is configured to provide, on a first user device, a user interface (41) representing multiple lighting devices, including a virtual lighting device (48) and a physical lighting device (46,47). The user interface indicates which of the represented lighting devices is physical and which of the represented lighting devices is virtual. A visual representation of the virtual lighting device is displayed in a mixed reality environment on a second user device. The system is further configured to receive one or more user inputs indicative of a first light setting for the physical lighting device and indicative of a second light setting for the virtual lighting device, control the physical lighting device according to the first light setting, and cause the second user device to represent a light effect rendered by the virtual lighting device, corresponding to the second light setting, in the mixed reality environment.

Description

[0001] Lighting control UI for controlling a physical lighting device and a virtual lighting device

[0002] FIELD OF THE INVENTION

[0003] The invention relates to a system for receiving user input indicative of a light setting and controlling one or more lighting devices according to said light setting.

[0004] The invention further relates to a method of receiving user input indicative of a light setting and controlling one or more lighting devices according to said light setting.

[0005] The invention also relates to a computer program product enabling a computer system to perform such a method.

[0006] BACKGROUND OF THE INVENTION

[0007] As smart phones and Augmented Reality (AR) devices are becoming more and more capable of merging physical world (e.g. camera stream) and virtual objects, the number of mixed reality applications is increasing. It will likely become common to use such applications for different purposes such as previewing and shopping for new house related goods, controlling smart home connected devices, previewing and merging physical and virtual objects (e.g., for social activities, gaming, etc.), etc.

[0008] US 2019 / 340306 Al discloses a system for collecting, managing and accessing lighting source metrics. DI further discloses methods and systems of augmented reality -based lighting design, which include detecting light sources in an augmented reality image; detecting at least one of surfaces and objects in the augmented reality image; facilitating disposition of at least one virtual light source in the augmented reality image, resulting in an updated augmented reality image; processing a near field and far field luminance characterization of the at least one virtual light source and the updated augmented reality image with a lighting space model; and depicting illumination of portions of the augmented reality image in response to the lighting space model.

[0009] The mixed reality system disclosed in US 2021 / 0110598 Al is able to show a virtual lighting device in the same room as a physical light device. However, unlike other household objects, simply adding a virtual lighting device to a room is not enough; it is important for the user to be able to adjust the virtual light effected rendered by the virtual lighting device to fully experience the virtual lighting device’s overall impact on the room atmosphere.

[0010] SUMMARY OF THE INVENTION

[0011] It is a first object of the invention to provide a system, which allows a user to adjust the virtual light effect rendered by a virtual lighting device represented in a mixed reality environment in an intuitive manner.

[0012] It is a second object of the invention to provide a method, which allows a user to adjust the virtual light effect rendered by a virtual lighting device represented in a mixed reality environment in an intuitive manner.

[0013] In a first aspect of the invention, a system for receiving user input indicative of a light setting and controlling one or more lighting devices according to said light setting comprises at least one transmitter and at least one processor configured to provide, on a first user device, a user interface representing a plurality of lighting devices, said plurality of lighting devices comprising a virtual lighting device and a physical lighting device, said user interface indicating which of said represented plurality of lighting devices is physical and which of said represented plurality of lighting devices is virtual, a visual representation of said virtual lighting device being displayed in a mixed reality environment on a second user device, receive, via said user interface, one or more user inputs indicative of a first light setting for said physical lighting device and indicative of a second light setting for said virtual lighting device, control, via said at least one transmitter, said physical lighting device according to said first light setting, and cause, via said at least one transmitter, said second user device to represent a light effect rendered by said virtual lighting device in said mixed reality environment, said light effect corresponding to said second light setting.

[0014] This system may be used to allow users to control virtual lighting devices with the same familiar light control user interface that the users can use to control physical lighting devices. The virtual lighting devices are clearly identified as being virtual in the user interface. Changes in the light settings of the virtual lighting devices due to the user’s interaction with the user interface are reflected in the mixed reality environment. The same user input may be indicative of both the first light setting for the physical lighting device and the second light setting for the virtual lighting device. The first light setting may be the same as the second light setting. Receiving the one or more user inputs may comprise receiving a first user input indicative of the first light setting and a second user input indicative of the second light setting. Said system may comprise said first user device and / or said second user device. Said at least one processor may be configured to provide said user interface to a different user than a user to which said mixed reality environment is provided on said second user device. Said second user device may be a mixed reality headset, a pair of mixed reality glasses, a mobile phone, or a tablet, for example. Said second user device may be used by the same user that uses the first user device or by a different user. Said at least one processor may be configured to provide said user interface via a non-augmented reality software application running on said first user device. Said at least one processor may be configured to provide said mixed reality environment also on said first user device. Said first user device may be a mobile phone or a tablet, for example.

[0015] Said at least one processor may be further configured to provide, on a third user device, a second user interface representing said plurality of lighting devices, said second user interface indicating which of said represented plurality of lighting devices is physical and which of said represented plurality of lighting devices is virtual, said second user interface being provided to a different user than said user interface, receive, via said second user interface, third user input indicative of a third light setting for said physical lighting device, control, via said at least one transmitter, said physical lighting device according to said third light setting, receive, via said second user interface, fourth user input indicative of a fourth light setting for said virtual lighting device, and cause, via said at least one transmitter, said second user device to represent a further light effect rendered by said virtual lighting device in said mixed reality environment, said further light effect corresponding to said fourth light setting.

[0016] For example, lighting control apps running on the first user device and the third user device may be synced such that the user of the third user device can see the same physical and virtual lighting devices as the user of the first user device.

[0017] Said at least one processor may be configured to receive a signal indicating that said virtual lighting device has been added to said mixed reality environment or to a lighting system comprising said physical lighting device. For example, a user may be able to add the virtual lighting device in the mixed reality environment on the second user device and the second user device or a mixed reality server may then inform a light controller or the first user device directly that the virtual lighting device has been added. After receipt of the signal, the virtual lighting device is represented in the user interface. This representation may be conditional, e.g. only if the first user device and the second user device are in the same room. Said at least one processor may be configured to, upon receiving said signal, add said virtual lighting device to a group of lighting devices represented in said user interface and / or associate said virtual lighting device with a light scene selectable in said user interface, said group of lighting devices comprising said physical lighting device. If the physical lighting device and the virtual lighting devices are in the same group or associated with the same light scene, the user may be able change both of their light settings with one user input. The group may correspond to a room or other area, for example. Also, a group of lighting devices may be associated with an automation / routine, for example.

[0018] Said at least one processor may be configured to obtain information indicative of a location of said second user device, determine a room based on said location, and upon receiving said signal, add said virtual lighting device to said group by adding said virtual lighting device to a group corresponding to said room. For example, when the user of the second user device adds a virtual lighting device when the user is in the living room, the virtual lighting device may be added to the living room group. In a more advanced implementation, the room is determined based on the location and a field of view. For example, when the user of the second user device adds a virtual lighting device when the user is looking at the kitchen from the living room, the virtual lighting device may be added to the kitchen group.

[0019] Said at least one processor may be configured to control said physical lighting device according to said first light setting by transmitting a lighting command to a light controller and cause said second user device to represent said light effect by transmitting a lighting command to said light controller. Alternatively, said at least one processor may be configured to control said physical lighting device according to said first light setting by transmitting a lighting command to a light controller and cause said second user device to represent said light effect by transmitting a lighting command or another type of message to said second user device or to a mixed reality server.

[0020] Said visual representation of said virtual lighting device may be further displayed in said mixed reality environment on a further user device. For example, if a user uses both the first user device and the second user device and wants to buy a new luminaire, this user may add a virtual lighting device (representing the new luminaire) on the second user device, control the virtual lighting device via the first user device, and thereby cause both users to see the results of controlling the virtual lighting device (on the second user device and the further user device, respectively). Said further user device may be a mixed reality headset, a pair of mixed reality glasses, a mobile phone, or a tablet, for example. In a second aspect of the invention, a method of receiving user input indicative of a light setting and controlling one or more lighting devices according to said light setting comprises providing, on a first user device, a user interface representing a plurality of lighting devices, said plurality of lighting devices comprising a virtual lighting device and a physical lighting device, said user interface indicating which of said represented plurality of lighting devices is physical and which of said represented plurality of lighting devices is virtual, a visual representation of said virtual lighting device being displayed in a mixed reality environment on a second user device, receiving, via said user interface, one or more user inputs indicative of a first light setting for said physical lighting device and indicative of a second light setting for said virtual lighting device, controlling said physical lighting device according to said first light setting, and causing said second user device to represent a light effect rendered by said virtual lighting device in said mixed reality environment, said light effect corresponding to said second light setting. Said method may be performed by software running on a programmable device. This software may be provided as a computer program product.

[0021] Moreover, a computer program for carrying out the methods described herein, as well as a non-transitory computer readable storage-medium storing the computer program are provided. A computer program may, for example, be downloaded by or uploaded to an existing device or be stored upon manufacturing of these systems.

[0022] A non-transitory computer-readable storage medium stores at least a first software code portion, the first software code portion, when executed or processed by a computer, being configured to perform executable operations for receiving user input indicative of a light setting and controlling one or more lighting devices according to said light setting.

[0023] The executable operations comprise providing, on a first user device, a user interface representing a plurality of lighting devices, said plurality of lighting devices comprising a virtual lighting device and a physical lighting device, said user interface indicating which of said represented plurality of lighting devices is physical and which of said represented plurality of lighting devices is virtual, a visual representation of said virtual lighting device being displayed in a mixed reality environment on a second user device, receiving, via said user interface, one or more user inputs indicative of a first light setting for said physical lighting device and indicative of a second light setting for said virtual lighting device, controlling said physical lighting device according to said first light setting, and causing said second user device to represent a light effect rendered by said virtual lighting device in said mixed reality environment, said light effect corresponding to said second light setting.

[0024] As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a device, a method or a computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit", "module" or "system." Functions described in this disclosure may be implemented as an algorithm executed by a processor / microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.

[0025] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer readable storage medium may include, but are not limited to, the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the present invention, a computer readable storage medium may be any tangible medium that can contain, or store, a program for use by or in connection with an instruction execution system, apparatus, or device.

[0026] A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java(TM), Swift, Dart, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0027] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or a central processing unit (CPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0028] These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0029] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0030] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0031] BRIEF DESCRIPTION OF THE DRAWINGS

[0032] These and other aspects of the invention are apparent from and will be further elucidated, by way of example, with reference to the drawings, in which:

[0033] Fig. 1 is a block diagram of an embodiment of the system;

[0034] Fig. 2 shows an example of a visual representation of a virtual lighting device in a mixed reality environment;

[0035] Fig. 3 shows an example of a user interface screen representing a virtual lighting device and a physical lighting device;

[0036] Fig. 4 is a flow chart of a first embodiment of the method;

[0037] Fig. 5 is a flow chart of a second embodiment of the method;

[0038] Fig. 6 is a flow chart of a third embodiment of the method;

[0039] Fig. 7 shows a first example of a user interface screen which may be provided in the method of Fig. 6;

[0040] Fig. 8 shows a second example of a user interface screen which may be provided in the method of Fig. 6; and

[0041] Fig. 9 is a block diagram of an exemplary data processing system for performing the method of the invention. Corresponding elements in the drawings are denoted by the same reference numeral.

[0042] DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] Fig. 1 shows an embodiment of the system for receiving user input indicative of a light setting and controlling one or more lighting devices according to the light setting. In this embodiment, the system comprises a mobile device 1. The mobile device 1 may be a mobile phone or a tablet, for example. In the embodiment of Fig. 1, the mobile device 1 is able to control physical lighting devices 31-32 via a (light) bridge 11, e.g. using Zigbee technology. The bridge 11 may be a Hue bridge, for example. The bridge 11 and the physical lighting devices 31-32 are part of a lighting system 30.

[0044] The bridge 11 is connected to a wireless LAN access point 13, e.g. via Ethernet or Wi-Fi. In an alternative embodiment, the mobile device 1 can alternatively or additionally control one or more of the physical lighting devices 31-32 without a bridge, e.g. directly via Bluetooth or via an Internet server 23. In the example of Fig. 1, a further mobile device 15 is also able to control the physical lighting devices 31-32. In the example of Fig. 1, physical lighting device 31 is a light bulb and physical lighting device 32 is a table lamp.

[0045] The mobile device 1 comprises a receiver 3, a transmitter 4, a processor 5, memory 7, a camera 6, and a display 9. The processor 5 is configured to provide, on a first user device, e.g. the mobile device 1, a user interface representing a plurality of lighting devices. The plurality of lighting devices comprises a virtual lighting device and the physical lighting devices 31 and 32. The user interface indicates which of the represented plurality of lighting devices is physical and which of the represented plurality of lighting devices is virtual. The user interface is provided via a non-augmented reality software application running on the first user device, e.g. mobile device 1.

[0046] A visual representation of the virtual lighting device is displayed in a mixed reality environment on a second user device 17. A user to which the mixed reality environment is provided on the second user device 17 may be the same as, or different from, the user to which the user interface is provided on the first user device. In the example of Fig. 1, the second user device 17 is a mixed reality headset. The second user device 17 may also be a more conventional device like a mobile phone or a tablet. The processor 5 may be configured to provide the mixed reality environment also on the first user device, e.g. mobile device 1. The mixed reality environment may be used, for example, to allow a user to preview lighting devices before purchasing, to provide help with installation of lighting devices (e.g., for a rail system) or to allow a user to see the impact of repositioning lighting devices that they might already have. The second user device 17 may be able to provide the mixed reality environment with or without the use of data stored on an optional mixed reality server 25. The mixed reality server 25 may be used to synchronize the mixed reality environment between multiple user devices, for example. The visual representation of the virtual lighting device may be further displayed in the mixed reality environment on a further user device (not shown in Fig. 1).

[0047] The processor 5 is further configured to receive, via the user interface, one or more user inputs indicative of a first light setting for a physical lighting device and indicative of a second light setting for the virtual lighting device, control, via the transmitter 4, the physical lighting device according to the first light setting, and cause, via the transmitter 4, the second user device 17 to represent a light effect rendered by the virtual lighting device 71 in the mixed reality environment. This light effect corresponds to the second light setting. The first and second user inputs may be received via display 9, for example, if display 9 is a touchscreen display.

[0048] The same user input may be indicative of both the first light setting for the physical lighting device and the second light setting for the virtual lighting device. The first light setting may be the same as the second light setting. The processor 5 may be configured to receive, via the user interface, a first input indicative of the first light setting and receive, via the user interface, a second input indicative of the second light setting.

[0049] The first user device typically comprises a screen, but does not need to comprise a screen, e.g. the user interface may comprise a voice interface instead. The voice interface might indicate, for example, that there are virtual lights and ask user to confirm the command, e.g., user: “Hey, Google, put my light in a living room to a Savanah Sunset scene”; Google: “Living room currently contains virtual light, should the light setting be applied to them as well?”; user: “Yes”. Instead of a mobile device, the first user device may be a smart home device like Amazon Echo or Google Home, for example. The first user device may also combine a screen and a voice interface.

[0050] Fig. 2 shows an example of a visual representation of a virtual lighting device in a mixed reality environment. Fig. 2 shows a captured image depicting a room with a television 63 placed on a TV bench 61 and a cabinet 65 next to the television 63. These devices are all physical. A visual representation of a virtual lighting device 71 and a visual representation of a virtual light effect 73 rendered by the virtual lighting device 71 are displayed superimposed over the captured image. The virtual lighting device is, like its representation, also referred to with the reference numeral 71. The virtual lighting device 71 looks like it has been placed on top of cabinet 61. The user may have selected the desired type of lighting device and the desired location.

[0051] Fig. 3 shows an example of a user interface screen representing a virtual lighting device and a physical lighting device. A user interface screen 41 is displayed on display 9 of mobile device 1, i.e. the first user device. The user interface screen 41 shows visual representations 43-45 of three light scenes. Each of these light scenes specifies different light settings. The user interface screen 41 further shows visual representations 46- 48 of three lighting devices. Visual representations 46 and 47 represent physical lighting devices 31 and 32 of Fig. 1, respectively. Visual representation 48 represents virtual lighting device 71 of Fig. 2. In the example of Fig. 3, a visual representation comprises the word “virtual” if it represents a virtual lighting device. A visual representation that does not comprise the word “virtual” represents a physical lighting device.

[0052] Each of the three light scenes is associated with one or more of the three lighting devices. A light scene may be associated with both one or more physical lighting devices and one or more virtual lighting devices. For example, the light scene represented by visual representation 43 may be associated with the physical lighting device represented by visual representation 46 and the virtual lighting device represented by visual representation 48. Recalling this light scene then affects both this physical lighting device and this virtual lighting device.

[0053] The system may further comprise a mobile device 15, see Fig. 1. The processor of mobile device 15 may be configured to provide, on a third user device, e.g. the mobile device 15, a second user interface representing the plurality of lighting devices, which indicates which of the represented plurality of lighting devices is physical and which of the represented plurality of lighting devices is virtual. The second user interface provided on the third user device is provided to a different user than the user interface provided on the first user device, but both user interfaces may be the same, e.g. comprising the user interface screen 41 of Fig. 3.

[0054] The processor of the mobile device 15 may be further configured to receive, via the second user interface, third user input indicative of a third light setting for the physical lighting device, control, via a transmitter of the mobile device 15, the physical lighting device according to the third light setting, receive, via the second user interface, fourth user input indicative of a fourth light setting for the virtual lighting device, and cause, via the transmitter of the mobile device 15, the second user device to represent a further light effect rendered by the virtual lighting device in the mixed reality environment. This further light effect corresponding to the fourth light setting.

[0055] The same user input may be indicative of both the third light setting for the physical lighting device and the fourth light setting for the virtual lighting device. The third light setting may be the same as the fourth light setting. In the embodiment of Fig. 1, the mobile device 15 is able to control physical lighting devices 31-32 via a (light) bridge 11, e.g. using Zigbee technology.

[0056] The third user device is a device that can be used for light control, e.g. via an app running on the third user device. The system may further comprise a further user device which provides a user interface that does not indicate which of the represented plurality of lighting devices is physical and which of the represented plurality of lighting devices is virtual. This further user device may be used by the user of the first user device or by another user. This further user device may comprise a voice interface instead of a screen, for example.

[0057] In a first example scenario, one person may be exploring how a new luminaire might look in the living room using the first user device and the second user device, while another person gives a command to the further user device, e.g. a Google Home, to activate a light scene. In a second example scenario, a person may be exploring how a new luminaire might look in the living room using the first user device and the second user device, while also inputting commands on the further user device via a voice interface to control one or more lighting devices.

[0058] If the user of the further user device gives a command to control a group of one or more lighting devices, the group comprises a virtual lighting device, and the user is not provided via another user device with a user interface in which virtual lighting devices are represented and in which there is a clear distinction between physical and virtual lighting devices, preferably only the physical lighting devices of the group are controlled.

[0059] In an implementation, virtual lighting devices are dynamically added to and removed from the user interface, e.g. of a lighting control app running on the first user device, depending on whether the second user device is located in the same area as the first user device and optionally, if the second user device is a mobile phone or tablet, depending on whether the second device is running a corresponding mixed reality application and / or is being held vertically, or if the second user device is a mixed reality headset or a pair of mixed reality glasses, whether the user is wearing the headset / glasses.

[0060] In this implementation, location, type, and number of virtual lighting devices in the user interface are dynamically updated based on user interaction in the mixed reality environment. Changes in the virtual lighting state due to interaction with the user interface are reflected in the mixed reality environment. The user interface may be used to control both physical and virtual lighting devices as a part of the same group (e.g. activating a scene, triggering automations, reacting on sensor trigger, etc.).

[0061] When a virtual lighting device is added in the mixed reality environment, its type and location (e.g. room) are sent to the mobile device 1 or to the bridge 11. When the virtual lighting device is added in the mixed reality environment, the virtual lighting device may be added to a group configured in the lighting system, e.g. the group corresponding to the room the second user device 17 is in. The user of the mobile device 1 need not be asked to perform the usual commissioning steps to add the virtual lighting device; the mobile device 1 may simply inform the user that the virtual lighting device has been added. The user may then be able to move the virtual lighting device from one group to another if it was incorrectly assigned. The virtual lighting device may get a color assignment for all light scenes that are associated with the group assigned to the virtual lighting device, as if the virtual lighting device were a physical lighting device, without updating any of the settings of the physical lighting devices.

[0062] The virtual plus physical light configuration may be saved on the mobile device 15, on the bridge 11, or the Internet server 23, for example. The next time the mixed reality environment is used, e.g. the second user device 17 is used, the user interface could then already show the virtual lighting device(s). Different virtual plus physical light configurations may be saved and the user of second user device 17 may be able to choose one of these configurations, e.g. depending on which mixed reality environment is loaded, which is then also communicated to the mobile device 1.

[0063] In the embodiment of Fig. 1, the processor 5 is configured to control the physical lighting device 31 or 32 according to the first light setting by transmitting a lighting command to the light controller 11. The processor 5 may be configured to cause the second user device 17 to represent the light effect rendered by the virtual lighting device in one of the following manners, for example: a. transmitting a lighting command to the bridge 11 ; b. transmitting a lighting command or another type of message to the second user device 17; c. transmitting a lighting command or another type of message to the mixed reality server 25.

[0064] In the case of option a), i.e. if virtual lighting devices are added via the bridge 11, then bridge 11 could send standard Zigbee commands to the physical lighting devices 31 and 32 and another type of message to the second user device 17, e.g. via Wi-Fi. In the case of options b) and c), the bridge 11 may not even be aware of the presence of a virtual lighting device and only receive lighting commands for the physical lighting devices 31 and 32. In these cases, the mobile device 1 transmits a lighting command or another type of message to the second user device 17 or to the mixed reality server 25.

[0065] If a virtual bridge is implemented on the second user device on which the mixed reality environment is provided, e.g. on second user device 17, the mobile device 1 can communicate with the second user device in the same way it communicates with bridge 11. A simple identifier may signal to the mobile device 1 that this bridge and its lighting devices are virtual. In an alternative embodiment, this virtual bridge may be the only bridge that the mobile device 1 communicates with and the virtual bridge then proxies commands for physical lighting devices 31 and 32 to physical bridge 11.

[0066] In the embodiment of the mobile device 1 shown in Fig. 1, the mobile device 1 comprises one processor 5. In an alternative embodiment, the mobile device 1 comprises multiple processors. The processor 5 of the mobile device 1 may be a general-purpose processor, e.g. from ARM or Qualcomm or an application-specific processor. The processor 5 of the mobile device 1 may run an Android or iOS operating system for example. The display 9 may comprise an LCD or OLED display panel, for example. The memory 7 may comprise one or more memory units. The memory 7 may comprise solid state memory, for example.

[0067] The receiver 3 and the transmitter 4 may use one or more wireless communication technologies such as Wi-Fi (IEEE 802.11) to communicate with the wireless LAN access point 13, for example. In an alternative embodiment, multiple receivers and / or multiple transmitters are used instead of a single receiver and a single transmitter. In the embodiment shown in Fig. 1, a separate receiver and a separate transmitter are used. In an alternative embodiment, the receiver 3 and the transmitter 4 are combined into a transceiver. Camera 6 may comprise a CMOS or CCD sensor, for example. The mobile device 1 may comprise other components typical for a mobile device such as a battery and a power connector. The invention may be implemented using a computer program running on one or more processors.

[0068] In the embodiment of Fig. 1, the system of the invention comprises one or two mobile devices and optionally a mixed reality headset. In an alternative embodiment, the system of the invention alternatively or additionally comprises a different device, e.g. a bridge or a cloud computer (cluster).

[0069] A first embodiment of the method of receiving user input indicative of a light setting and controlling one or more lighting devices according to the light setting is shown in Fig. 4. The method may be performed by the mobile device 1 of Fig. 1, for example.

[0070] A step 101 comprises providing, on a first user device, a user interface representing a plurality of lighting devices. The plurality of lighting devices comprises a virtual lighting device and a physical lighting device. The user interface indicates which of the represented plurality of lighting devices is physical and which of the represented plurality of lighting devices is virtual. A visual representation of the virtual lighting device is displayed in a mixed reality environment on a second user device. The second user device may be used by the same user that uses the first user device or by a different user.

[0071] A step 103 or a step 107 is performed after step 101. Step 103 comprises receiving, via the user interface provided in step 101, first user input indicative of a first light setting for the physical lighting device. A step 105 comprises controlling the physical lighting device according to the first light setting indicated in the first user input received in step 103. Step 101 is repeated after step 105, and the method then proceeds as shown in Fig. 4. When step 101 is repeated, an updated user interface may be provided to reflect that the physical lighting device is being controlled according to the first light setting, for example. Alternatively, no new information is output when step 101 is repeated, for example.

[0072] Step 107 comprises receiving, via the user interface provided in step 101, second user input indicative of a second light setting for the virtual lighting device. A step 109 comprises causing the second user device to represent a light effect rendered by the virtual lighting device in the mixed reality environment. The light effect corresponds to the second light setting indicated in the second user input received in step 107. Step 101 is repeated after step 109, and the method then proceeds as shown in Fig. 4. Additionally, one or more steps of one or more of the embodiments of Figs. 5-6 may be added to the embodiment of Fig. 4.

[0073] The same user input may be indicative of both the first light setting for the physical lighting device and the second light setting for the virtual lighting device. For example, by receiving a single user input indicative of both the first light setting and the second light setting, both steps 103 and 107 are performed. The first light setting may be the same as the second light setting.

[0074] A second embodiment of the method of receiving user input indicative of a light setting and controlling one or more lighting devices according to the light setting is shown in Fig. 5. The embodiment of Fig. 5 is an extension of the embodiment of Fig. 4.

[0075] Step 121 comprises providing, on a first user device, a user interface representing at least a physical lighting device. Step 103 or a step 123 is performed after step 121. Step 103 comprises receiving, via the user interface provided in step 121, first user input indicative of a first light setting for the physical lighting device. Step 105 comprises controlling the physical lighting device according to the first light setting indicated in the first user input received in step 103. Step 121 is repeated after step 105, and the method then proceeds as shown in Fig. 5.

[0076] Step 123 comprises receiving a signal indicating that a virtual lighting device has been added to a mixed reality environment or to a lighting system comprising the physical lighting device. Next, step 101 is performed. Step 101 comprises providing, on the first user device, a user interface representing a plurality of lighting devices. Step 101 may comprise updating the user interface provided earlier in step 121. The plurality of lighting devices comprises the virtual lighting device and the physical lighting device. The user interface indicates which of the represented plurality of lighting devices is physical and which of the represented plurality of lighting devices is virtual. A visual representation of the virtual lighting device is displayed in the mixed reality environment on a second user device. The addition of the virtual lighting device in the mixed reality environment by the user of the second user device may have caused the signal received in step 123 to be transmitted.

[0077] Step 103 or step 107 is performed after step 101. Step 103 comprises receiving, via the user interface provided in step 101, first user input indicative of a first light setting for the physical lighting device. A step 105 comprises controlling the physical lighting device according to the first light setting indicated in the first user input received in step 103. Step 101 is repeated after step 105, and the method then proceeds as shown in Fig. 5.

[0078] Step 107 comprises receiving, via the user interface provided in step 101, second user input indicative of a second light setting for the virtual lighting device. A step 109 comprises causing the second user device to represent a light effect rendered by the virtual lighting device in the mixed reality environment. The light effect corresponds to the second light setting indicated in the second user input received in step 107. Step 101 is repeated after step 109, and the method then proceeds as shown in Fig. 5. Additionally, one or more steps of the embodiment of Fig. 6 may be added to the embodiment of Fig. 5.

[0079] A third embodiment of the method of receiving user input indicative of a light setting and controlling one or more lighting devices according to the light setting is shown in Fig. 6. The embodiment of Fig. 6 is an extension of the embodiment of Fig. 5. In the embodiment of Fig. 6, step 131 and / or step 133 is / are performed between steps 123 and 101 of Fig. 5.

[0080] Step 131 comprises associating the virtual lighting device with a light scene selectable in the user interface. Step 133 comprises adding the virtual lighting device to a group of lighting devices represented in the user interface. This group of lighting devices comprises the physical lighting device.

[0081] Step 133 may comprise sub steps 135, 137, and 139. Step 135 comprises obtaining information indicative of a location of the second user device. This information may be obtained by determining a value representing the distance between the first user device and the second user device (e.g. received signal strength) and / or based on RF beacons installed in different rooms and associated with these rooms. Step 137 comprises determining a room based on the location indicated by the information obtained in step 135. Step 139 comprises adding the virtual lighting device to a group corresponding to the room determined in step 137.

[0082] For example, if a user previews how a new Hue Signe will look in the living room via the mixed reality environment provided on the second user device, a virtual Hue Signe may be added to a living room light group. In an advanced implementation, the room is determined based on the location and a field of view. For example, when the user of the second user device adds a virtual lighting device when the user is looking at the kitchen from the living room, the virtual lighting device may be added to the kitchen group. After the virtual lighting device has been added to a group, the user may be able to control its state using the user interface provided on the first device as an individual lighting device or as part of the group (e.g., scene recall).

[0083] Fig. 7 shows a first example of a user interface screen which may be provided in step 101 of Fig. 6. In the example of Fig. 7, light scene 44 has been selected and only the lighting devices associated with light scene 44 are represented in user interface screen 81: physical lighting device 32 corresponding to visual representation 47 and virtual lighting device 71 corresponding to visual representation 48. Virtual lighting device 71 was associated with light scene 44 in step 131 of Fig. 6. Fig. 8 shows a second example of a user interface screen which may be provided in step 101 of Fig. 6. In the example of Fig. 8, the second user device was located in the living room when the user added the virtual lighting device 71 to the mixed reality environment. The virtual lighting device 71 was therefore added to the living room group in step 133 of Fig. 6. As shown in user interface screen 91 of Fig. 8, the living room group comprises the physical lighting device 31 represented by the visual representation 46 in addition to the virtual lighting device 71 represented by the visual representation 48.

[0084] Fig. 9 depicts a block diagram illustrating an exemplary data processing system that may perform the method as described with reference to Figs. 4-6.

[0085] As shown in Fig. 9, the data processing system 300 may include at least one processor 302 coupled to memory elements 304 through a system bus 306. As such, the data processing system may store program code within memory elements 304. Further, the processor 302 may execute the program code accessed from the memory elements 304 via a system bus 306. In one aspect, the data processing system may be implemented as a computer that is suitable for storing and / or executing program code. It should be appreciated, however, that the data processing system 300 may be implemented in the form of any system including a processor and a memory that is capable of performing the functions described within this specification.

[0086] The memory elements 304 may include one or more physical memory devices such as, for example, local memory 308 and one or more bulk storage devices 310. The local memory may refer to random access memory or other non-persistent memory device(s) generally used during actual execution of the program code. A bulk storage device may be implemented as a hard drive or other persistent data storage device. The processing system 300 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the quantity of times program code must be retrieved from the bulk storage device 310 during execution. The processing system 300 may also be able to use memory elements of another processing system, e.g. if the processing system 300 is part of a cloud-computing platform.

[0087] Input / output (I / O) devices depicted as an input device 312 and an output device 314 optionally can be coupled to the data processing system. Examples of input devices may include, but are not limited to, a keyboard, a pointing device such as a mouse, a microphone (e.g. for voice and / or speech recognition), or the like. Examples of output devices may include, but are not limited to, a monitor or a display, speakers, or the like. Input and / or output devices may be coupled to the data processing system either directly or through intervening I / O controllers.

[0088] In an embodiment, the input and the output devices may be implemented as a combined input / output device (illustrated in Fig. 9 with a dashed line surrounding the input device 312 and the output device 314). An example of such a combined device is a touch sensitive display, also sometimes referred to as a “touch screen display” or simply “touch screen”. In such an embodiment, input to the device may be provided by a movement of a physical object, such as e.g. a stylus or a finger of a user, on or near the touch screen display.

[0089] A network adapter 316 may also be coupled to the data processing system to enable it to become coupled to other systems, computer systems, remote network devices, and / or remote storage devices through intervening private or public networks. The network adapter may comprise a data receiver for receiving data that is transmitted by said systems, devices and / or networks to the data processing system 300, and a data transmitter for transmitting data from the data processing system 300 to said systems, devices and / or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapter that may be used with the data processing system 300.

[0090] As pictured in Fig. 9, the memory elements 304 may store an application 318. In various embodiments, the application 318 may be stored in the local memory 308, the one or more bulk storage devices 310, or separate from the local memory and the bulk storage devices. It should be appreciated that the data processing system 300 may further execute an operating system (not shown in Fig. 9) that can facilitate execution of the application 318. The application 318, being implemented in the form of executable program code, can be executed by the data processing system 300, e.g., by the processor 302. Responsive to executing the application, the data processing system 300 may be configured to perform one or more operations or method steps described herein.

[0091] Fig. 9 shows the input device 312 and the output device 314 as being separate from the network adapter 316. However, additionally or alternatively, input may be received via the network adapter 316 and output be transmitted via the network adapter 316. For example, the data processing system 300 may be a cloud server. In this case, the input may be received from and the output may be transmitted to a user device that acts as a terminal.

[0092] Various embodiments of the invention may be implemented as a program product for use with a computer system, where the program(s) of the program product define functions of the embodiments (including the methods described herein). In one embodiment, the program(s) can be contained on a variety of non-transitory computer-readable storage media, where, as used herein, the expression “non-transitory computer readable storage media” comprises all computer-readable media, with the sole exception being a transitory, propagating signal. In another embodiment, the program(s) can be contained on a variety of transitory computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, ROM chips or any type of solid-state non-volatile semiconductor memory) on which information is permanently stored; and (ii) writable storage media (e.g., flash memory, floppy disks within a diskette drive or hard-disk drive or any type of solid-state random-access semiconductor memory) on which alterable information is stored. The computer program may be run on the processor 302 described herein.

[0093] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0094] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of embodiments of the present invention has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the implementations in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the present invention. The embodiments were chosen and described in order to best explain the principles and some practical applications of the present invention, and to enable others of ordinary skill in the art to understand the present invention for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

CLAIMS:

1. A system (1) for receiving user input indicative of a light setting and controlling one or more lighting devices according to said light setting, said system (1) comprising: at least one transmitter (4); and at least one processor (5) configured to:- provide, on a first user device, a user interface representing a plurality of lighting devices, said plurality of lighting devices (31,32,71) comprising a virtual lighting device (71) and a physical lighting device (31,32), said user interface indicating which of said represented plurality of lighting devices (31,32,71) is physical and which of said represented plurality of lighting devices (31,32,71) is virtual, a visual representation of said virtual lighting device (71) being displayed in a mixed reality environment on a second user device (17),- receive, via said user interface, one or more user inputs indicative of a first light setting for said physical lighting device (31,32) and indicative of a second light setting for said virtual lighting device (71) ,- control, via said at least one transmitter (4), said physical lighting device (31,32) according to said first light setting, and- cause, via said at least one transmitter (4), said second user device (17) to represent a light effect rendered by said virtual lighting device (71) in said mixed reality environment, said light effect corresponding to said second light setting.

2. A system (1) as claimed in claim 1, wherein said at least one processor (5) is further configured to:- provide, on a third user device (15), a second user interface representing said plurality of lighting devices (31,32,71), said second user interface indicating which of said represented plurality of lighting devices (31,32,71) is physical and which of said represented plurality of lighting devices (31,32,71) is virtual, said second user interface being provided to a different user than said user interface,- receive, via said second user interface, third user input indicative of a third light setting for said physical lighting device (31,32),- control, via said at least one transmitter (4), said physical lighting device (31,32) according to said third light setting,- receive, via said second user interface, fourth user input indicative of a fourth light setting for said virtual lighting device (71), and- cause, via said at least one transmitter (4), said second user device (17) to represent a further light effect rendered by said virtual lighting device (71) in said mixed reality environment, said further light effect corresponding to said fourth light setting.

3. A system (1) as claimed in claim 1 or 2, wherein said system comprises said first user device and / or said second user device (17).

4. A system (1) as claimed in any one of the preceding claims, wherein said at least one processor (5) is configured to provide said user interface to a different user than a user to which said mixed reality environment is provided on said second user device (17).

5. A system (1) as claimed in any one of the preceding claims, wherein said at least one processor (5) is configured to provide said user interface via a non-augmented reality software application running on said first user device.

6. A system (1) as claimed in any one of the preceding claims, wherein said at least one processor (5) is configured to provide said mixed reality environment also on said first user device.

7. A system (1) as claimed in any one of the preceding claims, wherein said at least one processor (5) is configured to receive a signal indicating that said virtual lighting device (71) has been added to said mixed reality environment or to a lighting system (30) comprising said physical lighting device (31,32).

8. A system (1) as claimed in claim 7, wherein said at least one processor (5) is configured to, upon receiving said signal, add said virtual lighting device (71) to a group of lighting devices represented in said user interface and / or associate said virtual lighting device(71) with a light scene selectable in said user interface, said group of lighting devices comprising said physical lighting device (31,32).

9. A system (1) as claimed in claim 8, wherein said at least one processor (5) is configured to:- obtain information indicative of a location of said second user device (17),- determine a room based on said location, and- upon receiving said signal, add said virtual lighting device (71) to said group by adding said virtual lighting device (71) to a group corresponding to said room.

10. A system (1) as claimed in any one of claims 1 to 9, wherein said at least one processor (5) is configured to control said physical lighting device (31,32) according to said first light setting by transmitting a lighting command to a light controller (11) and cause said second user device (17) to represent said light effect by transmitting a lighting command to said light controller (11).

11. A system (1) as claimed in any one of claims 1 to 9, wherein said at least one processor (5) is configured to control said physical lighting device (31,32) according to said first light setting by transmitting a lighting command to a light controller (11) and cause said second user device (17) to represent said light effect by transmitting a lighting command or another type of message to said second user device (17) or to a mixed reality server (25).

12. A system (1) as claimed in any one of the preceding claims, wherein said visual representation of said virtual lighting device is further displayed in said mixed reality environment on a fourth user device.

13. A method of receiving user input indicative of a light setting and controlling one or more lighting devices according to said light setting, said method comprising:- providing (101), on a first user device, a user interface representing a plurality of lighting devices, said plurality of lighting devices comprising a virtual lighting device and a physical lighting device, said user interface indicating which of said represented plurality of lighting devices is physical and which of said represented plurality of lighting devices is virtual, a visual representation of said virtual lighting device being displayed in a mixed reality environment on a second user device;- receiving (103), via said user interface, one or more user inputs indicative of a first light setting for said physical lighting device and indicative of a second light setting for said virtual lighting device (71);- controlling (105) said physical lighting device according to said first light setting; and- causing (109) said second user device to represent a light effect rendered by said virtual lighting device in said mixed reality environment, said light effect corresponding to said second light setting.

14. A computer program product for a computing device, the computer program product comprising computer program code to perform the method of claim 13 when the computer program product is run on a processing unit of the computing device.

Images