Displaying a representation of a virtual light effect

Abstract

An electronic device (1) for displaying a virtual representation (52) of a virtual light effect rendered by a virtual lighting device (51) is configured to obtain images captured by a camera, detect that a physical lighting device (31) is present in the images, determine the virtual representation of the virtual light effect rendered by the virtual lighting device, and display the images and, overlaid on the images, the virtual representation of the virtual light effect to a user of the electronic device. The physical lighting device renders a physical light effect (36). The electronic device is configured to determine the virtual representation of the virtual light effect based on a representation (42) of the physical light effect in the images and / or adjust a light setting of the physical lighting device based on the representation of the physical light effect in the images and / or the virtual light effect.

Description

[0001] DISPLAYING A REPRESENTATION OF A VIRTUAL LIGHT EFFECT

[0002] FIELD OF THE INVENTION

[0003] The invention relates to an electronic device for displaying a virtual representation of a virtual light effect rendered by a virtual lighting device.

[0004] The invention further relates to a method of displaying a virtual representation of a virtual light effect rendered by a virtual lighting device.

[0005] The invention also relates to a computer program product enabling a computer system to perform such a method.

[0006] BACKGROUND OF THE INVENTION

[0007] The number of Augmented Reality (AR) based applications is growing fast as smart phones and AR devices become more and more capable in merging physical world (camera stream) and virtual objects. In more and more AR-based applications, virtual light effects are represented. In certain AR-based applications, simply adding a virtual lighting device to a room is not enough; it is important for the user to see its light effect and overall impact on the room atmosphere.

[0008] US 2020 / 0410725 Al discloses a system for evaluation of an AR experience. Visitors of a theme park or amusement park wear an head mounted display to see virtual features / objects. This makes it possible to enhance guest experiences by providing adjustable virtual environments for different experiences in the same amusement park ride. To allow a developer to create virtual environments without having to physically visit each location of the amusement part, the head mounted display is configured to overlay a virtual object in a virtual space onto a backdrop, thereby enabling the developer to use the head mounted display to perceive the virtual object as positioned within a physical space.

[0009] The virtual space further comprises virtual lights which correspond to physical lights in the physical space. The system can adjust a hue and an intensity of virtual light rendered by a virtual light in the virtual space to match the hue and the intensity of light output by the corresponding physical light in the physical space. This allows developers to evaluate whether an AR system configured to generate the AR features and to overlay the AR features onto a real-world environment effectively adjusts an appearance of the AR features based on changes in various lighting conditions of the real-world environment.

[0010] US 2020 / 0187334 Al discloses systems and methods for generating a lighting design for a venue.

[0011] In the evaluation system of US 2020 / 0410725 Al, the virtual lights in the virtual space correspond to the physical lights in the physical space. However, this is not sufficient if the user wants to preview luminaires before purchasing, wants to get help with installation (e.g., for a rail system), or is considering repositioning lights that the user already has. In such AR applications, the user should be able to see in AR a virtual representation of a virtual light effect rendered by a virtual lighting device which does not correspond to a physical lighting device in the user’s surroundings (e.g. because the virtual lighting device is located at a different position than a corresponding physical lighting device in the user’s surroundings) and a representation of a physical light effected rendered by a physical lighting device.

[0012] SUMMARY OF THE INVENTION

[0013] It is advantageous to provide an electronic device, which allows a user to see in AR a virtual representation of a virtual light effect rendered by a virtual lighting device which does not correspond to a physical lighting device in the user’s surroundings and a representation of a physical light effected rendered by a physical lighting device, without creating an inconsistent viewing experience.

[0014] It is advantageous to provide a method, which allows a user to see in AR a virtual representation of a virtual light effect rendered by a virtual lighting device which does not correspond to a physical lighting device in the user’s surroundings and a representation of a physical light effected rendered by a physical lighting device, without creating an inconsistent viewing experience.

[0015] In a first aspect, an electronic device for displaying a virtual representation of a virtual light effect rendered by a virtual lighting device comprises at least one camera, at least one display, and at least one processor configured to obtain images captured by the at least one camera, detect that a physical lighting device is present in the images, the physical lighting device rendering a physical light effect, determine the virtual representation of the virtual light effect rendered by the virtual lighting device, and display the images and the virtual representation of the virtual light effect on the at least one display to a user of the electronic device, the virtual representation of the light effect being overlaid on the images. The at least one processor is configured to determine the virtual representation of the virtual light effect based on a representation of the physical light effect in the images and / or adjust a light setting of the physical lighting device based on at least one of the representation of the physical light effect in the images and the virtual light effect.

[0016] When a user sees in AR a physical lighting device and a virtual lighting device side-by-side, it is important to ensure that their light effects look similar. This may be realized by a) adjusting a light setting of the physical lighting device based on the virtual light effect and / or b) determining the virtual representation of the virtual light effect and / or adjusting a light setting of the physical lighting device based on a representation of the physical light effect in the images. The second option is especially beneficial, because physical light effects when viewed directly will look differently than when viewed via the display of the electronic device, which is largely caused by the camera being used to capture the images displayed on the display.

[0017] The at least one processor may be configured to ascertain a difference between the physical light effect and the representation of the physical light effect in the images, and determine the virtual representation of the virtual light effect and / or adjust the light setting of the physical lighting device based on the difference. The at least one processor may be configured to obtain characteristics of the at least one camera and / or the at least one display, e.g. from an Internet server, based on a model number of the electronic device, or from a memory of the electronic device, and ascertain the difference based on these characteristics. Alternatively, the at least one processor may be configured to obtain, from a memory of the electronic device or, based on a model number of the electronic device, from an Internet server, parameter values for calculating the difference and calculate the difference based on these parameter values.

[0018] The at least one processor may be configured to ascertain whether another physical lighting device is visible to the user but not detected in the images and, if another physical lighting device is ascertained to be visible to the user but not detected in the images, adjust the light setting of the physical lighting device based on at least one of the representation of the physical light effect in the images and the virtual light effect. If another physical lighting device is not detected in the images but is visible to the user directly, then determining the virtual representation of the virtual light effect based on a representation of the physical light effect in the images might create a mismatch between the light effects observed by the user in the AR view and the light effects observed by the user outside the AR view. It is therefore beneficial to adjust the light setting of the physical lighting device in this case.

[0019] The at least one processor may be configured to ascertain another physical light setting of the other physical lighting device and, if the other physical lighting device is ascertained to be visible to the user but not detected in the images, adjust the light setting of the physical lighting device based on the representation of the physical light effect in the images and the other physical light setting. This may be done to match e.g. the color of the representation of the physical light effect in the images to the color of the physical light effect rendered by the other physical lighting device. The other physical light setting may be obtained from the other physical lighting device or from a light controller, e.g. bridge, or ascertained by analyzing camera images, e.g. of a wide angle camera.

[0020] The at least one processor may be configured to determine the virtual light effect based on the other physical light setting. This may be done to also match e.g. the color of the virtual representation of the virtual light effect to the color of the physical light effect rendered by the other physical lighting device.

[0021] The at least one processor may be configured to obtain a virtual light setting of the virtual lighting device and determine the virtual light effect based on the virtual light setting. For example, the at least one processor may be configured to receive a physical light setting of the physical lighting device and obtain the virtual light setting of the virtual lighting device by determining the virtual light setting based on the physical light setting. Alternatively, the virtual light setting of the virtual lighting device may be obtained by the at least one processor by receiving a user input signal indicative of the virtual light setting, for example.

[0022] The at least one processor may be configured to display, on the at least one display, a virtual representation of the virtual lighting device overlaid on the images. This is beneficial if the user wants to preview luminaires before purchasing, wants to get help with installation (e.g., for a rail system), or is considering repositioning lights that the user already has.

[0023] The at least one processor may be configured to determine a currently active mode, and determine the virtual representation of the virtual light effect based on the representation of the physical light effect in the images if the currently active mode is a first mode and adjust the light setting of the physical lighting device based on at least one of the representation of the physical light effect in the images and the virtual light effect if the currently active mode is a second mode. This may be used to allows the user to configure the exact behavior of the system.

[0024] The at least one processor may be configured to obtain information indicative of an emissive effect of the physical lighting device, and determine the virtual representation of the virtual light effect based on the emissive effect of the physical lighting device. This may be used to determine the virtual representation of the virtual light effect more accurately.

[0025] In a second aspect, a method of displaying a virtual representation of a virtual light effect rendered by a virtual lighting device comprises obtaining images captured by at least one camera, detecting that a physical lighting device is present in the images, the physical lighting device rendering a physical light effect, determining the virtual representation of the virtual light effect rendered by the virtual lighting device, and displaying the images and the virtual representation of the virtual light effect, the virtual representation of the light effect being overlaid on the images.

[0026] The virtual representation of the virtual light effect is determined based on a representation of the physical light effect in the images and / or a light setting of the physical lighting device is adjusted based on at least one of the representation of the physical light effect in the images and the virtual light effect. The method may be performed by software running on a programmable device. This software may be provided as a computer program product.

[0027] Moreover, a computer program for carrying out the methods described herein, as well as a non-transitory computer readable storage-medium storing the computer program are provided. A computer program may, for example, be downloaded by or uploaded to an existing device or be stored upon manufacturing of these systems.

[0028] A non-transitory computer-readable storage medium stores at least one software code portion, the software code portion, when executed or processed by a computer, being configured to perform executable operations for displaying a virtual representation of a virtual light effect rendered by a virtual lighting device.

[0029] The executable operations comprise obtaining images captured by at least one camera, detecting that a physical lighting device is present in the images, the physical lighting device rendering a physical light effect, determining the virtual representation of the virtual light effect rendered by the virtual lighting device, and displaying the images and the virtual representation of the virtual light effect, the virtual representation of the light effect being overlaid on the images. The virtual representation of the virtual light effect is determined based on a representation of the physical light effect in the images and / or a light setting of the physical lighting device is adjusted based on at least one of the representation of the physical light effect in the images and the virtual light effect.

[0030] As will be appreciated by one skilled in the art, aspects of the present invention may take the form of a device, a method or a computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware implementation, an entirely software implementation (including firmware, resident software, micro-code, etc.) or an implementation combining software and hardware aspects that may all generally be referred to herein as a "circuit", "module" or "system." Functions described in this disclosure may be implemented as an algorithm executed by a processor / microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product in one or more computer readable medium(s) having computer readable program code stored thereon.

[0031] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer readable storage medium may include, but are not limited to, the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the present invention, a computer readable storage medium may be any tangible medium that can contain, or store, a program for use by or in connection with an instruction execution system, apparatus, or device.

[0032] A computer readable signal medium may include a propagated data signal with computer readable program code included therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0033] Program code on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java(TM), Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a standalone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0034] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to implementations of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or a central processing unit (CPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0035] These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0036] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0037] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices, methods and computer program products according to various implementations of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0038] BRIEF DESCRIPTION OF THE DRAWINGS

[0039] These and other aspects of the invention are apparent from and will be further elucidated, by way of example, with reference to the drawings, in which:

[0040] Fig. l is a block diagram of an implementation of the system;

[0041] Fig. 2 shows an example of an image and an overlaid virtual representation of a virtual light effect displayed by the system of Fig. 1;

[0042] Fig. 3 is a flow diagram of a first implementation of the method;

[0043] Fig. 4 is a flow diagram of a second implementation of the method;

[0044] Fig. 5 is a flow diagram of a third implementation of the method;

[0045] Fig. 6 shows an example of an image and an overlaid virtual representation of a virtual light effect displayed in the method of Fig. 5;

[0046] Fig. 7 is a flow diagram of a fourth implementation of the method;

[0047] Fig. 8 is a flow diagram of a fifth implementation of the method; Fig. 9 is a flow diagram of a sixth implementation of the method;

[0048] Fig. 10 is a flow diagram of a seventh implementation of the method; and Fig. 11 is a block diagram of an exemplary data processing system for performing the method.

[0049] Corresponding elements in the drawings are denoted by the same reference numeral.

[0050] DETAILED DESCRIPTION

[0051] Fig. 1 shows an implementation of the electronic device for displaying a virtual representation of a virtual light effect rendered by a virtual lighting device. In this implementation, the system is a mobile device 1. The mobile device 1 may be a mobile phone or a tablet, for example. The mobile device 1 is able to control lighting devices 31 and 32 via a (light) bridge 16, e.g. using Zigbee technology. The bridge 16 may be a Hue bridge, for example.

[0052] The bridge 16 is connected to a wireless LAN access point 17, e.g. via Ethernet or Wi-Fi. In an alternative implementation, the mobile device 1 can alternatively or additionally control one or more of the lighting devices 31 and 32 without a bridge, e.g. directly via Bluetooth or via Internet server 13. The Internet server 13 may be operated by a manufacturer of a lighting company, for example. The Internet server 13 is also connected to the Internet 11.

[0053] The mobile device 1 comprises a receiver 3, a transmitter 4, a processor 5, memory 7, a camera 8, and a touchscreen display 9. The processor 5 is configured to obtain images captured by camera 8 and detect that a physical lighting device, e.g. lighting device 31, is present in the images. The physical lighting device renders a physical light effect. The processor 5 is configured to determine the virtual representation of the virtual light effect rendered by the virtual lighting device and display the images and the virtual representation of the virtual light effect on the display 9 to a user of the mobile device 1. The virtual representation of the light effect is overlaid on the images.

[0054] The processor 5 is configured to determine the virtual representation of the virtual light effect based on a representation of the physical light effect in the images and / or adjust a light setting of the physical lighting device based on at least one of the representation of the physical light effect in the images and the virtual light effect. As a result, the virtual representation of the virtual light effect and the representation of the physical light effect in the images are matched. For example, the size, saturation, and / or color of the effects may be matched and / or the brightness distribution of the effects may be matched (e.g. in wall washing-like effects). Adjusting a light setting of the physical lighting device based on at least one of the representation of the physical light effect in the images may be beneficial in the case of overexposure or oversaturation of the physical light effect in the images, for example.

[0055] The AR feature described above may be part of an app running on the mobile device 1, for example. The app may be used for previewing lighting devices before purchase and support in commissioning and repositioning of lighting devices that are already purchased, for example. The app may have access to all lighting devices and their current state (e.g. current light settings). The app may be able to obtain the locations of the lighting devices in the applicable space (e.g. home or office), e.g. from the bridge 17 or the Internet server 13 (e.g. when the locations have been manually entered during commissioning) or with the help of camera(s) 19.

[0056] It is not required to obtain the locations of the lighting devices to identify the physical lighting device which is detected in the images, e.g. if a light setting of the physical lighting device needs to be adjusted. For example, the physical lighting device may be identified by determining whether any of the current states of the lighting devices match the current state of the physical lighting device detected in the images. For example, if based on the information from the bridge 17, there is only one lighting device currently rendering a blueish light effect and, based on analysis of the images, the physical lighting device detected in the images renders a blueish light effect, then there is a match.

[0057] Matching may not only be performed based on current state, but additionally or instead, based on the type of the lighting device. For example, if based on the information from the bridge 17, there is only one floor lamp and, based on analysis of the images, the physical lighting device detected in the images is a floor lamp, then there is a match. This is especially beneficial for unique luminaire types.

[0058] Another way of identifying the physical lighting device which is detected in the images without the necessity of obtaining the locations of the lighting devices is by using Visible Light Communication (VLC) with which the lighting device could communicate its identification directly to the AR application.

[0059] As soon as the user adds a virtual lighting device in the app, the app could start checking if there are any physical devices in the captured images, i.e. in the AR view. This check may be repeated every image or every few images. If a physical lighting device 31 is detected in the images, the virtual representation of the virtual light effect may be adjusted and / or a light setting of the physical lighting device 31 may be adjusted. For example, if no physical lighting device is detected in the images, the virtual lighting device may render virtual light settings chosen by the user without any adjustment, thereby creating a realistic rendering, and as soon as physical lighting device 31 is detected in the images and the representation of the physical light effect is oversaturated, the saturation of the virtual representation of the virtual light effect may also be increased, thereby creating an exaggerated (matched) rendering.

[0060] Alternatively, as soon as physical lighting device 31 is detected in the images and the representation of the physical light effect is oversaturated, the light setting of the physical lighting device 31 may be adjusted such that the representation of the physical light effect in the images looks like the physical light effect as observed by the user if the user looked, or would have looked, directly at physical lighting device 31 before its light setting was adjusted, thereby creating a realistic (matched) rendering.

[0061] Alternatively, both the virtual representation of the virtual light effect and the light setting of the physical lighting device 31 may be adjusted to create a rendering in between realistic and exaggerated. Alternatively or additionally, as soon as physical lighting device 31 is detected in the images, a light setting of the physical lighting device 31 may be adjusted based on the virtual light effect to match the virtual representation of the virtual light effect and the representation of the physical light effect in the images, e.g. such that the virtual light effect and the physical light effect appear to have the same color or a similar color in the AR view.

[0062] The physical lighting device detected in the images may render a static light effect or a dynamic light effect. If the physical lighting device renders a dynamic light effect, the light settings of the physical lighting device may be adjusted to reduce brightness changes and / or to slow down the light effect. This makes it easier to ascertain the differences between the physical dynamic light effect and the representation of the physical dynamic light effect in the images at different moments, and to match the virtual light effect to the physical light effect.

[0063] The detection of a physical lighting device in the images may be performed based on Al object recognition, for example. As soon as a physical lighting device is detected in the images, the app may poll the bridge 16 to obtain the settings of the physical lighting device (e.g. on / off / brightness / color), which can be used to determine the amount of over saturation that the use of the camera 8 and the display 9 will cause. The calculation of over saturation could be done using Al algorithms and may depend on the type of physical lighting device detected in the images. Matching the virtual representation of the virtual light effect and the representation of the physical light effect in the images is especially important if the virtual lighting device and the physical lighting device are of the same type.

[0064] Fig. 2 shows an example of an image displayed by the mobile device 1 of Fig. 1. In the image shown in Fig. 2, which has been captured by camera 8 of mobile device 1, a physical lighting device 31 is represented. The physical lighting device 31 renders a physical light effect 36. The physical light effect 36 is represented by a representation 42 in the image displayed on display 9. The color of this representation 42 may be oversaturated compared to how the color of the physical light effect 36 appears to the user directly, for example.

[0065] Overlaid on the image, a virtual representation 52 of a virtual light effect rendered by a virtual lighting device is displayed on the display 9. In the example of Fig. 2, a virtual representation 51 of the virtual lighting device is also displayed overlaid on the image.

[0066] Since the physical lighting device 31 is detected in the image, the virtual representation 52 of the virtual light effect is determined based on a representation of the physical light effect in the image and / or a light setting of the physical lighting device 31 is adjusted based on the representation of the physical light effect in the image and / or the virtual light effect. As a result, the virtual representation 52 of the virtual light effect and the representation of the physical light effect in the image are matched, e.g. have the same color and / or the same light effect size.

[0067] When a light setting of the physical lighting device 31 is adjusted based on a representation of the physical light effect in the image, the following behavior might be observed. When physical lighting device 31 comes into the field of view of camera 8, the virtual representation 42 may at first show a different light effect than desired, but after the light setting of the physical lighting device 31 has been adjusted, the virtual representation 42 in the next images will show the desired light effect. It may be possible for the user to configure the exact behavior of the system, e.g. via a configuration menu.

[0068] The processor 5 may be configured to ascertain whether another physical lighting device is visible to the user but not detected in the images and, if another physical lighting device, e.g. physical lighting device 32, is ascertained to be visible to the user but not detected in the images, adjust the light setting of the physical lighting device based on at least one of the representation of the physical light effect in the images and the virtual light effect. This is especially beneficial if the processor 5 is not able to control the other physical lighting device, which is outside the AR field-of-view, and thus not able to adjust the light setting of the other physical lighting device. The app may running on mobile device 1 may detect the presence and the state (e.g. current light settings) of physical lighting devices that are not in the field-of-view of camera but that might be in the field-of-view of the user. As described above, the app may have access to all lighting devices and their current state (e.g. current light settings) and the app may be able to obtain the locations of the lighting devices in the applicable space (e.g. home or office), e.g. from the bridge 17 or the Internet server 13 or with the help of camera(s) 19.

[0069] In an alternative implementation, the mobile device 1 has multiple cameras where the displayed images are captured by a smaller angle camera feed and a wider-angle camera is used for capturing the area around the area captured by the smaller angle camera, i.e. outside the AR field-of-view.

[0070] In the implementation of the mobile device 1 shown in Fig. 1, the mobile device 1 comprises one processor 5. In an alternative implementation, the mobile device 1 comprises multiple processors. The processor 5 of the mobile device 1 may be a general- purpose processor, e.g. from ARM or Qualcomm or an application-specific processor. The processor 5 of the mobile device 1 may run an Android or iOS operating system for example. The camera 8 may comprise a CMOS or CCD sensor, for example. The display 9 may comprise an LCD or OLED display panel, for example. The memory 7 may comprise one or more memory units. The memory 7 may comprise solid state memory, for example.

[0071] The receiver 3 and the transmitter 4 may use one or more wireless communication technologies, e.g. Wi-Fi (IEEE 802.11) for communicating with the wireless LAN access point 17, for example. In an alternative implementation, multiple receivers and / or multiple transmitters are used instead of a single receiver and a single transmitter. In the implementation shown in Fig. 1, a separate receiver and a separate transmitter are used. In an alternative implementation, the receiver 3 and the transmitter 4 are combined into a transceiver. The mobile device 1 may comprise other components typical for a mobile device such as a battery and a power connector.

[0072] The processor 5 may be configured in the manner described above by configuring the processor 5 to run a suitably programmed computer program product, e.g. a computer program product comprising computer program code to perform the method of Fig. 3.

[0073] A first implementation of the method of displaying a virtual representation of a virtual light effect rendered by a virtual lighting device is shown in Fig. 3. The method may be performed by the mobile device 1 of Fig.1, for example. A step 101 comprises obtaining images captured by at least one camera. A step 103 comprises detecting that a physical lighting device, which renders a physical light effect, is present in the images obtained in step 101. A step 105 comprises determining the virtual representation of the virtual light effect rendered by the virtual lighting device. An optional step 107 comprises adjusting a light setting of the physical lighting device.

[0074] A step 109 comprises displaying the images obtained in step 101 and the virtual representation of the virtual light effect determined in step 105. The virtual representation of the light effect is overlaid on the images. Optionally, step 109 further comprises displaying a virtual representation of the virtual lighting device overlaid on the images.

[0075] The virtual representation of the virtual light effect is determined in step 105 based on a representation of the physical light effect in the images and / or the light setting of the physical lighting device is adjusted in step 107 based on at least one of the representation of the physical light effect in the images and the virtual light effect. Additionally, step 121 of Fig. 4, steps 131-135 of Fig. 5, steps 151-155 of Fig. 7 (or steps 161 and 155 of Fig. 8), steps 181-189 of Fig. 9, and / or steps 191-193 of Fig. 10 may be added to the implementation of Fig. 3.

[0076] A second implementation of the method of displaying a virtual representation of a virtual light effect rendered by a virtual lighting device is shown in Fig. 4. The implementation of Fig. 4 is an extension of the implementation of Fig. 3. In the implementation of Fig. 4, a step 121 is performed between steps 103 and 105.

[0077] Step 121 comprises ascertaining a difference between the physical light effect and the representation of the physical light effect in the images. For example, step 121 may comprise obtaining characteristics of the at least one camera and / or the at least one display, e.g. from an Internet server, based on a model number of the electronic device, or from a memory of the electronic device, and ascertaining the difference based on these characteristics. Alternatively, step 121 may comprise obtaining, from a memory of the electronic device or, based on a model number of the electronic device, from an Internet server, parameter values for calculating the difference and calculating the difference based on these parameter values, for example.

[0078] When ascertaining the difference between the physical light effect and the representation of the physical light effect in the images, it may be taken into account whether the camera supports HDR (High Dynamic Range). HDR technology is implemented in many modern smartphones and cameras. HDR works by taking multiple photos at different exposure levels (usually a high, medium, and low exposure) in quick succession and then combining these images into a single photo. This process enhances the photo's overall dynamic range, meaning it can show more details in both the darkest and brightest areas of the picture. When the captured images that are shown in the AR view are HDR images, there will normally be less overexposure of the physical light effect in the images.

[0079] Based on the difference ascertained in step 121, the virtual representation of the virtual light effect is determined in step 105 and / or the light setting of the physical lighting device is adjusted in step 107. Additionally, steps 131-135 of Fig. 5, steps 151-155 of Fig. 7 (or steps 155 and 161 of Fig. 8), steps 181-189 of Fig. 9, and / or steps 191-193 of Fig. 10 may be added to the implementation of Fig. 4.

[0080] A third implementation of the method of displaying a virtual representation of a virtual light effect rendered by a virtual lighting device is shown in Fig. 5. The implementation of Fig. 5 is an extension of the implementation of Fig. 3. In the implementation of Fig. 5, step 107 of Fig. 3 is implemented by a step 135 and a step 133 is performed between steps 105 and 135.

[0081] Step 133 comprises ascertaining whether another physical lighting device is visible to the user but not detected in the images. When a user wears a head mounted device, lighting devices not detected in the images will not be visible to the user, but with a smartphone or tablet, a physical lighting device might not be represented in the AR view but still be visible to the user directly, i.e. outside the AR view. If another physical lighting device is ascertained to be visible to the user but not detected in the images, the light setting of the physical lighting device is adjusted in step 135 based on at least one of the representation of the physical light effect in the images and the virtual light effect.

[0082] If another physical lighting device is not detected in the images but is visible to the user directly, then determining the virtual representation of the virtual light effect based on a representation of the physical light effect in the images might create a mismatch between the light effects observed by the user in the AR view and the light effects observed by the user outside the AR view. It is therefore beneficial to adjust the light setting of the physical lighting device in this case.

[0083] An optional step 131 may be performed before step 105. Step 131 comprises ascertaining a physical light setting of the other physical lighting device. This physical light setting may be obtained from the other physical lighting device or from a light controller, e.g. bridge, or ascertained by analyzing camera images, e.g. of a wide angle camera. The virtual light effect may be determined in step 105 based on the other physical light setting ascertained in step 131, for example.

[0084] If step 131 is performed and the other physical lighting device is ascertained to be visible to the user but not detected in the images, the light setting of the physical lighting device may be adjusted in step 135 based on the representation of the physical light effect in the images and the other physical light setting ascertained in step 131. This may be done to match e.g. the color of the representation of the physical light effect in the images to the color of the physical light effect rendered by the other physical lighting device. Additionally, step 121 of Fig. 4, steps 151-155 of Fig. 7 (or steps 161 and 155 of Fig. 8), steps 181-189 of Fig. 9, and / or steps 191-193 of Fig. 10 may be added to the implementation of Fig. 5.

[0085] Fig. 6 shows an example of an image displayed by the mobile device 1 of Fig. 1 when the processor of the mobile device is configured to perform the method of Fig. 5. Like in the example of Fig. 2, a physical lighting device 31 is represented in the image. The physical lighting device 31 renders a physical light effect 37. The physical light effect 37 is represented by a representation 43 in the image displayed on display 9. The color of this representation 43 may be oversaturated compared to how the color of the physical light effect 37 appears to the user directly, for example.

[0086] Overlaid on the image, a virtual representation 53 of a virtual light effect rendered by a virtual lighting device is displayed on the display 9. In the example of Fig. 6, a virtual representation 51 of the virtual lighting device is also displayed overlaid on the image. Different from the example of Fig. 2, another physical lighting device 32 is also visible to the user of the mobile device 1, but not represented in the image displayed on the display 9. The physical lighting device 32 renders a physical light effect 39.

[0087] Since the physical lighting device 31 is detected in the image and the other physical lighting device 32 is ascertained to be visible to the user but not detected in the image, the light setting of the physical lighting device 31 is adjusted (in step 135 of Fig. 5) based on the representation of the physical light effect in the images and / or the virtual light effect.

[0088] By adjusting the light setting of the physical lighting device 31 based on the representation 43 of the physical light effect, any difference between the appearance of the light effect rendered by the lighting device 31 directly to the user and the appearance of this light effect in the images is compensated, thereby resulting in a match between the appearance of the representation 43 of the physical light effect 37 in the images and the appearance of the virtual representation 53 of the virtual light effect. In this case, it is not sufficient to determine the virtual representation 53 of the virtual light effect based on the representation of the physical light effect 37 in the image, because there would be a mismatch between the light effects represented in the images and the physical light effect 39 rendered by the physical lighting device 32, which is visible directly to the user.

[0089] A fourth implementation of the method of displaying a virtual representation of a virtual light effect rendered by a virtual lighting device is shown in Fig. 7. The implementation of Fig. 7 is an extension of the implementation of Fig. 3. In the implementation of Fig. 7, step 105 is implemented by a step 155 and steps 151 and 153 are performed between steps 103 and 155.

[0090] Step 151 comprises receiving a physical light setting of the physical lighting device. Step 153 comprises obtaining a virtual light setting of the virtual lighting device by determining the virtual light setting based on the physical light setting received in step 151. Step 155 comprises determining the virtual light effect based on the virtual light setting obtained in step 153. Additionally, step 121 of Fig. 4, steps 131-135 of Fig. 5, steps 181-189 of Fig. 9, and / or steps 191-193 of Fig. 10 may be added to the implementation of Fig. 7.

[0091] A fifth implementation of the method of displaying a virtual representation of a virtual light effect rendered by a virtual lighting device is shown in Fig. 8. The implementation of Fig. 8 is an extension of the implementation of Fig. 3. In the implementation of Fig. 8, step 105 is implemented by step 155 and a step 161 is performed between steps 103 and 155.

[0092] Step 161 comprises obtaining a virtual light setting of the virtual lighting device by receiving a user input signal indicative of the virtual light setting. Step 155 comprises determining the virtual light effect based on the virtual light setting obtained in step 161. Additionally, step 121 ofFig. 4, steps 131-135 ofFig. 5, steps 181-189 ofFig. 9, and / or steps 191-193 ofFig. 10 may be added to the implementation ofFig. 8.

[0093] A sixth implementation of the method of displaying a virtual representation of a virtual light effect rendered by a virtual lighting device is shown in Fig. 9. The implementation ofFig. 9 is an extension of the implementation ofFig. 3. Step 101 comprises obtaining images captured by at least one camera. Step 103 comprises detecting that a physical lighting device, which renders a physical light effect, is present in the images obtained in step 101.

[0094] Step 181 comprises determining a currently active mode. A step 183 comprises checking whether the currently active mode is a first mode (mode 1) or a second mode (mode 2). The first mode may be an exaggerated mode and the second mode may be a realistic mode, for example. Step 185 is performed if the currently active mode is the first mode. Step 187 is performed if the currently active mode is the second mode.

[0095] Step 185 comprises determining the virtual representation of the virtual light effect based on the representation of the physical light effect in the images. Step 187 comprises determining the virtual representation of the virtual light effect not based on, or only slightly based on, the representation of the physical light effect in the images. A step 189 is performed after step 187. Step 189 comprises adjusting the light setting of the physical lighting device based on at least one of the representation of the physical light effect in the images and the virtual light effect.

[0096] For example, in the first mode (step 185), the effects represented in the AR view may focus on higher, exaugurated effects rendering, which may be used for previewing an upcoming product (i.e. a product not yet in stores). In the second mode (steps 187 and 189), the effects represented in the AR view may focus on realistic light effects rendering, which may be used for previewing a luminaire when selecting and purchasing a luminaire. In both modes, the virtual and physical light effects in the AR view are matched, i.e. made similar, but in different ways.

[0097] Step 109 comprises displaying the images obtained in step 101 and the virtual representation of the virtual light effect determined in step 185 or step 187. The virtual representation of the light effect is overlaid on the images. Additionally, step 121 of Fig. 4, steps 131-135 of Fig. 5, steps 151-155 of Fig. 7 (or steps 161 and 155 of Fig. 8), and / or steps 191-193 of Fig. 10 may be added to the implementation of Fig. 9.

[0098] A seventh implementation of the method of displaying a virtual representation of a virtual light effect rendered by a virtual lighting device is shown in Fig. 10. The implementation of Fig. 10 is an extension of the implementation of Fig. 3. In the implementation of Fig. 10, step 105 is implemented by a step 193 and a step 191 is performed between steps 103 and 193.

[0099] Step 191 comprises obtaining information indicative of an emissive effect of the physical lighting device. Step 193 comprises determining the virtual representation of the virtual light effect based on the emissive effect of the physical lighting device as indicated in the information obtained in step 191. Thus, the emissive effect of the virtual lighting device is adjusted to the emissive effect of the physical lighting device. Additionally, step 121 of Fig. 4, steps 131-135 of Fig. 5, steps 151-155 of Fig. 7 (or steps 161 and 155 of Fig. 8), and / or steps 181-189 of Fig. 9 may be added to the implementation of Fig. 10. Fig. 11 depicts a block diagram illustrating an exemplary data processing system that may perform the method as described with reference to Figs. 3 to 5 and 7 to 10.

[0100] As shown in Fig. 11, the data processing system 300 may include at least one processor 302 coupled to memory elements 304 through a system bus 306. As such, the data processing system may store program code within memory elements 304. Further, the processor 302 may execute the program code accessed from the memory elements 304 via a system bus 306. In one aspect, the data processing system may be implemented as a computer that is suitable for storing and / or executing program code. It should be appreciated, however, that the data processing system 300 may be implemented in the form of any system including a processor and a memory that is capable of performing the functions described within this specification. The data processing system may be an Intemet / cloud server, for example.

[0101] The memory elements 304 may include one or more physical memory devices such as, for example, local memory 308 and one or more bulk storage devices 310. The local memory may refer to random access memory or other non-persistent memory device(s) generally used during actual execution of the program code. A bulk storage device may be implemented as a hard drive or other persistent data storage device. The processing system 300 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the quantity of times program code must be retrieved from the bulk storage device 310 during execution. The processing system 300 may also be able to use memory elements of another processing system, e.g. if the processing system 300 is part of a cloud-computing platform.

[0102] Input / output (VO) devices depicted as an input device 312 and an output device 314 optionally can be coupled to the data processing system. Examples of input devices may include, but are not limited to, a keyboard, a pointing device such as a mouse, a microphone (e.g. for voice and / or speech recognition), or the like. Examples of output devices may include, but are not limited to, a monitor or a display, speakers, or the like. Input and / or output devices may be coupled to the data processing system either directly or through intervening VO controllers.

[0103] The input and the output devices may be implemented as a combined input / output device (illustrated in Fig. 11 with a dashed line surrounding the input device 312 and the output device 314). An example of such a combined device is a touch sensitive display, also sometimes referred to as a “touch screen display” or simply “touch screen”. In such an implementation, input to the device may be provided by a movement of a physical object, such as e.g. a stylus or a finger of a user, on or near the touch screen display.

[0104] A network adapter 316 may also be coupled to the data processing system to enable it to become coupled to other systems, computer systems, remote network devices, and / or remote storage devices through intervening private or public networks. The network adapter may comprise a data receiver for receiving data that is transmitted by the systems, devices and / or networks to the data processing system 300, and a data transmitter for transmitting data from the data processing system 300 to the systems, devices and / or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapter that may be used with the data processing system 300.

[0105] As pictured in Fig. 11, the memory elements 304 may store an application 318. The application 318 may be stored in the local memory 308, the one or more bulk storage devices 310, or separate from the local memory and the bulk storage devices. It should be appreciated that the data processing system 300 may further execute an operating system (not shown in Fig. 11) that can facilitate execution of the application 318. The application 318, being implemented in the form of executable program code, can be executed by the data processing system 300, e.g., by the processor 302. Responsive to executing the application, the data processing system 300 may be configured to perform one or more operations or method steps described herein.

[0106] The invention may be implemented as a program product for use with a computer system, where the program(s) of the program product define functions. The program(s) may be contained on a variety of non-transitory computer-readable storage media, where, as used herein, the expression “non-transitory computer readable storage media” comprises all computer-readable media, with the sole exception being a transitory, propagating signal. The program(s) may also be contained on a variety of transitory computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, ROM chips or any type of solid-state non-volatile semiconductor memory) on which information is permanently stored; and (ii) writable storage media (e.g., flash memory, floppy disks within a diskette drive or hard-disk drive or any type of solid-state random-access semiconductor memory) on which alterable information is stored. The computer program may be run on the processor 302 described herein. The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0107] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The detailed description has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the implementations in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the present invention.

Claims

CLAIMS:

1. An electronic device (1) for displaying a virtual representation (52) of a virtual light effect rendered by a virtual lighting device, the electronic device (1) comprising: at least one camera (8); at least one display (9); and at least one processor (5) configured to:- obtain images captured by the at least one camera (8),- detect that a physical lighting device (31) is present in the images, the physical lighting device (31) rendering a physical light effect,- determine the virtual representation (52) of the virtual light effect rendered by the virtual lighting device, and- display the images and the virtual representation of the virtual light effect (52) on the at least one display (9) to a user of the electronic device (1), the virtual representation of the light effect being overlaid on the images, wherein the at least one processor is further configured to:- ascertain a difference between the physical light effect and the representation of the physical light effect in the images, and- determine the virtual representation (52) of the virtual light effect based on the difference between the physical light effect and the representation of the physical light effect in the images and / or adjust a light setting of the physical lighting device (31) based on the difference between the physical light effect and the representation of the physical light effect in the images.

2. An electronic device (1) as claimed in any one of the preceding claims, wherein the at least one processor (5) is configured to obtain a virtual light setting of the virtual lighting device and determine the virtual light effect based on the virtual light setting.

3. An electronic device (1) as claimed in claim 2, wherein the at least one processor (5) is configured to receive a physical light setting of the physical lighting device(31) and obtain the virtual light setting of the virtual lighting device by determining the virtual light setting based on the physical light setting.

4. An electronic device (1) as claimed in claim 2, wherein the at least one processor (5) is configured to obtain the virtual light setting of the virtual lighting device by receiving a user input signal indicative of the virtual light setting.

5. An electronic device (1) as claimed in any one of the preceding claims, wherein the at least one processor (5) is configured to display, on the at least one display (9), a virtual representation (51) of the virtual lighting device overlaid on the images.

6. An electronic device (1) as claimed in any one of the preceding claims, wherein the at least one processor (5) is configured to:- determine a currently active mode, and- determine the virtual representation (52) of the virtual light effect based on based on the representation of the physical light effect in the images if the currently active mode is a first mode and adjust the light setting of the physical lighting device (31) based on at least one of the representation of the physical light effect in the images and the virtual light effect if the currently active mode is a second mode.

7. An electronic device (1) as claimed in any one of the preceding claims, wherein the at least one processor (5) is configured to:- obtain information indicative of an emissive effect of the physical lighting device (31), and- determine the virtual representation (52) of the virtual light effect based on the emissive effect of the physical lighting device (31).

8. A method of displaying a virtual representation of a virtual light effect rendered by a virtual lighting device, the method comprising:- obtaining (101) images captured by at least one camera;- detecting (103) that a physical lighting device is present in the images, the physical lighting device rendering a physical light effect;- determining (105) the virtual representation of the virtual light effect rendered by the virtual lighting device; and- displaying (109) the images and the virtual representation of the virtual light effect, the virtual representation of the light effect being overlaid on the images,- ascertaining a difference between the physical light effect and the representation of the physical light effect in the images, wherein the virtual representation of the virtual light effect is determined based on the difference between the physical light effect and the representation of the physical light effect in the images and / or a light setting of the physical lighting device is adjusted based on the difference between the physical light effect and the representation of the physical light effect in the images.

9. A computer program product for a computing device, the computer program product comprising computer program code to perform the method of claim 8 when the computer program product is run on a processing unit of the computing device.

Images