representation of a virtual light effect

Abstract

An electronic device (1) for displaying a virtual representation (52) of a virtual light effect rendered by a virtual lighting device (51) is configured to obtain an image captured by a camera, detect that a physical lighting device (31) is present in the image, the physical lighting device rendering a physical light effect (36). Determine a virtual representation of the virtual light effect rendered by the virtual lighting device, and display the image and the virtual representation of the virtual light effect overlaid on the image to a user of the electronic device. The electronic device is configured to determine the virtual representation of the virtual light effect based on a representation (42) of the physical light effect in the image and / or to adjust light settings of the physical lighting device based on the representation of the physical light effect in the image and / or the virtual light effect.

Description

Technical Field

[0001] This invention relates to an electronic device for displaying a virtual representation of virtual lighting effects presented by a virtual lighting device.

[0002] The present invention also relates to a method for displaying a virtual representation of virtual lighting effects presented by a virtual lighting device.

[0003] The present invention also relates to a computer program product that enables a computer system to execute this method. Background Technology

[0004] As smartphones and AR devices become increasingly capable of merging the physical world (camera stream) with virtual objects, the number of augmented reality (AR) based applications is growing rapidly. Among the growing number of AR-based applications, virtual lighting effects are being presented. In some AR-based applications, simply adding virtual lighting fixtures to a room is insufficient; it is important for users to see the lighting effects and their overall impact on the room's atmosphere.

[0005] US 2020 / 0410725 A1 discloses a system for evaluating AR experiences. Visitors to theme parks or amusement parks wear head-mounted displays to view virtual features / objects. This allows for enhanced visitor experiences by providing adjustable virtual environments for different experiences within the same amusement park ride. To allow developers to create virtual environments without physically visiting every location in the amusement park, the head-mounted display is configured to overlay virtual objects in the virtual space onto a background, thereby enabling developers to perceive the virtual objects as being physically present in the space using the head-mounted display.

[0006] The virtual space also includes virtual lights that correspond to physical lights in the physical space. The system can adjust the hue and intensity of the virtual light emitted by these virtual lights in the virtual space to match the hue and intensity of the light output by the corresponding physical lights in the physical space. This allows developers to evaluate whether an AR system configured to generate AR features and overlay them onto a real-world environment effectively adjusts the appearance of the AR features based on variations in lighting conditions in the real-world environment.

[0007] US 2020 / 0187334 A1 discloses systems and methods for generating lighting designs for venues.

[0008] In the evaluation system of US 2020 / 0410725 A1, virtual lights in virtual space correspond to physical lights in physical space. However, this is insufficient if a user wants to preview the lights before purchasing, needs installation assistance (e.g., for track systems), or is considering rearranging existing lights. In such AR applications, users should be able to see in AR a virtual representation of the virtual lighting effects presented by virtual lighting equipment that does not correspond to physical lighting equipment in the user's surrounding environment (e.g., because the virtual lighting equipment is located in a different location than the corresponding physical lighting equipment in the user's surrounding environment), as well as a representation of the physical lighting effects presented by the physical lighting equipment. Summary of the Invention

[0009] It is advantageous to provide an electronic device that allows a user to see, in AR, a virtual representation of virtual lighting effects presented by virtual lighting devices that do not correspond to physical lighting devices in the user's surrounding environment, as well as a representation of physical lighting effects presented by physical lighting devices, without producing an inconsistent viewing experience.

[0010] One approach is advantageous in allowing users to see, in AR, a virtual representation of virtual lighting effects presented by virtual lighting devices that do not correspond to physical lighting devices in the user's surrounding environment, as well as a representation of physical lighting effects presented by physical lighting devices, without creating an inconsistent viewing experience.

[0011] In a first aspect, an electronic device for displaying a virtual representation of a virtual lighting effect presented by a virtual lighting device includes: at least one camera, at least one display, and at least one processor configured to: acquire an image captured by the at least one camera, detect the presence of a physical lighting device in the image, the physical lighting device presenting a physical lighting effect, determine a virtual representation of the virtual lighting effect presented by the virtual lighting device, and display the image and the virtual representation of the virtual lighting effect to a user of the electronic device on the at least one display, the virtual representation of the lighting effect being overlaid on the image.

[0012] At least one processor is configured to: determine a virtual representation of a virtual lighting effect based on a representation of physical lighting effects in an image, and / or adjust the light settings of a physical lighting device based on at least one of the representation of physical lighting effects in an image and the virtual lighting effect.

[0013] When users see physical and virtual lighting devices side-by-side in AR, it's important to ensure their lighting effects appear similar. This can be achieved by: a) adjusting the lighting settings of the physical lighting devices based on the virtual lighting effects, and / or b) determining a virtual representation of the virtual lighting effects, and / or adjusting the lighting settings of the physical lighting devices based on the representation of the physical lighting effects in the image. The second option is particularly beneficial because the physical lighting effects will look different when viewed directly from the screen of an electronic device compared to when viewed from the screen, primarily due to the camera used to capture the image displayed on the screen.

[0014] At least one processor can be configured to determine the difference between physical lighting effects and their representation in an image, and based on that difference, determine a virtual representation of the virtual lighting effects and / or adjust the lighting settings of the physical lighting equipment. At least one processor can be configured to, for example, obtain characteristics of at least one camera and / or at least one display from an internet server based on the model of the electronic device, or from the memory of the electronic device, and determine the difference based on those characteristics. Alternatively, at least one processor can be configured to obtain parameter values ​​for calculating the difference from the memory of the electronic device, or from an internet server based on the model of the electronic device, and calculate the difference based on those parameter values.

[0015] At least one processor can be configured to: determine whether another physical lighting device is visible to the user but not detected in the image, and if it is determined that the other physical lighting device is visible to the user but not detected in the image, adjust the light settings of the physical lighting device based on at least one of the representation of physical lighting effects in the image and virtual lighting effects. If the other physical lighting device is not detected in the image but is directly visible to the user, determining the virtual representation of the virtual lighting effects based on the representation of physical lighting effects in the image may create a mismatch between the lighting effects observed by the user in the AR view and the lighting effects observed by the user outside the AR view. Therefore, adjusting the light settings of the physical lighting device is beneficial in this case.

[0016] At least one processor can be configured to: determine another physical light setting of another physical lighting device, and if the determined other physical lighting device is visible to the user but not detected in the image, adjust the light setting of the physical lighting device based on the representation of the physical lighting effect in the image and the other physical light setting. This can be done, for example, to match the color of the representation of the physical lighting effect in the image to the color of the physical lighting effect presented by the other physical lighting device. The other physical light setting can be obtained from the other physical lighting device or from a lighting controller (e.g., a bridge), or determined by analyzing, for example, camera images from a wide-angle camera.

[0017] At least one processor can be configured to determine the virtual lighting effect based on another physical lighting setup. This can also be done, for example, to match the color of a virtual representation of the virtual lighting effect to the color of a physical lighting effect rendered by another physical lighting device.

[0018] At least one processor can be configured to obtain the virtual light settings of a virtual lighting device and determine virtual lighting effects based on the virtual light settings. For example, at least one processor can be configured to receive the physical light settings of a physical lighting device and obtain the virtual light settings of the virtual lighting device by determining the virtual light settings based on the physical light settings. Alternatively, the virtual light settings of the virtual lighting device can be obtained by at least one processor, for example, by receiving a user input signal indicating the virtual light settings.

[0019] At least one processor can be configured to display a virtual representation of a virtual lighting fixture overlaid on an image on at least one display. This is beneficial if a user wants to preview the lighting fixtures before purchasing, wants installation assistance (e.g., for track systems), or is considering rearranging existing lights.

[0020] At least one processor can be configured to: determine the current activity mode, and if the current activity mode is a first mode, determine a virtual representation of the virtual lighting effect based on the representation of the physical lighting effect in the image, and if the current activity mode is a second mode, adjust the light settings of the physical lighting device based on at least one of the representation of the physical lighting effect in the image and the virtual lighting effect. This can be used to allow the user to configure the exact behavior of the system.

[0021] At least one processor can be configured to: acquire information indicating the luminous effect of a physical lighting device, and determine a virtual representation of the virtual lighting effect based on the luminous effect of the physical lighting device. This can be used to determine the virtual representation of the virtual lighting effect more accurately.

[0022] In a second aspect, a method for displaying a virtual representation of a virtual lighting effect presented by a virtual lighting device includes: obtaining an image captured by at least one camera; detecting the presence of a physical lighting device in the image that presents a physical lighting effect; determining a virtual representation of the virtual lighting effect presented by the virtual lighting device; and displaying the image and the virtual representation of the virtual lighting effect overlaid on the image.

[0023] The method determines a virtual representation of a virtual lighting effect based on a representation of physical lighting effects in an image and / or adjusts the light settings of a physical lighting device based on at least one of the representation of physical lighting effects in an image and virtual lighting effects. This method can be executed by software running on a programmable device. The software can be provided as a computer program product.

[0024] In addition, a computer program for performing the methods described herein is provided, as well as a non-transitory computer-readable storage medium for storing the computer program. The computer program may be downloaded or uploaded to an existing device, for example, or stored during the manufacture of these systems.

[0025] A non-transitory computer-readable storage medium stores at least one portion of software code that, when executed or processed by a computer, is configured to perform executable operations for displaying a virtual representation of virtual lighting effects rendered by a virtual lighting device.

[0026] The executable operation includes: acquiring an image captured by at least one camera, detecting the presence of a physical lighting device in the image, the physical lighting device presenting a physical lighting effect, determining a virtual representation of a virtual lighting effect presented by a virtual lighting device, and displaying the image and the virtual representation of the virtual lighting effect, the virtual representation of the lighting effect being overlaid on the image.

[0027] The virtual representation of the virtual lighting effect is determined based on the representation of the physical lighting effect in the image, and / or the light settings of the physical lighting device are adjusted based on at least one of the representation of the physical lighting effect in the image and the virtual lighting effect.

[0028] As those skilled in the art will appreciate, aspects of the present invention can take the form of an apparatus, method, or computer program product. Therefore, aspects of the present invention can take the form of a completely hardware implementation, a completely software implementation (including firmware, resident software, microcode, etc.), or a combination of software and hardware aspects, all of which are generally referred to herein as “circuit,” “module,” or “system.” The functions described in this disclosure can be implemented as algorithms executed by a computer’s processor / microprocessor. Furthermore, aspects of the present invention can take the form of a computer program product on one or more computer-readable media having computer-readable program code stored thereon.

[0029] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this invention, a computer-readable storage medium can be any tangible medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device.

[0030] A computer-readable signal medium may include a propagated data signal having computer-readable program code included therein (e.g., in baseband or as part of a carrier wave). Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and may convey, propagate, or deliver a program used by or in conjunction with an instruction execution system, apparatus, or device.

[0031] The program code on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic, cable, RF, or any suitable combination thereof. The computer program code for carrying out the operations of various aspects of the invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java™, Smalltalk, or C++) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The program code may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., using an internet service provider via the internet).

[0032] Various aspects of the invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, particularly a microprocessor or central processing unit (CPU), to produce a machine such that the instructions, executable via the processor of the computer, other programmable data processing apparatus, or other device, create means for implementing the functions / actions specified in the flowchart illustrations and / or one or more block diagram blocks.

[0033] These computer program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing apparatus, or other device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing, which includes instructions that implement functions / actions specified in flowcharts and / or one or more block diagrams.

[0034] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide for implementing the functions / actions specified in the flowchart and / or one or more block diagram boxes.

[0035] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing a specified logical function(s). It should also be noted that in some alternative embodiments, the functions described in the blocks may not appear in the order shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or sometimes these blocks may be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware, or a combination of dedicated hardware and computer instructions, that performs the specified function or action. Attached Figure Description

[0036] Referring to the accompanying drawings, these and other aspects of the invention will be clear and further illustrated by way of example, in which: Figure 1 This is a block diagram of one implementation of the system; Figure 2 It shows the result of Figure 1 Examples of virtual representations of the images displayed by the system and the virtual lighting effects it covers; Figure 3 This is a flowchart of the first embodiment of the method; Figure 4 This is a flowchart of the second embodiment of the method; Figure 5 This is a flowchart of the third implementation of the method; Figure 6 It shows in Figure 5 Examples of images and virtual representations of the overlaid virtual lighting effects shown in the method; Figure 7 This is a flowchart of the fourth implementation of the method; Figure 8 This is a flowchart of the fifth implementation of the method; Figure 9 This is a flowchart of the sixth implementation of the method; Figure 10 This is a flowchart of the seventh embodiment of the method; and Figure 11 This is a block diagram of an exemplary data processing system for executing methods.

[0037] Corresponding elements in the accompanying drawings are labeled with the same reference numerals. Detailed Implementation

[0038] Figure 1 An embodiment of an electronic device for displaying a virtual representation of virtual lighting effects presented by virtual lighting devices is shown. In this embodiment, the system is a mobile device 1. The mobile device 1 may be, for example, a mobile phone or a tablet computer. The mobile device 1 is capable of controlling lighting devices 31 and 32 via a (lighting) bridge 16 (e.g., using Zigbee technology). The bridge 16 may be, for example, a Hue bridge.

[0039] Bridge 16 is connected to wireless LAN access point 17, for example, via Ethernet or Wi-Fi. In an alternative embodiment, mobile device 1 may alternatively or additionally control one or more of lighting devices 31 and 32, for example, directly via Bluetooth or via Internet server 13, without a bridge. Internet server 13 may be operated by the manufacturer of the lighting company, for example. Internet server 13 is also connected to Internet 11.

[0040] Mobile device 1 includes receiver 3, transmitter 4, processor 5, memory 7, camera 8, and touchscreen display 9. Processor 5 is configured to acquire an image captured by camera 8 and detect the presence of a physical lighting device (e.g., lighting device 31) in the image. This physical lighting device presents a physical lighting effect. Processor 5 is configured to determine a virtual representation of a virtual lighting effect presented by a virtual lighting device and display the image and the virtual representation of the virtual lighting effect to the user of mobile device 1 on display 9. The virtual representation of the lighting effect is overlaid on the image.

[0041] Processor 5 is configured to determine a virtual representation of a virtual lighting effect based on a representation of physical lighting effects in an image, and / or to adjust the light settings of a physical lighting device based on at least one of the representation of physical lighting effects in an image and the virtual lighting effect. As a result, the virtual representation of the virtual lighting effect matches the representation of physical lighting effects in the image. For example, the size, saturation, and / or color of the effect can be matched, and / or the brightness distribution of the effect can be matched (e.g., in a wall-washing effect).

[0042] For example, in cases where the physical lighting effect in an image is overexposed or oversaturated, it may be beneficial to adjust the light settings of the physical lighting device based on at least one of the representations of the physical lighting effect in the image.

[0043] The aforementioned AR features could, for example, be part of an application running on mobile device 1. This application could be used, for example, to preview lighting equipment before purchase and to support network initialization and rearrangement of purchased lighting equipment. The application could access all lighting equipment and their current status (e.g., current light settings). The application could be able to obtain, for example, the location of the lighting equipment in the applicable space (e.g., home or office) from bridge 17 or internet server 13 (e.g., when the location has been manually entered during network initialization) or by means of cameras (multiple) 19.

[0044] For example, when adjusting the light settings of physical lighting equipment, it is not necessary to obtain the location of the lighting equipment to identify the physical lighting equipment detected in the image. For example, a physical lighting equipment can be identified by determining whether any of the current states of the lighting equipment matches the current state of a physical lighting equipment detected in the image. For example, if based on information from bridge 17, only one lighting equipment is currently exhibiting a bluish light effect, and based on the analysis of the image, the physical lighting equipment detected in the image also exhibits a bluish light effect, then a match exists.

[0045] Matching can be performed not only based on the current state, but also additionally or alternatively based on the type of lighting fixture. For example, if there is only one floor lamp based on the information from bridge 17, and the physical lighting fixture detected in the image is a floor lamp based on the analysis of the image, then a match exists. This is particularly useful for unique lamp types.

[0046] Another way to identify physical lighting devices detected in an image without knowing their location is by using visible light communication (VLC), which allows the lighting device's identifier to be directly transmitted to the AR application.

[0047] Once the user adds a virtual lighting device to the application, the application can begin checking for the presence of any physical devices in the captured images (i.e., in the AR view). This check can be repeated for each image or every few images. If a physical lighting device 31 is detected in the image, the virtual representation of the virtual lighting effect can be adjusted, and / or the light settings of the physical lighting device 31 can be adjusted.

[0048] For example, if no physical lighting device is detected in the image, the virtual lighting device can present the user-selected virtual light settings without any adjustments, thereby producing a realistic presentation. And once a physical lighting device 31 is detected in the image and the representation of the physical lighting effect is oversaturated, the saturation of the virtual representation of the virtual lighting effect can also be increased, thereby producing an exaggerated (matching) presentation.

[0049] Alternatively, once physical lighting device 31 is detected in the image and the representation of physical lighting effect is oversaturated, the light settings of physical lighting device 31 can be adjusted so that the representation of physical lighting effect in the image looks like the physical lighting effect that the user would have observed if the user had directly viewed physical lighting device 31 before the light settings of physical lighting device 31 were adjusted, or if the user would have directly viewed physical lighting device 31, thereby producing a realistic (matching) presentation.

[0050] Alternatively, both the virtual representation of the virtual lighting effect and the light settings of the physical lighting device 31 can be adjusted to create a presentation that falls between realism and exaggeration. Alternatively or additionally, once the physical lighting device 31 is detected in the image, the light settings of the physical lighting device 31 can be adjusted based on the virtual lighting effect to match the virtual representation of the virtual lighting effect with the representation of the physical lighting effect in the image, for example, making the virtual lighting effect and the physical lighting effect appear to have the same or similar colors in the AR view.

[0051] Physical lighting devices detected in an image can exhibit static or dynamic lighting effects. If a physical lighting device exhibits dynamic lighting effects, its light settings can be adjusted to reduce brightness variations and / or slow down the lighting effect. This makes it easier to identify the differences between physical dynamic lighting effects at different times and their representation in the image, and to match virtual lighting effects with physical lighting effects.

[0052] For example, the detection of physical lighting devices in an image can be performed based on AI object recognition. Once a physical lighting device is detected in the image, the application can poll the bridge 16 to obtain the settings of the physical lighting device (e.g., on / off / brightness / color), which can be used to determine the amount of oversaturation that will result from using the camera 8 and display 9. The calculation of oversaturation can be done using AI algorithms and can depend on the type of physical lighting device detected in the image. If the virtual lighting device and the physical lighting device are of the same type, it is particularly important to match the virtual representation of the virtual lighting effect with the representation of the physical lighting effect in the image.

[0053] Figure 2 It shows the result of Figure 1 An example of an image displayed on mobile device 1. Figure 2 The image captured by the camera 8 of the mobile device 1, shown in the diagram, represents a physical lighting device 31. The physical lighting device 31 presents a physical lighting effect 36. The physical lighting effect 36 is represented by a representation 42 in the image displayed on the display 9. For example, the color of the representation 42 may be oversaturated compared to the way the color of the physical lighting effect 36 is presented directly to the user.

[0054] A virtual representation 52, overlaid on the image and rendered by a virtual lighting device, is displayed on monitor 9. Figure 2 In the example, a virtual representation 51 of the virtual lighting device is also overlaid on the image.

[0055] Since physical lighting device 31 is detected in the image, a virtual representation 52 of virtual lighting effect is determined based on the representation of physical lighting effect in the image, and / or the light settings of physical lighting device 31 are adjusted based on the representation of physical lighting effect in the image and / or the virtual lighting effect. As a result, the virtual representation 52 of virtual lighting effect matches the representation of physical lighting effect in the image, for example, having the same color and / or the same lighting effect size.

[0056] When the light settings of the physical lighting device 31 are adjusted based on the representation of physical lighting effects in the image, the following behavior can be observed. When the physical lighting device 31 enters the field of view of the camera 8, the virtual representation 42 may initially display a different lighting effect than expected, but after the light settings of the physical lighting device 31 have been adjusted, the virtual representation 42 in subsequent images will display the desired lighting effect. The user can configure the exact behavior of the system, for example, via the configuration menu.

[0057] Processor 5 can be configured to determine whether another physical lighting device is visible to the user but not detected in the image, and if it is determined that another physical lighting device (e.g., physical lighting device 32) is visible to the user but not detected in the image, adjust the light settings of the physical lighting device based on at least one of the representation of physical lighting effects in the image and virtual lighting effects. This is particularly useful if processor 5 is unable to control another physical lighting device located outside the AR field of view and therefore cannot adjust the light settings of that other physical lighting device.

[0058] An application that can run on mobile device 1 can detect the presence and status (e.g., current light settings) of physical lighting devices that are not in the camera's field of view but may be in the user's field of view. As described above, the application can access all lighting devices and their current status (e.g., current light settings), and the application can be able to obtain the location of the lighting devices in the applicable space (e.g., home or office) for example from bridge 17 or internet server 13 or by means of cameras (multiple) 19.

[0059] In an alternative implementation, the mobile device 1 has multiple cameras, wherein the displayed image is captured by a feed from a smaller-angle camera, and a wider-angle camera is used to capture the area surrounding the area captured by the smaller-angle camera, i.e., the area outside the AR field of view.

[0060] exist Figure 1 In the embodiment of the mobile device 1 shown, the mobile device 1 includes a processor 5. In alternative embodiments, the mobile device 1 includes multiple processors. The processor 5 of the mobile device 1 may be a general-purpose processor, such as a processor from ARM or Qualcomm, or a dedicated processor. The processor 5 of the mobile device 1 may, for example, run the Android or iOS operating system. The camera 8 may, for example, include a CMOS or CCD sensor. The display 9 may, for example, include an LCD or OLED display panel. The memory 7 may include one or more storage units. The memory 7 may, for example, include solid-state memory.

[0061] Receiver 3 and transmitter 4 can communicate with wireless LAN access point 17 using one or more wireless communication technologies (e.g., Wi-Fi (IEEE 802.11)). In alternative embodiments, multiple receivers and / or multiple transmitters are used instead of a single receiver and a single transmitter. Figure 1 In the illustrated embodiment, a separate receiver and a separate transmitter are used. In an alternative embodiment, receiver 3 and transmitter 4 are combined into a transceiver. The mobile device 1 may include other components typically used in mobile devices, such as a battery and power connector.

[0062] Processor 5 can be configured to run appropriately programmed computer program products (e.g., including those for executing...). Figure 3 The computer program product (computer program code) is configured in the manner described above.

[0063] A first embodiment of a method for displaying a virtual representation of virtual lighting effects rendered by virtual lighting devices is shown in Figure 3 In the middle. This method can, for example, be derived from... Figure 1 The mobile device 1 executes.

[0064] Step 101 includes obtaining an image captured by at least one camera. Step 103 includes detecting the presence of a physical lighting device that renders a physical lighting effect in the image obtained in step 101. Step 105 includes determining a virtual representation of a virtual lighting effect rendered by a virtual lighting device. Optional step 107 includes adjusting the light settings of the physical lighting device.

[0065] Step 109 includes displaying the image obtained in step 101 and a virtual representation of the virtual lighting effect determined in step 105. This virtual representation of the lighting effect is overlaid on the image. Optionally, step 109 also includes displaying a virtual representation of a virtual lighting device overlaid on the image.

[0066] In step 105, a virtual representation of the virtual lighting effect is determined based on the representation of the physical lighting effect in the image, and / or in step 107, the light settings of the physical lighting device are adjusted based on at least one of the representation of the physical lighting effect in the image and the virtual lighting effect. Furthermore, Figure 4 Step 121 Figure 5 Steps 131-135 Figure 7 Steps 151-155 (or Figure 8 Steps 161 and 155) Figure 9 Steps 181-189 and / or Figure 10 Steps 191-193 can be added Figure 3 In the implementation method.

[0067] A second embodiment of the method for displaying a virtual representation of virtual lighting effects presented by a virtual lighting device is shown in Figure 4 middle. Figure 4 The implementation method is Figure 3 An extension of the implementation method. In Figure 4 In this implementation, step 121 is performed between step 103 and step 105.

[0068] Step 121 includes identifying the difference between the physical lighting effect and its representation in the image. For example, step 121 may include: obtaining, for instance, characteristics of at least one camera and / or at least one display from an internet server based on the model of the electronic device, or from the memory of the electronic device, and identifying the difference based on these characteristics. Alternatively, for instance, step 121 may include: obtaining parameter values ​​for calculating the difference from the memory of the electronic device, or from an internet server based on the model of the electronic device, and calculating the difference based on these parameter values.

[0069] When determining the difference between physical lighting effects and their representation in an image, consider whether the camera supports HDR (High Dynamic Range). HDR technology is implemented in many modern smartphones and cameras. HDR works by rapidly taking multiple photos at different exposure levels (typically high, medium, and low exposure) and then combining these images into a single photograph. This process enhances the overall dynamic range of the photo, meaning it can display more detail in both the darkest and brightest areas of the image. When the captured image displayed in an AR view is an HDR image, the physical lighting effects in the image will typically exhibit less overexposure.

[0070] Based on the differences identified in step 121, a virtual representation of the virtual lighting effect is determined in step 105, and / or the light settings of the physical lighting equipment are adjusted in step 107. Furthermore, Figure 5 Steps 131-135 Figure 7 Steps 151-155 (or Figure 8 Steps 155 and 161) Figure 9 Steps 181-189 and / or Figure 10 Steps 191-193 can be added Figure 4 In the implementation method.

[0071] A third embodiment of the method for displaying a virtual representation of virtual lighting effects presented by a virtual lighting device is shown in Figure 5 middle. Figure 5 The implementation method is Figure 3 An extension of the implementation method. In Figure 5 In the implementation method, Figure 3Step 107 is implemented by step 135, and step 133 is performed between step 105 and step 135.

[0072] Step 133 includes determining whether another physical lighting device is visible to the user but not detected in the image. When the user is wearing a head-mounted device, a lighting device not detected in the image will not be visible to the user. However, for smartphones or tablets, a physical lighting device may not be represented in the AR view but is still directly visible to the user, i.e., outside the AR view. If another physical lighting device is determined to be visible to the user but not detected in the image, then in step 135, the light settings of the physical lighting device are adjusted based on at least one of the representation of physical lighting effects in the image and virtual lighting effects.

[0073] If another physical lighting device is not detected in the image but is directly visible to the user, the virtual representation of the virtual lighting effect, determined based on the representation of the physical lighting effect in the image, may create a mismatch between the lighting effect observed by the user in the AR view and the lighting effect observed by the user outside the AR view. Therefore, it is beneficial to adjust the light settings of the physical lighting device in this case.

[0074] Optional step 131 may be performed prior to step 105. Step 131 includes identifying the physical light settings of another physical lighting device. These physical light settings may be obtained from the other physical lighting device or from a lighting controller (e.g., a bridge), or identified by analyzing camera images, such as those from a wide-angle camera. For example, virtual lighting effects may be determined in step 105 based on the other physical light settings identified in step 131.

[0075] If step 131 is performed and it is determined that another physical lighting device is visible to the user but not detected in the image, the light settings of the physical lighting device can be adjusted in step 135 based on the representation of the physical lighting effect in the image and the other physical lighting settings identified in step 131. This can be done, for example, to match the color of the representation of the physical lighting effect in the image with the color of the physical lighting effect presented by the other physical lighting device. Furthermore, Figure 4 Step 121 Figure 7 Steps 151-155 (or Figure 8 Steps 161 and 155) Figure 9 Steps 181-189 and / or Figure 10 Steps 191-193 can be added Figure 5 In the implementation method.

[0076] Figure 6 This illustrates when the mobile device's processor is configured to execute... Figure 5 The method is by Figure 1 An example of an image displayed on mobile device 1. For example... Figure 2As in the example, a physical lighting device 31 is represented in the image. The physical lighting device 31 presents a physical lighting effect 37. The physical lighting effect 37 is represented by a representation 43 in the image displayed on the display 9. For example, the color of the representation 43 may be oversaturated compared to the way the color of the physical lighting effect 37 is presented directly to the user.

[0077] A virtual representation 53, overlaid on the image and rendered by a virtual lighting device, is displayed on monitor 9. Figure 6 In the example, a virtual representation 51 of the virtual lighting equipment is also overlaid on the image. Figure 2 Unlike the previous example, another physical lighting device 32 is also visible to the user of the mobile device 1, but is not represented in the image displayed on the display 9. The physical lighting device 32 presents physical lighting effect 39.

[0078] Since physical lighting device 31 is detected in the image, and another physical lighting device 32 is identified as visible to the user but not detected in the image, the light settings of physical lighting device 31 are adjusted based on the representation of physical lighting effects and / or virtual lighting effects in the image. Figure 5 (in step 135).

[0079] By adjusting the light settings of the physical lighting device 31 based on the physical light effect representation 43, any difference between the appearance of the light effect directly presented to the user by the lighting device 31 and the appearance of the light effect in the image is compensated, thereby resulting in a match between the appearance of the physical light effect 37 representation 43 in the image and the appearance of the virtual light effect representation 53.

[0080] In this case, it is insufficient to determine the virtual representation 53 of the virtual lighting effect based on the representation of the physical lighting effect 37 in the image, because there will be a mismatch between the lighting effect represented in the image and the physical lighting effect 39 presented by the physical lighting device 32 and directly visible to the user.

[0081] A fourth embodiment of the method for displaying a virtual representation of virtual lighting effects presented by a virtual lighting device is shown in Figure 7 middle. Figure 7 The implementation method is Figure 3 An extension of the implementation method. In Figure 7 In the implementation of the method, step 105 is performed by step 155, and steps 151 and 153 are executed between steps 103 and 155.

[0082] Step 151 includes receiving the physical light settings of the physical lighting device. Step 153 includes obtaining the virtual light settings of the virtual lighting device by determining virtual light settings based on the physical light settings received in step 151. Step 155 includes determining a virtual lighting effect based on the virtual light settings obtained in step 153. Furthermore, Figure 4 Step 121 Figure 5 Steps 131-135 Figure 9 Steps 181-189 and / or Figure 10 Steps 191-193 can be added Figure 7 In the implementation method.

[0083] A fifth embodiment of the method for displaying a virtual representation of virtual lighting effects presented by a virtual lighting device is shown in Figure 8 middle. Figure 8 The implementation method is Figure 3 An extension of the implementation method. In Figure 8 In the implementation of this method, step 105 is performed by step 155, and step 161 is executed between step 103 and step 155.

[0084] Step 161 includes obtaining the virtual light settings of the virtual lighting device by receiving a user input signal indicating virtual light settings. Step 155 includes determining the virtual lighting effect based on the virtual light settings obtained in step 161. Furthermore, Figure 4 Step 121 Figure 5 Steps 131-135 Figure 9 Steps 181-189 and / or Figure 10 Steps 191-193 can be added Figure 8 In the implementation method.

[0085] A sixth embodiment of a method for displaying a virtual representation of virtual lighting effects rendered by virtual lighting devices is shown in Figure 9 middle. Figure 9 The implementation method is Figure 3 An extension of the implementation method. Step 101 includes obtaining an image captured by at least one camera. Step 103 includes detecting the presence of a physical lighting device exhibiting physical lighting effects in the image obtained in step 101.

[0086] Step 181 includes determining the current activity mode. Step 183 includes checking whether the current activity mode is a first mode (mode 1) or a second mode (mode 2). For example, the first mode could be an exaggerated mode, and the second mode could be a realistic mode. If the current activity mode is the first mode, proceed to step 185. If the current activity mode is the second mode, proceed to step 187.

[0087] Step 185 includes determining a virtual representation of the virtual lighting effect based on the representation of the physical lighting effect in the image. Step 187 includes determining a virtual representation of the virtual lighting effect not based on, or only slightly based on, the representation of the physical lighting effect in the image. Step 189 is performed after step 187. Step 189 includes adjusting the light settings of the physical lighting device based on at least one of the representation of the physical lighting effect in the image and the virtual lighting effect.

[0088] For example, in the first mode (step 185), the effect represented in the AR view can focus on a higher, more exaggerated effect, which can be used to preview upcoming products (i.e., products not yet in stores). In the second mode (steps 187 and 189), the effect represented in the AR view can focus on a realistic lighting effect, which can be used to preview lighting fixtures when selecting and purchasing them. In both modes, the virtual lighting effects in the AR view are matched with the physical lighting effects, making them similar but in different ways.

[0089] Step 109 includes displaying the image obtained in step 101 and a virtual representation of the virtual lighting effect determined in step 185 or 187. This virtual representation of the lighting effect is overlaid on the image. Furthermore, Figure 4 Step 121 Figure 5 Steps 131-135 Figure 7 Steps 151-155 (or Figure 8 Steps 161 and 155) and / or Figure 10 Steps 191-193 can be added Figure 9 In the implementation method.

[0090] A seventh embodiment of a method for displaying a virtual representation of virtual lighting effects rendered by virtual lighting devices is shown in Figure 10 middle. Figure 10 The implementation method is Figure 3 An extension of the implementation method. In Figure 10 In the implementation of this method, step 105 is performed by step 193, and step 191 is performed between step 103 and step 193.

[0091] Step 191 includes obtaining information indicating the luminous effect of the physical lighting equipment. Step 193 includes determining a virtual representation of the virtual lighting effect based on the luminous effect of the physical lighting equipment indicated in the information obtained in step 191. Thus, the luminous effect of the virtual lighting equipment is adjusted to match the luminous effect of the physical lighting equipment. Furthermore, Figure 4 Step 121 Figure 5 Steps 131-135 Figure 7 Steps 151-155 (or Figure 8 Steps 161 and 155) and / or Figure 9 Steps 181-189 can be added Figure 10 In the implementation method.

[0092] Figure 11 The description indicates that the procedure can be performed as shown in the reference. Figures 3 to 5 and Figures 7 to 10 A block diagram of an exemplary data processing system for the described method.

[0093] like Figure 11 As shown, the data processing system 300 may include at least one processor 302 coupled to a memory element 304 via a system bus 306. Thus, the data processing system can store program code within the memory element 304. Furthermore, the processor 302 can execute program code accessed from the memory element 304 via the system bus 306. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program code. However, it should be understood that the data processing system 300 may be implemented in the form of any system including a processor and memory capable of performing the functions described herein. For example, the data processing system may be an Internet / cloud server.

[0094] Memory element 304 may include one or more physical memory devices, such as, for example, local memory 308 and one or more mass storage devices 310. Local memory may refer to random access memory or (multiple) other non-persistent memory devices generally used during the actual execution of program code. Mass storage devices may be implemented as hard disk drives or other persistent data storage devices. Processing system 300 may also include one or more cache memories (not shown) that provide temporary storage for at least some program code to reduce the number of times program code must be retrieved from mass storage device 310 during execution. For example, if processing system 300 is part of a cloud computing platform, processing system 300 may also be able to use memory elements of another processing system.

[0095] Optionally, the input / output (I / O) devices depicted as input device 312 and output device 314 can be coupled to the data processing system. Examples of input devices may include, but are not limited to, a keyboard, a pointing device such as a mouse, a microphone (e.g., for sound and / or speech recognition), etc. Examples of output devices may include, but are not limited to, a monitor or display, a speaker, etc. The input and / or output devices can be coupled to the data processing system directly or through an intermediate I / O controller.

[0096] Input and output devices can be implemented as a combined input / output device (in... Figure 11(Dashed lines are used to illustrate input device 312 and output device 314). An example of such a combined device is a touch-sensitive display, sometimes also called a "touchscreen display" or simply a "touchscreen". In such an implementation, input to the device can be provided by the movement of a physical object, such as a stylus or a user's finger, on or near the touchscreen display.

[0097] Network adapter 316 can also be coupled to the data processing system to enable it to be coupled to other systems, computer systems, remote network devices, and / or remote storage devices via an intermediate private or public network. The network adapter may include a data receiver for receiving data transmitted to the data processing system 300 from the system, devices, and / or network, and a data transmitter for transmitting data from the data processing system 300 to the system, devices, and / or network. Modems, cable modems, and Ethernet cards are examples of different types of network adapters that can be used with the data processing system 300.

[0098] like Figure 11 As illustrated, memory element 304 can store application program 318. Application program 318 can be stored in local memory 308, one or more mass storage devices 310, or separately from both local memory and mass storage devices. It should be understood that data processing system 300 can further execute an operating system that facilitates the execution of application program 318. Figure 11 (Not shown in the image). The application program 318, implemented in the form of executable program code, can be executed by the data processing system 300 (e.g., by the processor 302). In response to executing the application program, the data processing system 300 can be configured to perform one or more operational or method steps described herein.

[0099] This invention can be implemented as a program product for use with a computer system, wherein the program product(s) define functions. The program(s) can be contained on a variety of non-transitory computer-readable storage media, wherein, as used herein, the expression “non-transitory computer-readable storage media” includes all computer-readable media, with the sole exception of transient propagation signals. The program(s) can also be contained on a variety of transient computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media on which information is permanently stored (e.g., read-only memory devices within a computer, such as CD-ROM discs readable by a CD-ROM drive, ROM chips, or any type of solid-state non-volatile semiconductor memory); and (ii) writable storage media on which changeable information is stored (e.g., flash memory, floppy disks within a floppy disk drive or hard disk drive, or any type of solid-state random access semiconductor memory). The computer program can run on the processor 302 described herein.

[0100] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0101] The corresponding structures, materials, actions, and equivalents of all means or steps plus functional elements in the following claims are intended to include any structure, material, or action for performing a function in combination with other claimed elements as specifically claimed. Specific embodiments have been shown for illustrative purposes but are not intended to be exhaustive or limited to the embodiments in the disclosed forms. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention.

Claims

1. An electronic device (1) for displaying a virtual representation (52) of virtual lighting effects rendered by a virtual lighting device, the electronic device (1) comprising: At least one camera (8); At least one display (9); as well as At least one processor (5) is configured as follows: - Obtain images captured by the at least one camera (8), - Detect the presence of a physical lighting device (31) in the image, the physical lighting device (31) exhibiting physical lighting effects, - Determine the virtual representation of the virtual lighting effect presented by the virtual lighting device (52), and - The image and the virtual representation (52) of the virtual light effect are displayed to the user of the electronic device (1) on at least one display (9), the virtual representation of the light effect being overlaid on the image. The at least one processor is further configured to: - Identify the difference between the physical lighting effect and its representation in the image, and - Determine the virtual representation of the virtual light effect (52) based on the difference between the physical light effect and the representation of the physical light effect in the image and / or adjust the light settings of the physical lighting device (31) based on the difference between the physical light effect and the representation of the physical light effect in the image.

2. The electronic device (1) according to any one of the preceding claims, wherein the at least one processor (5) is configured to obtain the virtual light settings of the virtual lighting device and determine the virtual light effect based on the virtual light settings.

3. The electronic device (1) according to claim 2, wherein the at least one processor (5) is configured to receive the physical light settings of the physical lighting device (31) and obtain the virtual light settings of the virtual lighting device by determining the virtual light settings based on the physical light settings.

4. The electronic device (1) according to claim 2, wherein the at least one processor (5) is configured to obtain the virtual light settings of the virtual lighting device by receiving a user input signal instructing the virtual light settings.

5. The electronic device (1) according to any one of the preceding claims, wherein the at least one processor (5) is configured to display a virtual representation (51) of the virtual lighting device overlaid on the image on the at least one display (9).

6. The electronic device (1) according to any one of the preceding claims, wherein the at least one processor (5) is configured to: - Determine the current activity mode, and - If the current activity mode is the first mode, the virtual representation (52) of the virtual light effect is determined based on the representation of the physical light effect in the image, and if the current activity mode is the second mode, the light settings of the physical lighting device (31) are adjusted based on at least one of the representation of the physical light effect in the image and the virtual light effect.

7. The electronic device (1) according to any one of the preceding claims, wherein the at least one processor (5) is configured to: - Obtain information indicating the luminous effect of the physical lighting device (31), and - The virtual representation (52) of the virtual light effect is determined based on the luminous effect of the physical lighting device (31).

8. A method for displaying a virtual representation of virtual lighting effects rendered by a virtual lighting device, the method comprising: - Obtain (101) images captured by at least one camera; -Detect (103) that there is a physical lighting device in the image, and the physical lighting device presents a physical lighting effect; - Determine (105) the virtual representation of the virtual lighting effect presented by the virtual lighting device; and - Display (109) the image and the virtual representation of the virtual light effect, the virtual representation of the light effect overlaid on the image, - Identify the difference between the physical lighting effect and the representation of the physical lighting effect in the image. The virtual representation of the virtual lighting effect is determined based on the difference between the physical lighting effect and the representation of the physical lighting effect in the image, and / or the light settings of the physical lighting device are adjusted based on the difference between the physical lighting effect and the representation of the physical lighting effect in the image.

9. A computer program product for a computing device, the computer program product comprising computer program code for executing the method of claim 8 when the computer program product is run on a processing unit of the computing device.

Images