Lighting control device, lighting arrangement and method for controlling operation thereof

Abstract

The invention is directed to a lighting control device for controlling operation of lighting devices (202, 203, 204) located in an environment (206), comprising an input data ascertaining unit (102) configured to ascertain user input data (UD) that is correlatable to a lighting scene to be applied, environment data (ED) indicative of an spatial configuration of the environment and lighting device data (LD) indicative of operational capabilities of the lighting devices. An image data generation unit (104) is configured to generate image data (ID) indicative of an image based on the ascertained data and a light setting determination unit (106) is configured to determine respective light settings indicative of lighting operation parameters for the lighting devices and to provide control signals (CS) indicative of the respective light settings for controlling the lighting devices.

Description

[0001] Lighting control device, lighting arrangement and method for controlling operation thereof

[0002] FIELD OF THE INVENTION

[0003] The present invention is directed to a lighting control device for controlling operation of a lighting arrangement including one or more lighting devices located in an environment. The invention is further directed to a lighting arrangement including said lighting control device and one or more lighting devices controllable by said lighting control device. The invention is further directed to a computer program.

[0004] BACKGROUND OF THE INVENTION

[0005] Document WO2021 / 144232 Al describes a controller or control device and a method for generating light settings for a plurality of lighting units or lighting devices. The method comprises obtaining images, extracting a plurality of colors from the image, selecting a subset of colors and generating light settings based on the selected subset of colors.

[0006] Regarding the control of a lighting arrangement, there have been some efforts to control luminaires using natural language processing (NLP). Voice-activated smart home systems like Amazon’s Alexa, Google Home, and Apple’s Siri utilize NLP to understand and execute user commands. However, these can process simple tasks such as turning light on / off or dimming the light. Recently, to accommodate more specific or complex demands from users, custom NLP solutions are developed by training a machine learning model. However, to guarantee the quality and stability of the control, this approach requires a huge amount of dataset for training and the model needs to be updated regularly.

[0007] SUMMARY OF THE INVENTION

[0008] It would be beneficial to enable a more accurate control of the lighting settings that also is better adapted to the user’s environment.

[0009] According to a first aspect of the invention, a lighting control device is disclosed. The lighting control device is configured to control operation of a lighting arrangement, which is an arrangement that includes one or more lighting devices that are located in a given environment, such as a room, a warehouse, a street, a parking lot, etc. The lighting control device comprises an input data ascertaining unit that is configured to ascertain user input data that is correlatable to a lighting scene to be applied to the environment and lighting device data indicative of operational capabilities of the respective lighting devices. The input data ascertaining unit may be configured to ascertain environment data indicative of a spatial configuration of the environment.

[0010] The lighting control device also comprises an image data generation unit that is configured to generate image data indicative of an image based on the ascertained user input data and lighting device data. The image data generation unit may be configured to generate image data indicative of an image based on the ascertained user input data, environment data and lighting device data.

[0011] The lighting control device further comprises a light setting determination unit that is configured to, using the generated image data, extract color data from the image data, determine respective light settings indicative of lighting operation parameters for the one or more lighting devices using the color data, and to provide control signals indicative of the respective light settings for controlling the lighting devices.

[0012] According to the invention, the determination of the light settings for the lighting device of the arrangement is based on user input data which is correlatable or can be associated to a lighting scene or lighting configuration to be applied within the environment and also on the spatial configuration of the environment where the lighting devices are located. The spatial configuration of the environment is extracted from the ascertained environment data, which is preferably associated to a particular point-of-view of the user. The user input data is thus data provided by the user and that serves as a starting point for the determination of the light settings. The light settings are also tailored in dependence on the operational capabilities of the different lighting devices, as given by the, such that no lighting device is forced to operate under conditions that are not realistic or available for said lighting device (e.g., a color temperature of 2700K for a lighting device that has an operation range in terms of color temperature between 4000K and 5000K).

[0013] The user input data is interpreted as a suggestion, vague or explicit, of a particular lighting scene that is not necessarily pre-specified. A lighting scene is defined as a set of operation conditions of one or more lighting devices, wherein at least one of the lighting devices has controllable lighting parameters, for example in terms of, for example, color spectrum, and / or light intensity, and / or color temperature, and / or direction of illumination, and / or light beam forming, etc. The respective controllable parameters of the lighting devices are given by the lighting devices data. The lighting scene can be a static scene, or a dynamic scene, wherein the operation parameters of at least one of the lighting device changes with time. A lighting scene refers to the result of combining the different light settings applied to the different lighting devices.

[0014] The image data from which the color data is extracted, which is then used to determine the light settings for the lighting devices, is generated based on the environment data, and thus depends on the spatial configuration of the environment wherein the lighting devices are placed. The color data includes information regarding the colors that are present in the image data.

[0015] This enables a more realistic rendering of the user’s intent, as indicated by the user input data, via the lighting units or lighting devices. Further it enables a more personalized use of smart lighting given the operational capabilities of the lighting devices and the spatial layout of the environment where the lighting devices are operating.

[0016] In the following, embodiments of the lighting control device of the first aspect of the invention will be disclosed.

[0017] In an embodiment, the user input data ascertained by the input data ascertaining unit comprises text data and / or image data and / or video data and / or audio data. For instance the input user data can include a text file including text data indicative of the user’s intent. Additionally, or alternatively, the user input data can be an image file or a video file comprising image or video data from which a user intent can be extrapolated. Further, the user input data can comprise an audio file including audio data, showing also a user’s intent or mood related to which the lighting units or lighting devices are to be operated. For instance, a text file including the text “I am happy and I feel like flying” can be provided as user input data, and suitable light settings for the lighting devices will be provided based on said input. The user input data can be also an image of video file, for instance of a party at sunset by the beach, or an audio file, also indicative of a current mood or intent of the user upon which the lighting. The audio file can be converted into a text file or provided as is. The same applies to the image or video file.

[0018] Preferably, in an embodiment, the environment data is determined from environment image data, which can be provided by the user, in particular in the form of a point-of-view image of the environment data is indicative of object parameters indicative of object properties and / or object location of the objects located in the environment and / or delimiting the environment. The environment data can further comprise context awareness sensing data, for example indicative of how many people are present and / or where they are present in the environment, and, optionally also of a respective field-of-view. Additionally, or alternatively, the environment data can also be indicative of a current activity carried out by one or more of the people in the environment, or of an interaction between two or more people, such as fighting, hugging, etc. Here, the image data is further generated based on the object parameters. The environment data can be obtained using image data or video data representing the environment. Alternatively, the environment data can be obtained using audio data of text data describing the spatial configuration of the environment. Preferably, the image data is indicative of a current point-of-view of a user in the environment. The object parameters refer to properties and / or location of different objects in the environment or delimiting the environment. The objects can therefore include walls, ceilings, floors, furniture, doors, windows, plants. The objects can also include lighting devices, in particular the lighting devices whose operation is to be controlled based on the user input data. The object properties are preferably indicative of a color, size, reflectivity, texture (e.g. hard / soft), material, etc., of the respective object. The object location refers to a location of the object in the environment and / or to a relative location between two or more objects. For instance, the environment data can be indicative of a ceiling height or a room surface. The environment data is preferably indicative of a color distribution of the objects located in the environment. The environment data is preferably indicative of object parameters that are indicative of object location of at least one of the lighting devices of the lighting arrangement located in the environment, and wherein the image data is generated based on the object parameters. In other words, in a preferred embodiment, at least one of the objects for which the object parameters are extracted is a lighting device of the lighting arrangement. The environment data is preferably indicative of the position of the lighting devices that are identifiable in the environment, for instance in an image of the environment, more particularly, a point-of-view image of the user.

[0019] Additionally, or alternatively, the environment data can be indicative of subject parameters indicative of subject properties and / or subject location of the subjects located in the environment. The subject properties can be based on vital sign or activity data indicative of the user and / or of the people located in the environment. The vital sign or activity data can be indicative of an emotional state of the subject (e.g., happy, sad, etc.), or sleepiness state of the subject.

[0020] Additionally, or alternatively, the environment data can be include environmental parameters indicative of environmental conditions in the environment, including, but not limited to, a temperature in the environment, a humidity in the environment, weather conditions in the environment, a time of day, etc. The image data is preferably generated using the object parameters and / or on the subject parameters, and / or environment parameters. For instance if the environment parameters are indicative of hot and humid conditions in the environment, the generated image can be generated to be indicative of a rainforest before a thunderstorm.

[0021] In an embodiment, the environment data can be provided as a depth image data, for instance as a 3D point cloud data of a scanned environment or room using LiDAR, or as a neural radiance field (NeRF).

[0022] The lighting device data can be for instance stored in a memory unit accessible by the lighting control device or integrated with it. The lighting device data pertaining to the lighting devices of the lighting devices can be transferred or generated during a commissioning process of the lighting arrangement. If new lighting devices are added to the arrangement the lighting device data is updated. Additionally, or alternatively, the lighting device data can be determined using the environment data, when the environment data is indicative of at least some of the lighting devices. For instance, environment data in the form of an image can include the image representation of one or more lighting devices of the lighting arrangement, such as, for example, a wall sconce fixed to a wall and a pendant luminaire fixed to the ceiling. The lighting device data can be advantageously used to restrict which colors the image data can include. For instance, lighting devices typically cannot generate brown light (as it is a subtractive color) and hence the image generator unit is preferably configured not use browns in the image data. Similarly, an RGB light cannot generate all the colors visible to the human eye.

[0023] Preferably, in an embodiment, the image data generation unit that is configured to generate image data indicative of an image based on the ascertained user input data, environment data and lighting device data comprises or is based on a generative artificial intelligence (Al) model. The use of pre-trained large language models (LLM) enable the use of LLM models without the need of training with a large amount of dataset. Thus, in this embodiment LLM models for Generative Al are used for determining a tentative guess of the user’s intention from a the user input data, in particular in the form of a textual description, and generating an associated image, characterized or defined by respective image data, to control light elements considering the operational capabilities of the lighting device, as given by the ascertained lighting device data, as well as the spatial placement of different lighting devices and objects (furniture, plants, etc.) in the environment, as indicated by the environment data. The Al generated image data serves thus as an intermediatory from the user’s input data to the actuation of the lighting devices with a color palette / brightness matching the user input data. There is thus no need to further train the generative Al model with additional sets of training data.

[0024] Thus, in an embodiment, the lighting control device advantageously comprises a prompt generation unit that is configured to receive the user input data, the environment data and the lighting device data, to generate a prompt based on the user input data, the environment data and the lighting device data and to provide said prompt to the image data generation unit. The prompt can be a natural language description. The prompt can be a multimodal prompt. In the case of a natural language description, the prompt includes the user input data if this is provided as text data, or, for instance a text conversion of an audio file or a textual description of an image or video file, which can be generated by a suitable Al tool such as Vision Al that generates captions an image descriptions. The prompt with the user input data, preferably in text form is augmented with information about the location and capabilities of the available light source, in particular based on the environment data, which preferably is indicative of the user’s current field of view. The prompt is also augmented with object information regarding the objects located in the environment.

[0025] Thus, in a preferred embodiment, the image data generation unit is configured to provide the received prompt to a machine learning model, for example configured as a text-to-image model or as a multimodal model (e.g. Google Gemini), that is configured to generate the image data based on the prompt.

[0026] Additionally, the machine learning model can be configured to generate, based on the received prompt, an additional complementary output in the form of an audio file, and / or to control a smell dispenser for providing a predetermined scent, and / or to control a tactile feedback unit. These additional complementary outputs are associated to the image data. For instance, if the image data represents or includes a beach, the audio file can be indicative of a wave sound. If the image data represents a flower field, a smell dispenser can be controlled to provide a suitable related scent. If the image data is indicative of a train, a tactile feedback unit, for instance located in an armchair, can be controlled to provide vibrations similar to those of a train, to name a few illustrative examples.

[0027] Preferably, the machine learning model is based on a diffusion model. Among the several algorithmic approaches to generate image from text, an approach based on a diffusion model is preferred. Other alternative approaches include, for example, generative adversarial networks or GANs or a decoder of a variable auto-encoder model. In the case of diffusion model, in particular, a foundational model, such as, but not limited to, Stable Diffusion vl.4 can be leveraged. The architecture of the diffusion model can be modified and fine-tuned for this particular application. In particular, contextual parameters can be introduced. With these, the diffusion process can be conditioned to generate an image that accurately captures the scene that the user would like to render on the given environment and lights.

[0028] For example, a first approach can be directed to training a conditional diffusion model to generate the desired image from a supervised dataset. If a labeled dataset is available, synthetic data can be generated. Another applicable conditioning method involves adapting pre-trained unconditional diffusion models to new conditions using the learned internal representations of a denoiser network. This approach is effective for various conditional generation tasks, including attribute-conditioned generation and mask- conditioned generation. By using the unconditional denoiser network as a feature extractor, guidance can be provided that is robust to the initial inaccurate estimates of xO (i.e., original input data), and quickly leam the guidance directions from a small set of labeled samples. The denoiser network is trained with varying noise levels in its inputs and has learned to extract features of different scales at different time steps. The conditioning on the image process is done to accommodate the spatial properties of the environment, the properties of the lighting devices, and the preferences of the user indicated in the user input data. If required, training can be done for the desired properties.

[0029] The augmented prompt that includes the user input data, alongside with the augmentation in terms of device capabilities and object properties and location is submitted or provided to the diffusion model and the diffusion model generates image data indicative of a generated image. After the image data is generated, the light setting determination unit is configured to extract color data from the image data. The color data includes information regarding the colors that are present in the image data. The color data may also include brightness data for each color e.g. blue is has a brightness value and the red in the image has a brightness value indicative of a fainter color. The color data can also include color uniformity data indicative of how much of the image is which color; which colors are adjacent to each other in the image and which ones are not; sharp vs. gradual color transition from first region of first color to second region of second color, etc. In an embodiment, this is done using a dedicated module, such as a hue color extraction module that is capable of extracting a color palette from an image. Also, the light setting determination unit is configured to determine the palette to be displayed or rendered by the different lighting devices in the environment. Alternatively, a visual question answering module (VQA) can be used to extract the color for each lighting device the generated image. In a further developed embodiment, instead of 2D image data, a 3D intermediary image comprising 3D image data can be generated. The prompt is feed into the text-to-image generation model, such as stable diffusion models or DALL-E, to produce a 3D image or model. The generated image is evaluated to determine if the generated image contains the content that satisfies the requirements by applying techniques like visual question answering (VQA). Then the color is extracted for each lighting device from the generated image.

[0030] Regardless of whether the image data is representative of a 2D or a 3D image, the light setting determination is then configured to generate and provide the necessary control instructions for operating the lighting devices in accordance with the light setting.

[0031] In another embodiment, the lighting control device further comprises a user feedback unit configured to receive user feedback data indicative of a degree of satisfaction with the determined respective light-settings. The user feedback can be for instance in the form of a predetermined voice command or gesture, or can be provided by interacting with a dedicated user interface, for instance an app on a mobile device. In this embodiment, the image data generation unit is configured to generate alternative image data using user input data, environment data and lighting device data upon determining, based on the user feedback data that the degree of satisfaction is below a predetermined threshold. The image data generation unit can generate a completely new image data or only re-generate portions of the image. For instance, if the feedback user is indicative of the fact the user liked a particular aspect of the image but was dissatisfied with another aspect, the image data generation unit can re-generate only that part of the image with which the user was not satisfied. For example, if the user liked the composition of the image but did not like the full moon and would like to have a new moon, the image data generation unit can re-adapt the image data by eliminating the moon.

[0032] Thus, if the user indicates that he or she is not satisfied with the light scene selected and implemented and / or with the image data generated and displayed by providing a corresponding user feedback, the image data generation unit generates a subsequent image data that is provided to the light determination unit to determine the corresponding light settings and to generate and provide the corresponding control signals. The user feedback data may be further indicative of the reasons why the user is dissatisfied and the corresponding prompt for generating the subsequent image data may also include an indication to these reasons. In another embodiment, the lighting control device of any of the preceding claims, further comprising an image providing unit that is configured to provide the image data for displaying a corresponding image to the user. The image providing unit is signally connected to the image data generation unit for receiving the image data and comprises a display for displaying the image generated and based upon which, the light settings have been determined. The image providing unit can be, for instance a television, a smart phone, a tablet, a digital frame, a computer including a monitor, or any other device that can receive image data and display the corresponding image. The image providing unit can also be an augmented reality (AR) headset configured to provide only a portion of the corresponding image to the user, which depends on the actual orientation of the user, as determined by the AR headset. Further, the image providing unit can be configured to overlay one or more salient parts of features of the image onto the view of the environment. For instance, the generated image data can include certain features (e.g., a palm tree, a setting sun), which can be overlaid onto the current view of the room or environment provided by the AR headset.

[0033] In a further developed embodiment, the image providing unit is configured to provide the image data for displaying the image to the user in dependence on a predetermined image content criterion and / or a predetermined image resolution criterion. For instance, if the intermediary image generated by the diffusion model (image data) is a very low resolution image not deemed to be human interpretable (though still useful for the color palette picking algorithm), the image providing unit can be configured to not show the image or the image data generation unit can be configured to not provide the image data for display. Further, depending on how this specific image must be refined before it can be shown to this user, the system decides how many diffusion steps can be skipped.

[0034] Additionally, or alternatively, the image can be displayed or nor in dependence of a riskiness of the intermediary image. If, for example, the user requests a romantic lighting scene, the image providing unit can be configured to not display the intermediary image as the diffusion model may have generated inappropriate content. The guardrails may be conditioned on geographical location or country regulations.

[0035] If it is determined that the image is to be shown to the user, the image data generation unit can be configured to re-generate the image data to with a higher resolution (while for the purpose of extracting the color data and determining the lighting settings for the environment, a low resolution image will suffice). The generation of a higher resolution image can be triggered if the user challenges the automated lighting controls based on the low resolution intermediate image data. The intermediate image is a kind of explainable-AI visualization for the user why the lighting control device has determined the applied light settings.

[0036] A second aspect of the present invention is formed by a lighting arrangement. The lighting arrangement comprises a lighting control device in accordance with the first aspect of the invention and one or more lighting devices that are configured to receive control signals indicative of the respective light settings and to operate in dependence thereof.

[0037] Thus, the lighting arrangement of the second aspect of the invention shares the advantages of the lighting control device of the first aspect.

[0038] In an embodiment wherein the lighting control device further comprises an image providing unit that is configured to provide the image data for displaying a corresponding image to the user, the arrangement further comprises a display device configured to receive the image data and to display the image defined by the image data.

[0039] A third aspect of the present invention is formed by a method for controlling operation of a light control device for controlling operation of a lighting arrangement including one or more lighting devices located in an environment. The method of the third aspect comprises ascertaining user input data that is correlatable to a lighting scene to be applied to the environment; environment data indicative of an spatial configuration of the environment; and lighting device data indicative of operational capabilities of the respective lighting devices.

[0040] The method also comprises generating image data based on the ascertained user input data, environment data and lighting device data; and using the generated image data, extracting color data from the image data, determining respective light settings indicative of lighting operation parameters for the one or more lighting devices using the color data, and providing control signals indicative of the respective light settings for controlling the lighting devices.

[0041] Thus, the method of the third aspect shares the advantages of the lighting control device of the first aspect, or of any of its embodiments.

[0042] In an embodiment of the method, the method further comprises providing the image data indicative of an image for displaying the image to the user.

[0043] According to a fourth aspect of the invention, a method for operating a lighting arrangement is disclosed. The method comprises performing the method of the third aspect of the invention, and operating the lighting devices in dependence on the control signals. A fifth aspect of the invention is formed by a computer program comprising instruction which, when executed by a lighting control device, cause the lighting control device to perform the steps of the method of the third or of the fourth aspect.

[0044] It shall be understood that the lighting control device of claim 1, the lighting arrangement of claim 10, the method for controlling operation of a light control device of claim 12, the method for controlling operation of a lighting arrangement of claim 14 and the computer program of claim 15, have similar and / or identical preferred embodiments, in particular, as defined in the dependent claims.

[0045] It shall be understood that a preferred embodiment of the present invention can also be any combination of the dependent claims or above embodiments with the respective independent claim.

[0046] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0047] BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In the following drawings:

[0049] Fig. 1 shows a schematic block diagram of an embodiment of a lighting arrangement including an exemplary lighting control device in accordance with the invention;

[0050] Fig. 2 shows a schematic block diagram of another embodiment of a lighting arrangement including another exemplary lighting control device in accordance with the invention;

[0051] Fig. 3 shows a schematic block diagram of another embodiment of a lighting arrangement including another exemplary lighting control device in accordance with the invention; and

[0052] Fig. 4 shows a flow diagram of an exemplary method for controlling operation of a lighting arrangement in accordance with the invention.

[0053] DETAILED DESCRIPTION OF EMBODIMENTS

[0054] Fig. 1 shows a schematic block diagram of an embodiment of a lighting arrangement 200 including an exemplary lighting control device 100 in accordance with the invention. The lighting control device 100 is configured to control operation of the lighting devices 202, 203, 204 located in an environment 206, for example, a living room. The lighting control device 100 comprises an input data ascertaining unit 102 that is configured to ascertain user input data UD that is correlatable to a lighting scene to be applied to the environment. The user input data can for instance be a text file, an image file, a video file or an audio file and serves as a starting point for the generation of control signals for controlling operation of the lighting devices. It may be a direct indication of the user’s intention such as “I want to create a light scene based on bear flag of California” or “I want to create a light scene based on the following image” where the user input data also includes the image file the user is referring to, or a more abstract text such as “I want to fly” from which a light scene can be inferred.

[0055] The data ascertainment unit is also configured to ascertain environment data ED indicative of a spatial configuration of the environment 306 where the lighting devices

[0056] 202, 203, 204 are located. Preferably, the environment data ED is received as an image file 103 representing an image of the environment, for example an image of a point-of-view of a user. In particular, the environment data ED is indicative of object parameters indicative of object properties and / or object location of the objects located in the environment and / or delimiting the environment. For instance, in the environment shown in the image, there is a table 206 in the lower central part of the image. The environment data can also be indicative of spatial properties of the environment, such as, but not limited to , size, color and reflections of the walls, ceiling and / or floor of the environment, location of the lighting devices relative to each other and to the user, for instance when the image is a point-of-view image taken by the user.

[0057] The data ascertainment unit is further configured to ascertain lighting device data LD that is indicative of operational capabilities of the respective lighting devices 202,

[0058] 203, 204. The lighting device data LD thus indicates how the different lighting devices can be operated, for instance in terms of intensity ranges, variable color spectrum, color temperature ranges, orientation or directivity of the emitted light, etc. For instance, lighting device 202 is a direct pendant luminaire with a LED light source that can emit white and color light (>16 million colors) with a controllable intensity up to 1100 lumen and a color temperature range between 2000K and 6500K, with a lumen output at 4000K of 1055 lumen and a lumen output at 2700 of 806 lumen. Lighting device 203 is a table top lamp with an LED light source that can emit white light with a controllable intensity up to 800 lumen and a color temperature range from 2200K and 6500K, with a lumen output at 400K of 806 lumen and a lumen output at 2700K of 570 lumen. Lighting device 204 is a stand light with an LED light source that can emit white and color light (>16 million colors) with a controllable intensity up to 470 lumen and a color temperature range between 2000K and 6500K, with a lumen output at 4000K of 470 lumen and a lumen output at 2700 K of 320 lumen. The selection of the parameter values for operation of the lighting device is performed based on control signals that can be provided via a communication channel, which can be wireless or wired. The lighting device data LD can for instance be extracted from the environmental data, e.g., from the image of the environment, for example by performing an object recognition process, for instance based on a machine learning model, on the image data to identify the objects and classify them according to the respective operational capabilities. Additionally, or alternatively, the lighting device data LD pertaining to the lighting devices

[0059] 202, 203, 204 in the environment 206 can be stored in a memory unit 105 accessible by the lighting control device 100. The data ascertaining unit can use both the lighting device data stored in the memory unit 105 and the image 103 of the environment 206 represented by the environment data ED to determine which of the lighting devices are to be controlled. For instance, the memory unit can store lighting device data pertaining to lighting devices installed in different environments (kitchen, living room, master bedroom, etc.) and the actual set of lighting devices to be controlled or the given environment where the lighting devices to be controlled are installed can be extracted from the image 103 by analyzing said image.

[0060] The lighting control device 100 further comprises an image data generation unit 104 that is configured to generate image data ID indicative of an image 107. The generation of the image data is based on the ascertained user input data UD, environment data ED and lighting device data LD of the lighting devices that are to be controlled 202,

[0061] 203, and 204. In particular, the image data is further generated based on the object parameters of the objects located in the environment as extracted from the environment data. For example, the image generation unit generates an image that includes a sun a house and a palm tree, based on an exemplary “I need holidays” user input data and the image 103. The combination of the features in the generated image is based on the general concept of the user input data, where the location and the color of the features in the generated image 107 is correlated to the spatial configuration of the objects in the image 103 provided as environment data. Then, a light setting determination unit 106 is configured to, using the generated image data ID, extract color data from the image data, determine respective light settings indicative of lighting operation parameters for the one or more lighting devices 202, 203, 204 using the color data, and to provide control signals CS indicative of the respective light settings for controlling the lighting devices 202, 203, 204. The light settings may specify a color palette and a distribution of the colors and color temperatures among the available lighting devices 202, 203, 204. Further, the light settings may optionally specify how the colors change over time. Colors from the image generated 107 may be mapped to lighting devices 202, 203, 204 by determining colors at locations in the image 107 which correspond to the locations of the lighting device. When mapping colors from the generated image 107 to lighting devices 202, 203, 204, it may be taken into account whether a lighting device is capable of rendering color or white-only. For instance, in the example shown in Fig. 1, both lighting devices 202, 204 can render different colors, and lighting device 203 can render only white light. The light setting determination unit 106 determines a light setting rendering green for the lighting device 204, based on the color of the palm tree represented in the generated image 107 and a warm yellow light for the lighting device 202, based on the sun represented in the generated image 107, and a white light with a color temperature of 4500K for the table top light 203, based on the color of the house represented in the image 107. If, for instance, the lighting device 204 did not have the operational capability of rendering colors, based on the ascertained lighting device data LD, then the light setting determination unit 106 would have determined an alternative light setting for said lighting device. Or, alternatively, the image generation unit would have generated an alternative image in accordance with the limited operational capability in terms of color rendering of the lighting device 204. Control signals indicative of the respective light settings are then provided to the lighting devices 202, 203, 204 for controlling their operation for rendering a light scene correlatable to the user’s intent as indicated in the user input data UD and also based on the spatial configuration of the environment and on the capabilities of the respective lighting devices 202, 203, 204.

[0062] Fig. 2 shows a schematic block diagram of another embodiment of a lighting arrangement including another exemplary lighting control device in accordance with the invention. This discussion will focus of the differences between the lighting arrangement 200 of Fig. 1 and the lighting arrangement 200 of Fig. 2. Those technical features having a similar or identical function will be referred to using the same reference signs and numbers.

[0063] In particular, the lighting control device 100 of Fig. 2 further comprises a prompt generation unit 108 that is configured to receive the user input data UD, the environment data ED and the lighting device data LD and to generate a prompt P based on the user input data, the environment data and the lighting device data and to provide said prompt P to the image data generation unit 104. The image data generation unit 104 is configured to provide the received prompt to a machine learning model ML configured as a text-to-image model configured to generate the image data based on the prompt. Preferably, the machine learning model ML is based on a diffusion model. Thus, the lighting control device 100 is configured to translate user’s intent for a lighting experience to rending the intended scene by generating an intermediary image that visually represent the intended lighting experience of the user from which subsequently light settings for a given environment are extracted and applied to the lighting devices of the lighting arrangement. The generation of the intermediary image data ID is based on the user input data, the environment data and the lighting device data.

[0064] Among the several algorithmic approaches to generate image from the prompt P, which can be a natural language description or a multimodal prompt, a diffusion model approach is preferred, since it is considered to be better than other approaches (such as Generative Adversarial Networks or GANs) on many aspects. A foundational model (such as Stable Diffusion vl.4) is leveraged, and then its architecture can be modified and fine-tuned for this particular application. Preferably, contextual parameters are introduced. By using the contextual parameters the diffusion process can be conditioned to generate an image that accurately captures the scene that the user would like to render on the given environment and lights. A first approach is to train a conditional diffusion model to generate the desired image from a supervised dataset. If a labeled dataset is available, synthetic data can be generated. Another conditioning method involves adapting pre-trained unconditional diffusion models to new conditions using the learned internal representations of the denoiser network. This approach is effective for various conditional generation tasks, including attribute-conditioned generation and mask-conditioned generation. By using the unconditional denoiser network as a feature extractor, guidance can be provided, which is robust to the initial inaccurate estimates of xO (i.e., original input data), and the guidance directions can be quickly learnt from a small set of labeled samples. The denoiser network is trained with varying noise levels in its inputs and has learned to extract features of different scales at different time steps.

[0065] Considering all the information and data about the environment (location and properties of objects and lighting devices, such as material, color, relative distance, orientation) and about the operational capabilities of the lighting devices (e.g., color spectrum, color temperature, intensity, directionality, etc.) the prompt generation unit 108 specifies a prompt for the diffusion model and the intermediary image is generated.

[0066] In an example, the user provided prompt as user input data states or is indicative of: “I am happy and I feel like flying”. The user input data, as an input prompt UD, is augmented at the prompt generation unit 108 for example with information about the location of the available light source for the user’s current field of view and about the object properties of the objects in the environment. For instance, the prompt augmentation states: “a first light is a pendant luminaire located 1 m away from the user in north western direction at a height of 2.5 m above a white table. A second light is a color-tunable LED stand light and is located on a distance of 4m of the user in southern direction close to the wall. A third lighting device is a color-temperature-tunable LED light located on the white table. The walls and the ceiling are white and the floor has black and white tiles on a checkerboard pattern. Please generate an image which reflects the user’s emotion and has prominent features at the location of the first, second and third light.”

[0067] The prompt provided by the user as user input data is provided alongside the augmentation as an augmented prompt P to the diffusion model and the diffusion model generates image data ID that represents an image 107 based on the data ascertained by the data ascertaining unit 102, in particular including information related to the spatial configuration of the environment.

[0068] After the image data ID is generated, for example representing a generated image 107, a color extraction module that is capable of extracting a color palette from an image (such as the standard hue color extraction module) can be used to extract the palette to be displayed by the first lighting device 202, the second lighting device 203 and the third lighting device 204. Alternatively, a visual question answering module (VQA) can be used to extract the color for each light element from the generated image. For example, if there are 2 lighting devices of the lighting arrangement in the environment, 2 colors are extracted from the image. If there are 3 lighting devices belonging to the arrangement in the environment, 3 colors are extracted from the generated image at the right locations.

[0069] For instance, the diffusion model generated from a “I am happy” user data input can result in a tropical sunset scene as image data 107 shown in Fig. 2 with an orange sun lowering in the right side of the user’s field of view where the lighting device 204 is located from the user’s point-of-view as shown in the image 103 an purple / red clouds in the top part of the image, where the pending light 202 is located and a white hammock between two palm trees, loosely inspired in the environment data regarding the object table in the provided image 103. Alternatively, instead of a 2D intermediary image 107, a 3D intermediary image (not shown) can be used. Colors are extracted from the image data representing the image 107 and light setting based on the extracted colors are determined for operating the lighting devices 202, 203, 204.

[0070] Fig. 3 shows a schematic block diagram of another embodiment of a lighting arrangement including another exemplary lighting control device in accordance with the invention. This discussion will focus of the differences between the lighting arrangement 200 of Fig. 3 and the lighting arrangements 200 of Figs. 1 and 2. Those technical features having a similar or identical function will be referred to using the same reference signs and numbers.

[0071] In the lighting arrangement 200 of Fig. 3, the lighting control device 100 further comprises a user feedback unit 110 that is configured to receive user feedback data FD indicative of a degree of satisfaction (or dissatisfaction) with the determined and implemented light-settings. In this exemplary lighting control device, the image data generation unit 104 is configured to generate alternative image data IDb that represents an alternative image 107b using the user input data UD, environment data ED and lighting device data LD upon determining, based on the user feedback data FD, that the degree of satisfaction is below a predetermined threshold.

[0072] If the user is dissatisfied with the light settings generated based on the image 107 generated in Fig. 2, the user can express this dissatisfaction interacting with the user feedback unit (e.g., voice activated, via a dedicated user interface, via an app, etc.) and the image data generation unit generates another image 107b based on the same ascertained data UD, ED, LD.

[0073] Optionally, the lighting control device 100 of any of the Figs. 1, 2 or 3 may comprise an image providing unit 112 that is configured to provide the image data (e.g., IDb) for displaying the corresponding generated image (e.g., 107b) to the user, as shown exemplarily in Fig. 3. Optionally, the image providing unit 112 is configured to provide the image data for displaying the image to the user in dependence on a predetermined image content criterion and / or a predetermined image resolution criterion. For instance, if the intermediary image 107b generated by the diffusion model in the image data generation unit 104 is a very low resolution image not deemed to be human interpretable (though still useful for the color palette picking algorithm), the image providing data can be configured to not show the image. Further, depending on how this specific image must be refined before it can be shown to this user, the system decides how many diffusion steps can be skipped. Additionally, or alternatively, the image 107b can be displayed or nor in dependence of a riskiness of the intermediary image. If, for example, the user requests a romantic lighting scene, the image providing unit can be configured to not display the intermediary image as the diffusion model may have generated inappropriate content. The guardrails may be conditioned on geographical location or country regulations.

[0074] Fig. 4 shows a flow diagram of an exemplary method 300 for controlling operation of a lighting arrangement 200 in accordance with the invention. The lighting arrangement comprises one or more lighting devices located in an environment, such as a room. The method 300 comprises, in a step 302, ascertaining user input data that is correlatable to a lighting scene to be applied to the environment, and that serves as a starting point in the determination of the lighting scene. The user input data is for example a text prompt, or an audio file, or an image or video file. The method also comprises, in a step 304, ascertaining environment data indicative of a spatial configuration of the environment. The environment data is preferably indicative of object parameters that are indicative of object properties and / or object location of the objects located in the environment and / or delimiting the environment. The environment data is preferably provided as an image file. The method also comprises, in a step 306, ascertaining lighting device data indicative of operational capabilities of the respective lighting devices. The method includes, in a step 308, generating image data based on the ascertained user input data, environment data and lighting device data. This is optionally done by performing a step 307 for generating an augmented prompt based on the user input data, the environment data and the lighting device data and providing said augmented prompt to a machine learning model configured as a text-to-image model configured to generate the image data based on the prompt, in particular a diffusion model. The method further includes, in a step 310, extracting color data from the image data, in a step 312, determining respective light settings indicative of lighting operation parameters for the one or more lighting devices using the color data, and, in a step 314, providing control signals indicative of the respective light settings for controlling the lighting devices. The steps 302-314 described above correspond to an exemplary method 300b for controlling operation of a lighting control device 100, which may additionally comprise, in a step 316, providing the generated image data indicative of an image for displaying the image to the user, for instance to a display device. The method 300 for controlling operation a lighting arrangement further comprises, in a step 318, operating the lighting devices in dependence on the control signals.

[0075] In summary, the invention is directed to a lighting control device for controlling operation of lighting devices located in an environment, comprising an input data ascertaining unit configured to ascertain user input data that is correlatable to a lighting scene to be applied, environment data indicative of an spatial configuration of the environment and lighting device data indicative of operational capabilities of the lighting devices. An image data generation unit is configured to generate image data indicative of an image based on the ascertained data and a light setting determination unit is configured to determine respective light settings indicative of lighting operation parameters for the lighting devices and to provide control signals indicative of the respective light settings for controlling the lighting devices.

[0076] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0077] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0078] A single unit or device may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0079] A computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

CLAIMS:

1. Lighting control device (100) for controlling operation of a lighting arrangement (200) including one or more lighting devices (202, 203, 204) located in an environment (206), the lighting control device comprising: an input data ascertaining unit (102) configured to ascertain: user input data (UD) that is correlatable to a lighting scene to be applied to the environment; lighting device data (LD) indicative of operational capabilities of the respective lighting devices; an image data generation unit (104) configured to generate image data (ID) indicative of an image based on the ascertained user input data and lighting device data; and a light setting determination unit (106) configured to, using the generated image data, extract color data from the image data, determine respective light settings indicative of lighting operation parameters for the one or more lighting devices using the color data, and to provide control signals (CS) indicative of the respective light settings for controlling the lighting devices.

2. The lighting control device of claim 1, wherein the input data ascertaining unit (102) configured to ascertain: environment data (ED) indicative of a spatial configuration of the environment; and wherein the image data generation unit (104) is configured to generate the image data (ID) indicative of an image based on the ascertained user input data, environment data and lighting device data.

3. The lighting control device of claim 1 or 2, wherein the user input data ascertained by the input data ascertaining unit comprises text data and / or image data and / or video data and / or audio data.

4. The lighting control device of claim 2, wherein the environment data is indicative of object parameters that are indicative of object properties and / or object location of the objects located in the environment and / or delimiting the environment, and / or of subject parameters indicative of subject properties and / or subject location of the subjects located in the environment, and / or environmental parameters indicative of environmental conditions in the environment, and wherein the image data is generated based on the object parameters and / or on the subject parameters, and / or the environment parameters.

5. The lighting control device of claim 2, wherein the environment data is indicative of object parameters that are indicative of object location of at least one of the lighting devices of the lighting arrangement located in the environment, and wherein the image data is generated based on the object parameters6. The lighting control device of any of the preceding claims, further comprising a prompt generation unit (108) that is configured to receive the user input data and the lighting device data and to generate a prompt based on the user input data, the environment data and the lighting device data and to provide said prompt to the image data generation unit.

7. The lighting control device of any of claim 6, wherein the image data generation unit is configured to provide the received prompt to a machine learning model configured to generate the image data based on the prompt.

8. The lighting control device of claim 6, wherein the machine learning model is based on a diffusion model or a GAN model or the decoder of a Variable Autoencoder Model.

9. The lighting control device of any of the preceding claims, further comprising a user feedback unit configured to receive user feedback data indicative of a degree of satisfaction with the determined respective light-settings, and wherein the image data generation unit is configured to generate alternative image data using user input data, environment data and lighting device data upon determining, based on the user feedback data, that the degree of satisfaction is below a predetermined threshold.

10. The lighting control device of any of the preceding claims, further comprising an image providing unit that is configured to provide the image data for displaying at least a portion of the corresponding image to the user.

11. The lighting control device of claim 10, wherein the image providing unit is configured to provide the image data for displaying the image to the user in dependence on a predetermined image content criterion and / or a predetermined image resolution criterion.

12. Lighting arrangement (200) comprising: a lighting control device (100) in accordance with any of the preceding claims; and one or more lighting devices (202, 203, 204) that are configured to receive control signals (CI) indicative of the respective light settings and to operate in dependence thereof.

13. The lighting arrangement of claim 12, wherein the lighting control device is in accordance with claim 10 or 11, further comprising a display device configured to receive the image data and to display the image defined by the image data.

14. Method (300b) for controlling operation of a light control device for controlling operation of a lighting arrangement including one or more lighting devices located in an environment, the method comprising: ascertaining (302, 304, 306): user input data that is correlatable to a lighting scene to be applied to the environment; lighting device data indicative of operational capabilities of the respective lighting devices; generating (308) image data based on the ascertained user input data and lighting device data; and using the generated image data, extracting (310) color data from the image data; determining (312) respective light settings indicative of lighting operation parameters for the one or more lighting devices using the color data; and providing (314) control signals indicative of the respective light settings for controlling the lighting devices.

15. Computer program comprising instruction which, when executed by a lighting control device, cause the lighting control device to perform the steps of the method of claim

Images