Light-effect control method and apparatus, and electronic device and storage medium

Abstract

A light-effect control method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring user information (S11); generating a static image or a dynamic image on the basis of the user information (S12); and controlling the light effect of a lighting device on the basis of the static image or the dynamic image (S13). By means of the method, a lighting device can be accurately controlled, and richer light effects can be provided, thereby improving the user experience.

Description

Lighting effect control method, device, electronic device and storage medium Technical Field

[0001] The present application relates to the field of lighting technology, and in particular to a lighting effect control method, device, electronic device, and storage medium. Background Art

[0002] With the continuous development of lighting technology, intelligent lighting has become a trend. Related technologies often control lighting effects based on fixed image templates. This can lead to a mismatch between user needs and lighting effects, making it difficult to accurately control lighting equipment and thus failing to provide users with a personalized user experience.

[0003] Summary of the Invention

[0004] In view of the above, it is necessary to provide a lighting effect control method, device, electronic device and storage medium that can solve the technical problems of difficulty in effectively controlling lighting equipment and poor user experience due to the mismatch between user needs and lighting effects.

[0005] On the one hand, the present application provides a lighting effect control method, which includes: obtaining user information, generating a static image or a dynamic image based on the user information, and controlling the lighting effect of a lighting device based on the static image or the dynamic image.

[0006] In some embodiments of the present application, the user information includes at least one of text, video, image and sound.

[0007] In some embodiments of the present application, generating a static image or a dynamic image based on the user information includes: if the user information includes theme information, generating a static image or a dynamic image of the theme corresponding to the theme information according to a preset style; if the user information includes theme information and style information, generating a static image or a dynamic image of the theme corresponding to the theme information according to the style corresponding to the style information.

[0008] In some embodiments of the present application, generating a static image or a dynamic image based on the user information includes: processing the user information based on artificial intelligence content generation technology to generate the static image or the dynamic image.

[0009] In some embodiments of the present application, the processing of the user information based on artificial intelligence content generation technology to generate the static image or the dynamic image includes: inputting the user information into a diffusion model to generate the static image or the dynamic image.

[0010] In some embodiments of the present application, the method further includes: performing pixelation processing on the static image or the dynamic image, and controlling the lighting effect of the lighting device according to the pixelated static image or dynamic image.

[0011] In some embodiments of the present application, the pixelating the static image or the dynamic image includes: pixelating the static image or the dynamic image based on an image translation neural network model.

[0012] In some embodiments of the present application, controlling the lighting effects of the lighting device based on the static image includes: controlling the lighting effects of each of the lighting devices or each light-emitting element in the lighting device based on the pixel value of each pixel in the static image, or controlling the lighting effects of each of the lighting devices or each light-emitting element in the lighting device based on the pixel values ​​of multiple pixels in the static image.

[0013] On the other hand, the present application provides a lighting effect control device that runs on an electronic device, and the lighting effect control device includes: an acquisition unit for acquiring user information, a generation unit for generating a static image or a dynamic image based on the user information, and a control unit for controlling the lighting effect of the lighting device based on the static image or the dynamic image.

[0014] On the other hand, the present application provides an electronic device, which includes: a memory storing at least one instruction; and a processor executing at least one instruction to implement the lighting effect control method.

[0015] On the other hand, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the lighting effect control method.

[0016] Through the above-described implementation, user information is obtained. Since user information can reflect the user's lighting effect requirements, static or dynamic images that meet the user's needs can be flexibly generated based on the user information, thereby accurately controlling the lighting device. Furthermore, due to the accurate control of the lighting device, dynamic images can provide richer lighting effects, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG1 is a diagram showing an application scenario of a lighting device control method provided in an embodiment of the present application.

[0018] FIG2 is a flow chart of a lighting effect control method provided in an embodiment of the present application.

[0019] FIG3 is a schematic diagram of a realistic-style static image provided by an embodiment of the present application.

[0020] FIG4 is a schematic diagram of a pixel-style static image provided by an embodiment of the present application.

[0021] FIG5 is a flowchart of a method for training a diffusion model according to an embodiment of the present application.

[0022] FIG6 is a flowchart of a method for controlling lighting effects provided in another embodiment of the present application.

[0023] FIG7 is a flowchart of a method for training an image translation neural network model according to an embodiment of the present application.

[0024] FIG8 is a functional module diagram of a lighting effect control device provided in an embodiment of the present application.

[0025] FIG9 is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of this application clearer, this application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0028] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0029] With the continuous development of lighting technology, intelligent lighting has become a trend. Related technologies often control lighting effects based on fixed image templates. This can lead to a mismatch between user needs and lighting effects, making it difficult to accurately control lighting equipment and thus failing to provide users with a personalized user experience.

[0030] To address the above technical issues, the present application provides a lighting effect control method, apparatus, electronic device, and storage medium that can accurately control lighting equipment and provide richer lighting effects, thereby improving user experience. The lighting control method provided in the embodiments of the present application can be executed by one or more electronic devices.

[0031] As shown in Figure 1, it is an application scenario diagram of the lighting device control method provided by an embodiment of the present application. In Figure 1, an electronic device 10 communicates with multiple lighting devices 20 respectively, and Figure 1 only shows one lighting device.

[0032] For example, the lighting device 20 in Figure 1 is a light string, which constitutes a curtain light. The curtain light shows a pixel-style pumpkin element, and the lighting effect (lighting effect) of the lamp beads on the curtain light can also be synchronized with the sound (such as music), so that the lighting effect is more in line with the scene. In addition to the light string shown in Figure 1, the lighting device 20 can be various types of lamps that can provide lighting functions. For example, the lighting device 20 can also be an LED lamp, an energy-saving lamp, a floodlight, a security lamp, a light strip, and an eaves lamp. The light string is composed of a plurality of lamp beads, each of which has an integrated circuit integrated in it. Through a signal line, the controller connects the integrated circuits of all the lamp beads in the light string in series, thereby realizing control of the entire light string.

[0033] The electronic device 10 in Figure 1 is a mobile phone. Besides the mobile phone shown in Figure 1 , the electronic device 10 may also be a tablet computer, a laptop computer, or a computer. The present embodiment does not impose any restrictions on the specific type of the electronic device 10. The electronic device 10 can connect to multiple lighting devices 20 via Bluetooth, hotspots, Wi-Fi, or other methods, thereby enabling the electronic device 10 to control the multiple lighting devices 20.

[0034] FIG2 is a flowchart of a lighting effect control method according to an embodiment of the present application. The order of the steps in the flowchart may be adjusted based on different needs, and some steps may be omitted. The method is performed by an electronic device, such as the electronic device 10 shown in FIG1 .

[0035] S11, obtain user information.

[0036] In some embodiments of the present application, user information includes at least one of text, video, image and sound, wherein the sound may include various file formats such as instant voice and audio files, and the sound may be sound effects, voice and music, etc. The images in the user information include various image formats such as static images and dynamic images (such as images in GIF format). User information may include text, video, image and sound input by the user, and the user information may be obtained in various ways such as the user typing on the keyboard, the user handwriting input on the interactive interface of the electronic device, the electronic device converting the voice recorded by the microphone, or the electronic device parsing the image, and the present application does not impose any restrictions on this.

[0037] In some embodiments of the present application, user information includes theme information, or user information includes theme information and style information. The theme information is used to control the theme of the static image or dynamic image below, and the style information is used to control the style of the static image or dynamic image. The theme can be the image content in the static image or dynamic image, and the style can be the visual style of the static image or dynamic image. Themes include, but are not limited to, festivals, daily life, hobbies, natural scenery, and entertainment. For example, festivals can include the Spring Festival, Mid-Autumn Festival, Christmas, Thanksgiving, and Halloween, daily life can include family life, work scenes, and school life, hobbies can include reading, music, painting, photography, fitness, and travel, and entertainment can include games, sports, movies, animation, and celebrities.

[0038] Styles include, but are not limited to, realistic style and pixel style. Realistic style is a form of expression that pursues realism, and strives to reproduce real-life scenes and characters through detailed depiction and realistic expression techniques. Realistic-style images are characterized by clear textures, realistic pictures, rich and natural colors, delicate brushstrokes, and realistic light and shadow, presenting a realistic visual effect. Pixel style is a form of expression that uses pixels to express characters or scenes, and presents a pixelated effect by decomposing the image into pixels. Pixel-style images are characterized by bright colors, simplified details, a retro feel, and a sense of dynamism. They express the characteristics of the object through concise lines and shapes, and use high-contrast colors to express the theme of the picture. They usually use low-resolution image elements, simple image elements, and bright colors.

[0039] In some embodiments of the present application, the electronic device can convert the sound information in the user information into text information, decouple the dynamic images and / or videos in the user information into multi-frame images, and generate a static image or dynamic image corresponding to the user information through the converted text and / or multi-frame images.

[0040] In some embodiments of the present application, one category of user information may include only subject information, or may include both subject information and style information. For example, when the user information is text information, the text information may include only subject information, or may include both subject information and style information. Multiple categories of user information may include only subject information, or may include both subject information and style information. For example, when the user information includes text information and image information, the text information and the image information may include only subject information, or the text information and the image information may include both subject information and style information. For the case where the text information and the image information include both subject information and style information, the text information may include style information and the image information may include subject information, or the text information may include subject information and the image information may include style information, or both the text information and the image information may include subject information and style information.

[0041] S12, generating a static image or a dynamic image according to the user information.

[0042] In some embodiments of the present application, a dynamic image refers to an image that includes a dynamically changing effect. A dynamic image can display a process or multiple consecutive static images. When the user information includes a video, or the user information includes multiple consecutive frames of images, or the user information includes text information describing a dynamic process, the video or dynamic image can be directly obtained based on the user information. Alternatively, a video can be first generated based on the user information, and then multiple frames of images can be selected from the generated video to synthesize the dynamic image. The number of selected images can be set arbitrarily and is not limited by this application.

[0043] In some embodiments of the present application, the electronic device generating a static image or dynamic image based on user information includes: when the user information includes theme information, the electronic device generating a static image or dynamic image corresponding to the theme and a preset style based on the user information; or, when the user information includes theme information and style information, the electronic device generating a static image or dynamic image based on the user information includes: the electronic device generating a static image or dynamic image corresponding to the theme and a corresponding style based on the user information. The preset style can be set by the user, and this application does not limit this. For example, the preset style can be a realistic style or a pixel style.

[0044] Specifically, the electronic device processes user information based on artificial intelligence generated content technology (AI-Generated Content, AIGC) to generate static images or dynamic images with a theme and / or style. The AI-generated content technology can be set by itself, and this application does not limit this. For example, the AI-generated content technology can be a trained diffusion model (Diffusion or Stable Diffusion) or ChatGPT, etc.

[0045] In some embodiments, the diffusion model includes a text encoder model (Text Encoder), an image generator model (Image Generator), and an autoencoder (AutoEncoder). For example, the text encoder model may be a CLIP model, the image generator model may include a Unet network and a sampler algorithm, and the autoencoder may be an AutoEncoderKL. Because the diffusion model includes the text encoder model, the input information of the diffusion model includes text information.

[0046] In some other embodiments, the network architecture of the diffusion model can also be implemented through the attention mechanism and the convolutional neural network. The above method of implementing the diffusion model architecture is only an example. In actual applications, the network architecture of the diffusion model can also be implemented through other methods.

[0047] In one embodiment of the present application, taking the example of user information including text information, if the user information includes theme information and style information, the electronic device inputs the user information into the trained diffusion model to obtain a static image, including: the electronic device calls the text encoder model to encode the user information into a text embedding vector, the electronic device calls the latent seed (Latent Seed) to generate an image information tensor, inputs the text embedding vector and the image information tensor into the image information generator model to obtain a first image latent vector, uses the first image latent vector to iterate the image information tensor to obtain a first image latent vector corresponding to a preset number of iterations, and the electronic device calls the image decoder (Image Decoder) in the autoencoder to decode the first image latent vector corresponding to the preset number of iterations to obtain a static image of the corresponding theme and corresponding style. Among them, the preset number of iterations can be set by yourself, and this application does not limit this. For example, the preset number of iterations can be 50 times.

[0048] For example, when the preset style is realistic, if the user information is text information: a Christmas-themed photo, Santa Claus is standing in front of a house with a smile on his face and a Christmas gift hanging on his face. Among them, the text information can be obtained in a variety of ways, such as the user typing on the keyboard, the user handwriting input on the interactive interface of the electronic device, the electronic device converting the voice recorded by the microphone, or the electronic device parsing the image. The theme of the static image indicated by the user information is Christmas theme, and the style of the generated static image is not indicated. The electronic device can call the diffusion model to generate a realistic-style static image, as shown in Figure 3, which is a schematic diagram of a realistic-style static image provided by an embodiment of the present application. The realistic-style static image shown in Figure 3 includes Santa Claus wearing a Santa hat, a Christmas suit, holding Christmas gifts in both hands, and a smile on his face. The realistic-style static image shown in Figure 3 has the characteristics of clear texture, realistic picture, delicate brushstrokes, and realistic light and shadow.

[0049] In another embodiment of the present application, taking the example of user information including text information and image information, if the user information includes theme information and style information, the electronic device inputs the user information into the trained diffusion model to obtain a static image, including: the electronic device calls the text encoder model to encode the text information in the user information into a text embedding vector, the electronic device calls the image encoder (Image Encoder) in the autoencoder to encode the image information in the user information into an image embedding vector, inputs the text embedding vector and the image embedding vector into the image information generator model to obtain a second image latent vector, uses the second image latent vector to iterate the image embedding vector to obtain a second image latent vector corresponding to a preset number of iterations, and the electronic device calls the image decoder in the autoencoder to decode the second image latent vector corresponding to the preset number of iterations to obtain a static image of the corresponding theme and corresponding style.

[0050] For example, if the user information is text information: a pixelated photo with a Christmas theme, Santa Claus is standing in front of a house with a smile on his face and a Christmas gift hanging on his face. The text information can be obtained in a variety of ways, such as the user typing on the keyboard, the user handwriting on the interactive interface of the electronic device, the electronic device converting the voice recorded by the microphone, or the electronic device parsing the image. The user information indicates that the theme of the generated static image is a Christmas theme, and indicates that the style of the generated static image is a pixel style. The electronic device can call the diffusion model to generate a pixel-style static image. As shown in Figure 4, it is a schematic diagram of a pixel-style static image provided by an embodiment of the present application. The pixel-style static image shown in Figure 4 includes Santa Claus wearing a Santa hat, a Christmas suit, holding Christmas gifts in both hands, and a smile on his face. The pixel-style static image shown in Figure 4 presents a pixelated effect, which is characterized by simplified details, a retro feel, and a dynamic feel.

[0051] In other embodiments of the present application, taking the example of user information including dynamic images and / or videos, if the user information includes theme information and style information, since the dynamic images and / or videos in the user information include multiple frames of images, the electronic device inputs the multiple frames of the dynamic images and / or videos in the user information into the diffusion model to obtain a static image corresponding to each frame of the image, and synthesizes the static images corresponding to the multiple frames of the image to obtain a dynamic image corresponding to the theme and the corresponding style. The method for generating the static image corresponding to each frame of the image can refer to the method for generating the static image in the example, and will not be repeated in this application.

[0052] In other embodiments of the present application, for the case where user information only includes theme information, the generation method of static images or dynamic images corresponding to the theme and preset style can refer to the above examples, and this application will not explain them one by one.

[0053] In other embodiments of the present application, in order to improve the operating efficiency of electronic devices, artificial intelligence content generation technology can be deployed in devices such as servers, cloud or cloud servers, and electronic devices can communicate with devices such as servers, cloud or cloud servers through Bluetooth, hotspots, Wi-Fi, etc., so as to obtain static images or dynamic images generated by artificial intelligence content generation technology. For example, after receiving user information, the electronic device can send the user information to devices such as servers, cloud or cloud servers, and receive images sent from devices such as servers, cloud or cloud servers as static images or dynamic images. Among them, the server, cloud or cloud server can store diffusion models corresponding to multiple styles. When receiving user information sent from the electronic device, the corresponding diffusion model can be called to generate a static image or dynamic image of the style indicated by the user information.

[0054] In this embodiment, when the user information includes theme information and style information, a static image or dynamic image that meets the user's needs can be accurately generated. Since the user information can be set by the user, the generation of static images and dynamic images is flexible.

[0055] S13, controlling the lighting effect of the lighting device according to the static image or the dynamic image.

[0056] In some embodiments of the present application, the electronic device controls the lighting effects of a lighting device based on a static image, including: the electronic device controls the lighting effects of each lighting device or each light-emitting element in the lighting device based on the pixel value of each pixel point in the static image, or controls the lighting effects of each lighting device or each light-emitting element in the lighting device based on the pixel values ​​of multiple pixels in the static image.

[0057] Among them, lighting devices can include light strings, energy-saving lamps, floodlights, security lights, light strips, and eaves lights. A lighting device can be composed of multiple light-emitting elements. For example, if the lighting device is a light string composed of multiple lamp beads, a light-emitting element is one of the lamp beads in the light string. The pixel value of each pixel in a static image can control the lighting effect of a lighting device or a light-emitting element in a lighting device. Because a static image includes the pixel values ​​of multiple pixels, when there are multiple lighting devices or a lighting device is composed of multiple light-emitting elements, the pixel values ​​of multiple pixels in the static image can be used to control the lighting effect of each lighting device or each light-emitting element in the lighting device.

[0058] The electronic device can average or weighted average the pixel values ​​of multiple pixels, and control the lighting effect of a lighting device or a light-emitting element in the lighting device according to the averaged or weighted averaged pixel values. The number of the averaged or weighted averaged multiple pixels can be set voluntarily, and this application does not impose any restrictions on this. When averaging or weighted averaging the pixel values ​​of multiple pixels, the static image or dynamic image can be divided into regions, and the pixel values ​​of the pixels in each divided region can be weighted averaged or averaged, wherein a pre-trained image segmentation model can be used to divide the static image or dynamic image into regions, and the number of pixels in different regions is different. The pixels in different regions can be averaged or weighted averaged, and the lighting effect of a lighting device or a light-emitting element in the lighting device can be controlled according to the averaged or weighted averaged pixel values. The number of pixels in each region can be set voluntarily, and this application does not impose any restrictions on this. When the lighting device is a light string, the light-emitting element can be one of the lamp beads in the light string. The generated static or dynamic images can use different color spaces, including RGB (red, green, blue) color space, HSL (hue, saturation, brightness) or HSV (hue, saturation, value) and other color spaces.

[0059] If the generated static image or dynamic image is in another color space such as HSL or HSV, the electronic device can convert the static image or dynamic image from the HSL or HSV color space to the RGB color space, and then control the lighting effects of the lighting device based on the static image or dynamic image in RGB format obtained after the conversion. The pixel value obtained by averaging or weighted averaging multiple pixels in each area of ​​the static image can control the lighting effects of a lighting device or a light-emitting element in the lighting device. Since the static image can be divided into multiple areas, when there are multiple lighting devices or the lighting device is composed of multiple light-emitting elements, the pixel values ​​of the multiple pixels included in the static image can be used to control the lighting effects of each lighting device or each light-emitting element in the lighting device.

[0060] Specifically, because each pixel value of a static or dynamic image in RGB format is composed of three components: red (Red), green (Green), and blue (Blue), the pixel value of a static or dynamic image in RGB format is also called an RGB value. A lighting device or each light-emitting element includes light emitters corresponding to the three colors of red, green, and blue. For example, the light emitters may be light-emitting diodes (LEDs). The electronic device allocates the red, green, and blue components of each pixel value to the corresponding light emitter in the lighting device or each light-emitting element, thereby enabling the lighting device or light-emitting element to emit a lighting effect corresponding to the static image.

[0061] In other embodiments of the present application, since the dynamic image includes multiple frames of images, the electronic device can control the lighting effects of the lighting device in sequence according to each frame of the image, so that the lighting device emits a dynamic lighting effect. The process of controlling the lighting effects of the lighting device according to each frame of the dynamic image is basically the same as the above-mentioned process of controlling the lighting effects of the lighting device according to the static image. The process of controlling the lighting effects of the lighting device according to the dynamic image is similar to the process of controlling the lighting effects of the lighting device according to the static image, so this application will not repeat the description.

[0062] In other embodiments of the present application, if the user information includes sound information, such as music and / or sound effects, the electronic device can control the lighting effects of the lighting device based on the static image, dynamic image, and sound information in the user information. The electronic device distributes the red, green, and blue components of each pixel value to the corresponding light source in each lighting device or each light-emitting element, causing the lighting device or light-emitting element to emit the corresponding light. The electronic device also controls the light to produce light-changing effects such as slow flow, flashing, gradual change, and sweeping according to the rhythm of the music and / or sound effects, so that the lighting effects of the lighting device or light-emitting element better meet the user's needs.

[0063] In some embodiments of the present application, when the resolution of a static image or a dynamic image does not match the number of lighting devices, the electronic device can scale the static image and / or the dynamic image according to the number of lighting devices. For example, if the lighting device is a light string, the number of lamp beads is 520, and the resolution of the static image and / or the dynamic image is 512x512, the electronic device can downsample the static image and / or the dynamic image with a resolution of 512x512 to 26x20 through a nearest neighbor downsampling algorithm to obtain a static image and / or a dynamic image with a resolution of 26x20, and control the lighting effects of the 520 lamp beads according to the static image and / or the dynamic image with a resolution of 26x20. For example, the downsampling method can be nearest neighbor interpolation, bilinear interpolation, etc. The resolution of an image refers to the number of pixels included in the image, usually expressed as the number of horizontal pixels x the number of vertical pixels. For example, an image with a resolution of 480x800 is composed of 480 pixels in the horizontal direction and 800 pixels in the vertical direction (a total of 384,000 pixels).

[0064] In other embodiments of the present application, if the video or dynamic image generated based on user information is too large, the generated video or dynamic image can be encoded and compressed using a preset encoding format to generate a compressed packet in the preset encoding format. A decoder decodes the compressed packet in the preset encoding format to generate multiple decoded images, and the lighting effects of the lighting device are controlled based on the multiple decoded images. Preset encoding formats include, but are not limited to, MPEG-1, MPEG-2, MPEG-4, H.263, and H.264, and decoders include, but are not limited to, decoders in video codecs such as H.265, VP9, ​​and AV1. The process of controlling the lighting effects of the lighting device based on each decoded image is similar to the process of controlling the lighting effects of the lighting device based on a static image, and therefore will not be repeated herein. Through the above-described embodiments, user information is obtained. Since the user information can reflect the user's lighting effect requirements, static or dynamic images that meet the user's requirements can be flexibly generated based on the user information, thereby enabling accurate control of the lighting device. Furthermore, due to the accurate control of the lighting device, the dynamic images can provide richer lighting effects, thereby improving the user experience.

[0065] In some embodiments of the present application, before inputting user information into a diffusion model to generate static or dynamic images, the diffusion network corresponding to the diffusion model can be trained to obtain a pre-trained diffusion model. This pre-trained diffusion model can better meet user needs. Figure 5 is a flow chart of a diffusion model training method provided in one embodiment of the present application.

[0066] S121, obtaining training data.

[0067] In some embodiments of the present application, the training data includes multiple training images and description information for each training image, each training image having a corresponding style. For example, the styles of the training images include, but are not limited to, realistic style and pixel style. The image content in the training images may be emojis.

[0068] Each training image of the pixel style can be obtained by pixelating the original training image. The specific pixelation process can be referred to steps S231-S232 below. The original training image can be obtained from a preset dataset, wherein the preset dataset can be set by itself and is not limited in this application.

[0069] In some embodiments of the present application, the electronic device may call a natural language processing model to identify each training image and obtain description information of each training image. The natural language processing model may be set by itself, and this application does not limit this. For example, the natural language processing model may be a GPT4-V model, a BLIP model, a BLIP2 model, or a DeepBooru model. The description information of the training image is a description of the object in the training image, and the description information includes but is not limited to: words, phrases, and sentences. For example, if the image content in the training image is an emoticon package, the description information of the training image may include expressions, actions, and objects.

[0070] S122, based on the model fine-tuning algorithm, using the training data to adjust the diffusion network to obtain a diffusion model.

[0071] In some embodiments of the present application, model fine-tuning algorithms include, but are not limited to, Low-Rank Adaptation (LoRA) and DreamBooth. The electronic device uses training data to adjust the diffusion network based on the model fine-tuning algorithm to obtain the diffusion model. The process is essentially the same as the process of generating static and dynamic images using the diffusion model described above, and is not repeated here.

[0072] In this embodiment, after training is complete, if the user information includes both theme information and style information, the diffusion model can output a static or dynamic image corresponding to the theme and style. Alternatively, if the user information only includes theme information and does not include style information, the diffusion model can output a static or dynamic image corresponding to the theme and a preset style. The preset style can be set voluntarily, and this application does not impose any restrictions on this. For example, the preset style can be a realistic style or a pixel style. Style refers to the visual style of an image.

[0073] In some embodiments of the present application, as shown in FIG6 , a flow chart of a method for controlling lighting effects provided by another embodiment of the present application is shown, including the following steps:

[0074] S21, obtain user information.

[0075] In some embodiments of the present application, the user information and the process of obtaining the user information may refer to step S11, and the present application will not repeat the description here.

[0076] S22: Generate a static image or a dynamic image according to the user information.

[0077] In some embodiments of the present application, the generation process of static images and dynamic images can refer to step S12, and the present application will not repeat the description here.

[0078] S23, performing pixelation processing on the static image or the dynamic image.

[0079] In some embodiments of the present application, the electronic device may perform pixelation processing on a static image or a dynamic image based on a trained image translation neural network model (Cycle-Consistent Generative Adversarial Network, CycleGAN).

[0080] Among them, the trained image translation neural network model learns the pixelated mapping relationship. If the resolution of the static image or dynamic image is different from the resolution required by the input image of the image translation neural network model, the static image or dynamic image can be scaled first so that the resolution of the static image or dynamic image after scaling is the same as the resolution required by the input image of the image translation neural network model. For example, if the resolution of the static image or dynamic image is 512x512 and the resolution required by the input image of the image translation neural network model is 256x256, the electronic device can downsample the static image or dynamic image so that the resolution of the downsampled static image or dynamic image is 256x256.

[0081] S24, controlling the lighting effect of the lighting device according to the pixelated static image or dynamic image.

[0082] In some embodiments of the present application, the resolution of the pixelated still image or dynamic image output by the image translation neural network model is smaller than the resolution of the still image or dynamic image input to the image translation neural network model. For example, if the resolution of the still image or dynamic image input to the image translation neural network model is 512x512, the resolution of the pixelated still image or dynamic image output by the image translation neural network model is 256x256.

[0083] In other embodiments of the present application, in order to match the number of light-emitting elements in the lighting device, the pixelated static image or dynamic image can be downsampled, and the lighting effect of the lighting device can be controlled based on the downsampled static image or dynamic image. For example, if the lighting device is a curtain light with 520 lamp beads, and the resolution of the pixelated static image or dynamic image is 256x256, the electronic device can downsample the pixelated static image or dynamic image through the nearest neighbor downsampling algorithm, so that the resolution of the downsampled static image or dynamic image is 26x20. Alternatively, if the lighting device has 1024 lamp beads, and the resolution of the pixelated static image or dynamic image is 256x256, the electronic device can downsample the pixelated static image or dynamic image through the nearest neighbor downsampling algorithm, so that the resolution of the downsampled static image or dynamic image is 32x32.

[0084] There may be a mismatch between the resolution of static or dynamic images generated using AI-generated content technology and the number of light-emitting elements in a lighting device. When the number of pixels in the generated static or dynamic image is greater than the number of light-emitting elements in the lighting device, directly downsampling the generated static or dynamic image may cause the sampled static or dynamic image to exhibit problems such as detail loss, blurring, and color distortion. Using the sampled static or dynamic image to control the lighting effects of a lighting device may result in inaccurate lighting effects and a mismatch between the lighting effects and user information. In this embodiment, the static or dynamic image is pixelated using an image translation neural network model. Because the image translation neural network model learns the mapping relationship between pixelations and can output a pixelated static or dynamic image with a lower resolution than the original static or dynamic image, it can reduce detail loss, blurring, and color distortion, thereby ensuring the accuracy of the lighting effect and the adaptability of the lighting effect to the user information. The above-mentioned image translation neural network model is only an example, and other network models can also be used in actual applications.

[0085] In some embodiments of the present application, before using the image translation neural network model for pixelation processing, the image translation neural network needs to be trained in advance so that the trained image translation neural network model can learn the pixelation mapping relationship and thus output the pixelated image. As shown in Figure 7, it is a flow chart of the training method of the image translation neural network model provided in one embodiment of the present application, which specifically includes the following steps:

[0086] S231, constructing a pixelated dataset.

[0087] In some embodiments of the present application, constructing a pixelated data set by an electronic device includes: the electronic device acquiring multiple original images, wherein the image resolution of each original image is the same as the resolution required for the input image of the image translation neural network, and downsampling each original image through a filter to obtain an image of a preset resolution, the electronic device upsampling the image of the preset resolution through an interpolation algorithm so that the resolution of the upsampled image is the same as the resolution required for the input image of the image translation neural network, determining the upsampled image as the pixelated image corresponding to each original image, and determining the multiple original images and the pixelated image corresponding to each original image as image data in the pixelated data set.

[0088] Among them, the filter and interpolation algorithm can be set by yourself, and this application does not impose any restrictions on this. For example, the filter can be a Lanczos filter, and the interpolation algorithm can be a nearest neighbor interpolation algorithm. The preset resolution can be set by yourself, and this application does not impose any restrictions on this. For example, the preset resolution can be 80x80, 64x64, 48x48, 32x32, 16x16, etc. The downsampling of each original image by the filter can be random. The style of each pixelated image is a pixel style. For an introduction to the pixel style, please refer to step S11, and this application will not repeat the description.

[0089] For example, the image resolution of the original image and the resolution required for the input image of the image translation neural network are both 256x256. The electronic device can downsample each original image to a resolution of 64x64 through a Lanczos filter to obtain a 64x64 image, and upsample the 64x64 image to 256x256 through a nearest neighbor interpolation algorithm to obtain a pixelated image corresponding to each original image. The multiple original images and the pixelated image corresponding to each original image are determined as image data in the pixelated dataset.

[0090] In some embodiments of the present application, if the resolution of the original image is different from the resolution required by the input image of the image translation neural network, the electronic device can scale the original image so that the resolution of the scaled original image is the same as the resolution required by the input image of the image translation neural network. The original image can be scaled by a variety of methods, and the present application does not limit the scaling method. For example, continuing with the above embodiment, if the image resolution of the original image is 128x128 and the resolution required by the input image of the image translation neural network is 256x256, the electronic device can scale the resolution of each original image from 128x128 to 256x256 through bilinear interpolation.

[0091] S232: Based on each original image in the pixelated data set and the pixelated image of each original image, train the image translation neural network to obtain an image translation neural network model.

[0092] In some embodiments of the present application, during the training of the image translation neural network, the image translation neural network learns how to convert or map the style of the original image into a pixel style or a realistic style, so that the trained image translation neural network model can directly convert the style of any image into a pixel style or a realistic style. When the user information only includes subject information but does not include style information, the diffusion model can output a static image or a dynamic image corresponding to the subject and preset style. The preset style can be set by yourself, and this application does not impose any restrictions on this. For example, the preset style can be a realistic style or a pixel style. Style refers to the visual style of an image.

[0093] In some embodiments of the present application, an electronic device can train an image translation neural network based on each original image and the pixelated image of each original image in a pixelated dataset using an adversarial training method. During the training process, the image translation neural network includes two groups of generative adversarial networks, each group of generative adversarial networks includes a generator and a discriminator. To facilitate the description of the training process of the adversarial network, the generator and discriminator of one group of adversarial networks are referred to as the first generator and the first discriminator, respectively, and the generator and discriminator of the other group of adversarial networks are referred to as the second generator and the second discriminator, respectively.

[0094] During the training of the first generator and the first discriminator, the electronic device calls the first generator to generate a first generated image of a corresponding pixel style based on each original image, and calculates a first loss value of the first generator based on the difference between the pixelated image corresponding to each original image and the first generated image, and adjusts the first generator based on the first loss value. The electronic device calls the first discriminator to identify each first generated image, outputs a first identification result, calculates a second loss value of the first discriminator based on the difference between the first identification result and the first preset label, and adjusts the first discriminator based on the second loss value. The first identification result can be a label indicating whether the input first generated image is true or false. For example, when 1 is used to indicate true identification (identification of the first generated image not generated by the first generator) and 0 is used to indicate false identification (identification of the first generated image generated by the first generator), the first identification result can be 0 or 1, and the first preset label can be 0. The preset conditions can be set voluntarily and are not limited in this application. Since the first preset label indicates a label identified as false, when calculating the second loss value of the first discriminator based on the first preset label, the difference between the first identification result and the first preset label can be used as a basis for measuring the accuracy of the first discriminator's judgment. By minimizing this difference to adjust the parameters of the first discriminator, the ability of the first discriminator to judge the authenticity of the first generated image can be improved.

[0095] During the training of the first generator and the first discriminator, the first generator and the first discriminator have opposite training objectives and opposite losses. The training objective of the first generator is to generate a first generated image that is similar to the pixelated version of each original image, such that the first discriminator cannot distinguish it. The training objective of the first discriminator is to accurately distinguish the first generated image generated by the first generator from the corresponding pixelated image.

[0096] During the training process of the second generator and the second discriminator, the electronic device invokes the second generator to generate a second generated image corresponding to each original image based on the pixelated image corresponding to the original image. The electronic device calculates a third loss value for the second generator based on the difference between each original image and the corresponding second generated image, and adjusts the second generator based on the third loss value. The electronic device invokes the second discriminator to perform authentication on each second generated image, outputs a second authentication result, calculates a fourth loss value for the second discriminator based on the difference between the second authentication result and the second preset label, and adjusts the second discriminator based on the fourth loss value. The second authentication result may be a label indicating whether the input second generated image is true or false. For example, when 1 is used to indicate authentication as true (authentication of the second generated image not being generated by the second generator) and 0 is used to indicate authentication as false (authentication of the second generated image being generated by the second generator), the second authentication result may be 0 or 1, and the second preset label may be 0. The preset conditions can be set voluntarily and are not limited in this application. Since the second preset label indicates a label for authentication as false, when calculating the fourth loss value for the second discriminator based on the second preset label, the difference between the second authentication result and the second preset label may be used as a basis for measuring the accuracy of the second discriminator's judgment. By minimizing this difference and adjusting the parameters of the second discriminator, the ability of the second discriminator to judge the authenticity of the second generated image can be improved.

[0097] During the training of the second generator and the second discriminator, the second generator and the second discriminator have opposite training objectives and opposite losses. The training objective of the second generator is to generate a second generated image similar to the original image based on the pixelated image corresponding to each original image, so that the second discriminator cannot distinguish the second generated image. The training objective of the second discriminator is to accurately distinguish the second generated image generated by the second generator from the corresponding original image.

[0098] During the alternating training process of the first generator and the second generator, the electronic device calls the first generator to convert each second generated image generated by the second generator to obtain a third generated image corresponding to each pixelated image, calculates the fifth loss value based on each pixelated image and the corresponding third generated image, and adjusts the first generator based on the fifth loss value. The electronic device calls the second generator to convert each first generated image generated by the first generator to obtain a fourth generated image corresponding to each original image. The electronic device calculates the sixth loss value based on each original image and the corresponding fourth generated image, and adjusts the second generator based on the sixth loss value.

[0099] Among them, when all loss values ​​meet the corresponding preset conditions, no further adjustment is made, and the adjusted first generator is determined as the first generator model, and the adjusted second generator is determined as the second generator model. The trained image translation neural network model may include the first generator model and the second generator model. The preset conditions can be set arbitrarily, and this application does not impose any restrictions on this. For example, the preset condition may be that the loss value no longer changes, or that the amplitude of change between multiple consecutive loss values ​​is within a preset range. The preset range can be set arbitrarily, and this application does not impose any restrictions on symmetry.

[0100] In this embodiment, the first generator is in opposite directions to the second generator, and the first discriminator is in opposite directions to the second discriminator. Such a structure can realize pixelation (Pixelization) and depixelation (Depixelization). By training the first generator, the trained first generator model can learn the pixelation mapping relationship. When a static image or a dynamic image needs to be pixelated, the electronic device can call the first generator model to perform pixelation processing on the static image or the dynamic image to obtain a static image or a dynamic image after pixelation processing. By training the second generator, the trained second generator model can learn the depixelation mapping relationship. When a pixelated image needs to be depixelated, the electronic device can call the second generator model to perform depixelation processing on the pixelated image to obtain an image after depixelation processing.

[0101] Figure 8 shows a functional block diagram of a lighting effect control device according to one embodiment of the present application. The lighting effect control device 11 includes an acquisition unit 110, a generation unit 111, and a control unit 112. A module / unit as referred to herein refers to a series of computer-readable instruction segments that can be acquired by the processor 103 in Figure 9 and perform a fixed function, and is stored in the memory 102 in Figure 9. The functions of each module / unit in this embodiment will be described in detail in subsequent embodiments.

[0102] The acquiring unit 110 is configured to acquire user information.

[0103] In some embodiments of the present application, the user information includes at least one of text, video, image, and sound.

[0104] The generating unit 111 is configured to generate a static image or a dynamic image according to user information.

[0105] In some embodiments of the present application, if the user information includes theme information, the generation unit 111 is further used to generate a static image or dynamic image of the theme corresponding to the theme information according to a preset style. If the user information includes theme information and style information, the generation unit 111 is further used to generate a static image or dynamic image of the theme corresponding to the theme information according to the style corresponding to the style information.

[0106] In some embodiments of the present application, the generation unit 111 is further configured to process user information based on artificial intelligence content generation technology to generate static images or dynamic images.

[0107] In some embodiments of the present application, the generating unit 111 is further configured to input user information into a diffusion model to generate a static image or a dynamic image.

[0108] The control unit 112 is configured to control the lighting effects of the lighting device according to static images or dynamic images.

[0109] In some embodiments of the present application, the control unit 112 is further configured to perform pixelation processing on a static image or a dynamic image, and control a lighting effect of a lighting device according to the pixelated static image or dynamic image.

[0110] In some embodiments of the present application, the control unit 112 is further configured to perform pixelation processing on a static image or a dynamic image based on an image translation neural network model.

[0111] In some embodiments of the present application, the control unit 112 is further used to control the lighting effect of each lighting device or each light-emitting element in the lighting device according to the pixel value of each pixel point in the static image, or to control the lighting effect of each lighting device or each light-emitting element in the lighting device according to the pixel values ​​of multiple pixel points in the static image.

[0112] In some embodiments of the present application, the dynamic image includes multiple frames of static images, and the control unit 112 is further used to control the lighting effect of each lighting device or each light-emitting element in the lighting device according to the pixel value of each pixel point of each static image frame in the dynamic image, or to control the lighting effect of each lighting device or each light-emitting element in the lighting device according to the pixel values ​​of multiple pixel points of each static image frame in the dynamic image.

[0113] Figure 9 is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. As shown in Figure 9, the electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102, and the input / output interface 104 via the bus 105.

[0114] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB) and controller area network (CAN). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0115] The memory 102 may include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs). The RAM can be directly read and written by the processor 103 and can be used to store executable programs (e.g., machine instructions) of other running programs, as well as user and application data. The RAM may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0116] The non-volatile memory can also store executable programs and user and application data, etc., which can be pre-loaded into the random access memory for direct reading and writing by the processor 103. The non-volatile memory can include disk storage devices and flash memory.

[0117] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 103, the lighting effect control method executed on the electronic device 10 can be implemented.

[0118] In other embodiments, the electronic device 10 shown in FIG. 9 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0119] The processor 103 may include one or more processing units. For example, the processor 103 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0120] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned lighting effect control method.

[0121] The input / output interface 104 is used to provide a channel for user input or output. For example, the input / output interface 104 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0122] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the input / output interface 104 in the electronic device 10 .

[0123] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0124] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0125] The computer-readable storage medium may be the internal memory of the electronic device described in the above embodiments, such as the hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the electronic device.

[0126] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, applications required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0127] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0128] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0129] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0130] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0131] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for controlling lighting effects, characterized in that, The method includes: Obtaining user information; Generating a static image or a dynamic image according to the user information; Controlling the lighting effect of the lighting device according to the static image or the dynamic image.

2. The lighting effect control method according to claim 1, wherein, The user information includes at least one of text, video, image, and sound.

3. The lighting effect control method according to claim 1, characterized in that, The generating a static image or a dynamic image according to the user information includes: If the user information includes theme information, generating a static image or a dynamic image of the corresponding theme of the theme information according to a preset style; If the user information includes theme information and style information, generating a static image or a dynamic image of the corresponding theme of the theme information according to the style corresponding to the style information.

4. The lighting effect control method according to claim 1 or 3, characterized in that The generating a static image or a dynamic image according to the user information includes: Processing the user information based on artificial intelligence to generate content technology to generate the static image or the dynamic image.

5. The lighting effect control method according to claim 4, characterized in that The processing the user information based on artificial intelligence to generate content technology to generate the static image or the dynamic image includes: Inputting the user information into a diffusion model to generate the static image or the dynamic image.

6. The lighting effect control method according to claim 1, characterized in that, The method further includes: Performing pixelation processing on the static image or the dynamic image, and controlling the lighting effect of the lighting device according to the pixelated static image or dynamic image.

7. The lighting effect control method according to claim 6, characterized in that, The performing pixelation processing on the static image or the dynamic image includes: Performing pixelation processing on the static image or the dynamic image based on an image translation neural network model.

8. The lighting effect control method according to claim 1 or 7, characterized in that, The controlling the lighting effect of the lighting device according to the static image includes: Controlling the lighting effect of each lighting device or each light-emitting element in the lighting device according to the pixel value of each pixel point in the static image; or Controlling the lighting effect of each lighting device or each light-emitting element in the lighting device according to the pixel values of multiple pixel points in the static image.

9. A lighting effect control device, characterized in that, The lighting effect control device includes: An obtaining unit for obtaining user information; A generating unit for generating a static image or a dynamic image according to the user information; A controlling unit for controlling the lighting effect of the lighting device according to the static image or the dynamic image.

10. An electronic device, characterized in that, The electronic device includes: A memory storing at least one instruction; and A processor for executing the at least one instruction to implement the lighting effect control method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: At least one instruction is stored in the computer-readable storage medium, and when the at least one instruction is executed by a processor in an electronic device, the lighting effect control method according to any one of claims 1 to 8 is implemented.

Images