Image light effect processing method and device, readable storage medium and program product

Abstract

The application relates to the technical field of image processing, and provides an image light effect processing method and device, a readable storage medium and a program product. The method comprises the following steps: acquiring a reference image pair showing a target light effect change; the target light effect is a light effect to be added to a target image; the reference image pair comprises a first reference image without the target light effect and a second reference image with the target light effect; first light effect feature information is obtained according to the first reference image, and second light effect feature information is obtained according to the second reference image; the first light effect feature information and the second light effect feature information are subjected to light effect decoupling to obtain light effect embedding information generated due to the addition of the target light effect; and the target image is subjected to light effect addition processing according to the light effect embedding information and a light effect description text of the second reference image, so that the target image with the target light effect is obtained. By adopting the method, the target light effect can be captured and reproduced more robustly, and the visual authenticity and detail expressiveness of image light effect processing can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for processing the light effect of an image. Background Technology

[0002] The rapid development of digital content creation has spurred the demand for efficient, low-cost, and highly realistic visual tools. Advances in generative artificial intelligence technology and diffusion models in understanding image content and lighting logic have made it possible to intelligently simulate and adjust image lighting, thereby driving the emergence of image lighting effects processing technology, popularizing professional-grade lighting control, and significantly improving the efficiency and creative space of visual creation.

[0003] Currently, image lighting effect processing can be performed based on diffusion models, treating it as a conditional image generation task. Guided by conditions such as text, normal maps, or illumination intensity, the target image is gradually generated from noise. However, diffusion-based image lighting effect processing methods struggle to accurately and unambiguously model complex lighting effects, leading to insufficient expressive power and consequently affecting the final image lighting effect processing results. Summary of the Invention

[0004] Based on this, this application provides an image light effect processing method, apparatus, computer device, computer-readable storage medium, and computer program product, which can more robustly capture and reproduce target light effects, making the light and shadow transition of the target image with added target light effects natural and the details well preserved, effectively improving the visual realism and detail expression of image light effect processing.

[0005] On the one hand, this application provides a method for processing the light effect of an image, including:

[0006] Obtain a pair of reference images that display the changes in the target lighting effect; the target lighting effect is the lighting effect to be added to the target image; the pair of reference images includes a first reference image without the target lighting effect and a second reference image with the target lighting effect added.

[0007] First light effect feature information is obtained based on the first reference image, and second light effect feature information is obtained based on the second reference image;

[0008] The first light effect feature information and the second light effect feature information are decoupled to obtain the light effect embedding information generated by adding the target light effect;

[0009] Based on the light effect embedding information and the light effect description text of the second reference image, the target image is processed to add light effects, resulting in a target image with the added target light effects.

[0010] In one embodiment, obtaining the first light effect feature information based on the first reference image includes:

[0011] The first reference image is divided into multiple image blocks, and the multiple image blocks are combined to obtain a first reference image block sequence;

[0012] The first reference image block sequence is input into the visual encoder to obtain the first light effect feature information.

[0013] In one embodiment, the step of decoupling the first light effect feature information and the second light effect feature information to obtain the light effect embedding information generated by adding the target light effect includes:

[0014] The first-layer light effect adaptation unit of the light effect adapter decouples the first light effect feature information and the second light effect feature information to obtain the first-layer enhanced initial light effect embedding information.

[0015] The non-first-layer light effect adaptation unit of the light effect adapter decouples the enhanced initial light effect embedding information of the previous layer to obtain the enhanced initial light effect embedding information of the non-first layer.

[0016] Based on the initial light effect embedding information after the final layer enhancement, the light effect embedding information generated by adding the target light effect is obtained.

[0017] In one embodiment, each layer of the light effect adaptation unit includes a self-attention module, a cross-attention module, and a feedforward network module. The first-layer light effect adaptation unit of the light effect adapter decouples the first light effect feature information and the second light effect feature information to obtain the first-layer enhanced initial light effect embedding information, including:

[0018] The second light effect feature information is processed by the self-attention module to obtain the light effect feature information after self-attention processing.

[0019] The cross-attention module aligns and fuses the self-attention processed light effect feature information and the first light effect feature information to obtain cross-attention feature information, thereby obtaining the initial light effect embedding information generated by adding the target light effect.

[0020] The feedforward network module performs nonlinear transformation and feature enhancement on the initial light effect embedding information to obtain the first-layer enhanced initial light effect embedding information.

[0021] In one embodiment, obtaining the light effect description text of the second reference image includes:

[0022] The second reference image and preset prompt words are input into the visual language model to obtain the output of the visual language model; the preset prompt words are used to instruct the visual language model to describe the lighting effect of the second reference image.

[0023] Based on the output of the visual language model, the light effect description text of the second reference image is obtained.

[0024] In one embodiment, the step of adding light effects to the target image based on the light effect embedding information and the light effect description text of the second reference image to obtain a target image with the added target light effect includes:

[0025] The light effect description text is encoded to obtain light effect text features;

[0026] Based on the light effect embedding information and the light effect text features, the target image is processed to add light effects, resulting in a target image with the added target light effects.

[0027] In one embodiment, the step of adding light effects to the target image based on the light effect embedding information and the light effect text features to obtain a target image with the added target light effect includes:

[0028] The target image is context-encoded using a context embedder in the diffusion generation model to obtain image context features;

[0029] By using the context block in the diffusion generation model, the light effect embedding information, the light effect text features, and the image context features are fused to obtain the light effect condition features;

[0030] The target image is subjected to latent space compression and noise addition processing by using the variational autoencoder in the diffusion generation model to obtain noisy latent variables;

[0031] By using the diffusion backbone network in the diffusion generation model, the noisy latent variables are denoised in multiple steps under the guidance of the light effect condition features, resulting in a target image with the added target light effect.

[0032] On the one hand, this application also provides an image light effect processing apparatus, including:

[0033] The reference image pair acquisition module is used to acquire a reference image pair that displays the change of the target light effect; the target light effect is the light effect to be added to the target image; the reference image pair includes a first reference image without the target light effect and a second reference image with the target light effect added.

[0034] The light effect feature information acquisition module is used to obtain first light effect feature information based on the first reference image and second light effect feature information based on the second reference image;

[0035] The light effect embedding information acquisition module is used to decouple the first light effect feature information and the second light effect feature information to obtain the light effect embedding information generated by adding the target light effect.

[0036] The light effect addition processing module is used to perform light effect addition processing on the target image according to the light effect embedding information and the light effect description text of the second reference image, so as to obtain a target image with the target light effect added.

[0037] In another aspect, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the described method.

[0038] In one aspect, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, the computer program being executed by a processor using the methods described above.

[0039] In one aspect, this application also provides a computer program product. The computer program product includes a computer program that is executed by a processor using the methods described above.

[0040] The aforementioned image light effect processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire a pair of reference images displaying changes in target light effects; the target light effect is the light effect to be added to a target image; the reference image pair includes a first reference image without the target light effect added and a second reference image with the target light effect added; first light effect feature information is obtained based on the first reference image, and second light effect feature information is obtained based on the second reference image; light effect decoupling is performed on the first light effect feature information and the second light effect feature information to obtain light effect embedding information generated by adding the target light effect; light effect adding processing is performed on the target image based on the light effect embedding information and the light effect description text of the second reference image to obtain a target image with the target light effect added. This application uses a pair of reference images with and without target lighting effects to decouple and extract the target lighting effect. Because the reference image pair can provide richer, more intuitive and unambiguous lighting information, it can significantly improve the accuracy of lighting effect decoupling and the precision of conditional guidance. It can more robustly capture and reproduce the target lighting effect, making the light and shadow transition of the target image with added target lighting effect natural and the details well preserved, effectively improving the visual realism and detail expression of image lighting effect processing. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is an application environment diagram of an image light effect processing method in one embodiment;

[0043] Figure 2 This is a flowchart illustrating an image light effect processing method in one embodiment;

[0044] Figure 3 This is a schematic diagram of the algorithm framework for a video light effect processing method in one embodiment;

[0045] Figure 4 This is a structural block diagram of an image light effect processing device in one embodiment;

[0046] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] The image light effect processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 can acquire a pair of reference images displaying changes in target lighting effects; obtain first lighting effect feature information based on the first reference image, and second lighting effect feature information based on the second reference image; decouple the first and second lighting effect feature information to obtain lighting effect embedding information resulting from the addition of target lighting effects; and perform lighting effect addition processing on the target image based on the lighting effect embedding information and the lighting effect description text of the second reference image to obtain a target image with added target lighting effects.

[0049] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0050] In one exemplary embodiment, such as Figure 2 As shown, a method for processing the light effect of an image is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps S202 to S208. Wherein:

[0051] Step S202: Obtain a pair of reference images showing the change in target light effect; the target light effect is the light effect to be added to the target image; the pair of reference images includes a first reference image without the target light effect and a second reference image with the target light effect added.

[0052] Several reference images can be acquired, and at least one lighting effect can be added to each reference image to obtain a corresponding reference image with added lighting effects. For distinction, the original reference image without added lighting effects can be called the first reference image, and the reference image with added lighting effects can be called the second reference image. That is, the second reference image is obtained by adding lighting effects to the first reference image. These two reference images can be considered a pair, resulting in a reference image pair that exhibits corresponding lighting effect changes. Through this method, several reference image pairs can be obtained, and different reference image pairs can correspond to different lighting effect changes.

[0053] The target image can be the image to be processed with light effects; the user can add light effects to the target image, which can be called the target light effect. From several pairs of reference images, a pair of reference images that demonstrate the change in the target light effect can be determined. This pair of reference images includes a first reference image without the target light effect and a second reference image with the target light effect added.

[0054] Step S204: Obtain first light effect feature information based on the first reference image, and obtain second light effect feature information based on the second reference image.

[0055] Based on the visual representation capabilities of the visual encoder, the first reference image can be encoded into high-dimensional features to obtain the first light effect feature information.

[0056] Based on the visual representation capabilities of the visual encoder, the second reference image can be encoded into high-dimensional features to obtain the second light effect feature information.

[0057] The visual encoder can be a SigLIP visual encoder. SigLIP stands for Sigmoid Loss for Language-Image Pre-training.

[0058] Step S206: Decouple the first light effect feature information and the second light effect feature information to obtain the light effect embedding information generated by adding the target light effect.

[0059] A light adapter can be specifically designed for light effect control. This light adapter includes multiple layers of light effect adaptation units, each of which includes a self-attention module, a cross-attention module, and a feedforward network module.

[0060] Based on the aforementioned light effect adapter, the first light effect feature information and the second light effect feature information can be decoupled to obtain the light effect embedding information generated by adding the target light effect.

[0061] Step S208: Based on the light effect embedding information and the light effect description text of the second reference image, perform light effect addition processing on the target image to obtain a target image with added target light effects.

[0062] The light effect text features can be obtained based on the light effect description text of the second reference image.

[0063] Light effect embedding information, light effect text features, and target image can be input into the diffusion generation model. The diffusion generation model can then perform precise embedding guidance and auxiliary text guidance based on the light effect embedding information and light effect text features to perform diffusion generation processing on the target image. The decoupled light effect embedding information is stably applied to the target image to obtain a target image with added target light effects.

[0064] In the above image lighting effect processing method, a reference image pair with and without target lighting effect is used to decouple and extract the target lighting effect. Since the reference image pair can provide richer, more intuitive and unambiguous lighting information, it can significantly improve the accuracy of lighting effect decoupling and the precision of condition guidance. It can more robustly capture and reproduce the target lighting effect, making the light and shadow transition of the target image with added target lighting effect natural and the details well preserved, effectively improving the visual realism and detail expression of image lighting effect processing.

[0065] In one embodiment, the first light effect feature information is obtained based on the first reference image. The specific steps are as follows: the first reference image is divided into multiple image blocks, and the multiple image blocks are combined to obtain a first reference image block sequence; the first reference image block sequence is input into a visual encoder to obtain the first light effect feature information.

[0066] Before inputting the first reference image into the visual encoder, the first reference image can be preprocessed, for example, by resizing the first reference image to a size acceptable to the visual encoder (e.g., 384x384 pixels) and dividing the first reference image into multiple image patches.

[0067] The first reference image block sequence can be input into the visual encoder so that the visual encoder encodes the first reference image block sequence into a feature vector that includes rich semantic information, and the above feature vector can be used as the first light effect feature information.

[0068] In this embodiment, the first reference image is divided into multiple image blocks to obtain a first reference image block sequence; the first reference image block sequence is input into a visual encoder to obtain first light effect feature information, which can retain more spatial information.

[0069] In one embodiment, the second light effect feature information is obtained based on the second reference image. The specific steps are as follows: the second reference image is divided into multiple image blocks, and the multiple image blocks are combined to obtain a second reference image block sequence; the second reference image block sequence is input into a visual encoder to obtain the second light effect feature information.

[0070] Before inputting the second reference image into the visual encoder, the second reference image can be preprocessed, for example, by resizing the second reference image to a size acceptable to the visual encoder (e.g., 384x384 pixels) and dividing the second reference image into multiple image patches.

[0071] The second reference image block sequence can be input into the visual encoder so that the visual encoder encodes the second reference image block sequence into a feature vector that includes rich semantic information, and the above feature vector can be used as the second light effect feature information.

[0072] In this embodiment, the second reference image is divided into multiple image blocks to obtain a second reference image block sequence; the second reference image block sequence is input into a visual encoder to obtain second light effect feature information, which can retain more spatial information.

[0073] In one embodiment, the first light effect feature information and the second light effect feature information are decoupled to obtain the light effect embedding information generated by adding the target light effect. The specific steps are as follows: the first light effect feature information and the second light effect feature information are decoupled to obtain the first-layer enhanced initial light effect embedding information through the first-layer light effect adaptation unit of the light effect adapter; the previous-layer enhanced initial light effect embedding information is decoupled to obtain the non-first-layer enhanced initial light effect embedding information through the non-first-layer light effect adaptation unit of the light effect adapter; and the light effect embedding information generated by adding the target light effect is obtained based on the last-layer enhanced initial light effect embedding information.

[0074] The first and second light effect feature information can be spliced ​​together to obtain the spliced ​​light effect feature information.

[0075] The first-layer light effect adaptation unit of the light effect adapter can perform feature interaction, alignment, and fusion processing on the spliced ​​light effect feature information to obtain the first-layer enhanced initial light effect embedding information. Similarly, the non-first-layer enhanced initial light effect embedding information can be obtained by performing feature interaction, alignment, and fusion processing on the previous-layer enhanced initial light effect embedding information through the non-first-layer light effect adaptation unit of the light effect adapter.

[0076] The initial light effect embedding information after the final layer enhancement can be pooled to obtain the light effect embedding information generated by adding the target light effect.

[0077] In this embodiment, the first and second light effect feature information are decoupled by the multi-layer light effect adaptation unit of the light effect adapter. The understanding of the target light effect change is gradually deepened by the iterative structure of the multi-layer light effect adaptation unit, which can significantly improve the accuracy of light effect decoupling.

[0078] In one embodiment, each layer of the light effect adaptation unit includes a self-attention module, a cross-attention module, and a feedforward network module. The first-layer light effect adaptation unit of the light effect adapter decouples the first and second light effect feature information to obtain the first-layer enhanced initial light effect embedding information. The specific steps are as follows: The second light effect feature information is processed by the self-attention module to obtain the self-attention processed light effect feature information; the self-attention processed light effect feature information and the first light effect feature information are aligned and fused by the cross-attention module to obtain cross-attention feature information, thus obtaining the initial light effect embedding information generated by adding the target light effect; the initial light effect embedding information is subjected to nonlinear transformation and feature enhancement by the feedforward network module to obtain the first-layer enhanced initial light effect embedding information.

[0079] The self-attention module of the first-layer light effect adaptation unit can perform feature interaction processing on the second light effect feature information to obtain the light effect feature information after self-attention processing. This can identify the pattern and structure of the target light effect state, possibly highlighting the effect of global application or consistent shadow direction, and help integrate features related to the consistent effect of the target light effect in the target image.

[0080] The query vector can be obtained through the cross-attention module of the first-layer light effect adaptation unit based on the light effect feature information after self-attention processing; the key vector and value vector can be obtained based on the first light effect feature information; the attention weight can be obtained based on the correlation between the query vector and the key vector; the value vector can be weighted and fused based on the attention weight to obtain the cross-attention feature; and the initial light effect embedding information generated by adding the target light effect can be obtained based on the cross-attention feature information.

[0081] Through the aforementioned cross-attention module, the state of the reference image with and without target lighting effects can be clearly compared. By focusing on the first and second lighting effect feature information, the content that changes due to the change in target lighting effects can be identified, as well as the underlying scene content (which exists in both the first and second lighting effect feature information). This decouples the target lighting effects from the image content, significantly improving the accuracy of lighting effect decoupling.

[0082] The initial light effect embedding information can be nonlinearly transformed and feature-enhanced by the feedforward network module of the first-layer light effect adaptation unit to obtain the first-layer enhanced initial light effect embedding information, which can improve the feature representation ability of the initial light effect embedding information.

[0083] In this embodiment, the second light effect feature information is subjected to feature interaction processing to obtain the light effect feature information after self-attention processing; the light effect feature information after self-attention processing and the first light effect feature information are aligned and fused to obtain the initial light effect embedding information; the initial light effect embedding information is subjected to nonlinear transformation and feature enhancement to obtain the first-layer enhanced initial light effect embedding information, which can improve the accuracy of light effect decoupling.

[0084] In one embodiment, each layer of the light effect adaptation unit includes a self-attention module, a cross-attention module, and a feedforward network module. The non-first-layer light effect adaptation unit of the light effect adapter decouples the enhanced initial light effect embedding information from the previous layer to obtain the non-first-layer enhanced initial light effect embedding information. The specific steps are as follows: The self-attention module performs feature interaction processing on the second light effect feature information in the enhanced initial light effect embedding information from the previous layer to obtain the self-attention processed light effect feature information; the cross-attention module aligns and fuses the self-attention processed light effect feature information and the first light effect feature information in the enhanced initial light effect embedding information from the previous layer to obtain cross-attention feature information, thus obtaining the initial light effect embedding information generated by adding the target light effect; the feedforward network module performs nonlinear transformation and feature enhancement on the initial light effect embedding information to obtain the non-first-layer enhanced initial light effect embedding information.

[0085] The self-attention module of the non-first-layer light effect adaptation unit can perform feature interaction processing on the second light effect feature information in the enhanced initial light effect embedding information of the previous layer to obtain the light effect feature information after self-attention processing. This can identify the pattern and structure of the target light effect state, possibly highlighting the effect of global application or consistent shadow direction, and help integrate features related to the consistent effect of the target light effect in the target image.

[0086] The query vector can be obtained through the cross-attention module of the non-first-layer light effect adaptation unit based on the light effect feature information after self-attention processing; the key vector and value vector can be obtained based on the first light effect feature information in the enhanced initial light effect embedding information of the previous layer; the attention weight can be obtained based on the correlation between the query vector and the key vector; the value vector can be weighted and fused based on the attention weight to obtain the cross-attention feature; and the initial light effect embedding information generated by adding the target light effect can be obtained based on the cross-attention feature information.

[0087] Through the aforementioned cross-attention module, the state of the reference image with and without target lighting effects can be clearly compared. By focusing on the first and second lighting effect feature information, the content that changes due to the change in target lighting effects can be identified, as well as the underlying scene content (which exists in both the first and second lighting effect feature information). This decouples the target lighting effects from the image content, significantly improving the accuracy of lighting effect decoupling.

[0088] The initial light effect embedding information can be nonlinearly transformed and feature-enhanced by the feedforward network module of the non-first-layer light effect adaptation unit to obtain the non-first-layer enhanced initial light effect embedding information, which can improve the feature representation ability of the initial light effect embedding information.

[0089] In this embodiment, the second light effect feature information in the enhanced initial light effect embedding information of the previous layer is subjected to feature interaction processing to obtain the light effect feature information after self-attention processing; the light effect feature information after self-attention processing and the first light effect feature information in the enhanced initial light effect embedding information of the previous layer are aligned and fused to obtain the initial light effect embedding information; the initial light effect embedding information is subjected to nonlinear transformation and feature enhancement to obtain the initial light effect embedding information after non-first layer enhancement, which can improve the accuracy of light effect decoupling.

[0090] In one embodiment, the light effect description text of the second reference image is obtained by the following steps: inputting the second reference image and preset prompt words into the visual language model to obtain the output result of the visual language model; the preset prompt words are used to instruct the visual language model to describe the lighting effect of the second reference image; and the light effect description text of the second reference image is obtained based on the output result of the visual language model.

[0091] Preset prompts can be set to instruct the visual language model to describe the lighting effects of the second reference image, depending on the specific situation. For example, the preset prompts could be: "Please describe in detail the lighting effects and atmosphere in the second reference image. Focus on the direction, intensity, and color of the light source, the characteristics of the shadows, the highlight areas, and the overall contrast and mood. Provide a detailed lighting analysis report in natural language."

[0092] A second reference image and preset prompts can be input into a visual language model, so that the visual language model can infer the lighting effect of the second reference image based on the preset prompts and output the result.

[0093] The output of the visual language model can be used as the light effect description text of the second reference image.

[0094] In this embodiment, by inputting the second reference image and preset prompt words into the visual language model, a relatively accurate description of the light effect can be quickly obtained.

[0095] In one embodiment, the target image is processed by adding light effects based on the light effect embedding information and the light effect description text of the second reference image to obtain a target image with added target light effects. The specific steps are as follows: the light effect description text is encoded to obtain light effect text features; the target image is processed by adding light effects based on the light effect embedding information and the light effect text features to obtain a target image with added target light effects.

[0096] The text describing the light effect can be input into the Contrastive Language-Image Pre-training Text Encoder (CLIP TextEncoder), which will then encode the text describing the light effect to obtain the light effect text features. CLIP stands for Contrastive Language-Image Pre-training Text Encoder.

[0097] Light effect embedding information, light effect text features, and target image can be input into the diffusion generation model. The diffusion generation model can then perform precise embedding guidance and auxiliary text guidance based on the light effect embedding information and light effect text features to perform diffusion generation processing on the target image. The decoupled light effect embedding information is stably applied to the target image to obtain a target image with added target light effects.

[0098] In this embodiment, the text describing the light effect is encoded to obtain the light effect text features. Based on the light effect embedding information and the light effect text features, the target image is processed to add light effects, resulting in a target image with added target light effects. This can significantly improve the accuracy of light effect decoupling and the precision of conditional guidance, and can more robustly capture and reproduce target light effects. The target image with added target light effects has natural light and shadow transitions and good detail retention, effectively improving the visual realism and detail expression of image light effect processing.

[0099] In one embodiment, the target image is processed to add light effects based on the light effect embedding information and light effect text features to obtain a target image with added target light effects. The specific steps are as follows: the target image is context-encoded by the context embedder in the diffusion generation model to obtain image context features; the light effect embedding information, light effect text features and image context features are fused by the context block in the diffusion generation model to obtain light effect condition features; the target image is latent space compressed and denoised by the variational autoencoder in the diffusion generation model to obtain denoised latent variables; the denoised latent variables are denoised in multiple steps by the diffusion backbone network in the diffusion generation model under the guidance of the light effect condition features to obtain a target image with added target light effects.

[0100] A diffusion generation model can be obtained based on the VACE DiT diffusion generation module. VACE stands for Video All-in-one Creation and Editing, and DiT stands for Diffusion Transformer.

[0101] The lighting effect embedding information, lighting effect text features, and target image input into the condition units of the diffusion generation model can be uniformly formatted. Specifically, the target image is used as the main visual input, the lighting effect embedding information is processed into condition tokens acceptable to the diffusion generation model, and the lighting effect text features are used as text condition tokens for the diffusion generation model.

[0102] A high-dimensional target image can be encoded into a compact, semantically rich context embedding vector by using a context embedder in a diffusion generative model, which serves as the image context feature.

[0103] The diffusion backbone network in the diffusion generation model can perform multi-step iterative denoising of noisy latent variables under the guidance of lighting effect condition features, generating noise-free latent variables. In multiple denoising steps, the lighting effect condition features continuously guide the diffusion backbone network, enabling it to not only restore the content and motion of the target image during denoising, but also simultaneously render the target lighting effect onto the target image.

[0104] Once the denoising process is complete, the generated noiseless latent variables can be converted back from the latent space to the pixel space using the decoder in the variational autoencoder to obtain the target image with the added target lighting effect.

[0105] In this embodiment, the embedded information of the light effect, the textual features of the light effect, and the image context features are fused to obtain the conditional features of the light effect. Guided by these conditional features, the noisy latent variables are denoised through multiple iterative steps to obtain the target image with the added target light effect. During the stepwise denoising process, precise embedding guidance and auxiliary text guidance can be provided based on the conditional features of the light effect, which helps to smoothly and harmoniously integrate the light effect into the structure and texture of the target image, thereby making the light and shadow transitions of the target image with the added target light effect natural and preserving good details.

[0106] The image lighting effect processing method of this application can process a single frame image or a collection of multiple frames arranged in a time sequence. In this case, the collection of images can be referred to as a video. When the processing object is a video, the image lighting effect processing method of this application can be referred to as a video lighting effect processing method, or a video relighting method.

[0107] Video lighting is a more challenging and valuable cutting-edge research area than image lighting. The video lighting task combines computer vision, computer graphics, and deep learning. Its core objective is to modify and edit the lighting effects in a video sequence while preserving the original content of the video (such as object identity, texture, scene structure, and original motion). Unlike static image lighting, video lighting not only needs to process the spatial lighting of each frame but also needs to ensure that these lighting changes are continuous and stable over time to avoid artifacts.

[0108] Once video lighting technology matures, it will revolutionize digital content creation and human-computer interaction. In the fields of film and visual effects (VFX), video lighting technology allows artists to arbitrarily change the atmosphere of a scene during post-production, such as changing daytime to nighttime, or more realistically integrating computer graphics (CG) effects into live-action footage. In augmented reality (AR), video lighting technology is key to achieving immersion, seamlessly blending virtual objects with the real environment by analyzing video stream lighting and applying it. Simultaneously, video lighting technology provides powerful post-editing tools for short video creators and ordinary users, allowing them to easily salvage videos with lighting defects. Furthermore, when training artificial intelligence (AI) models such as those for autonomous driving, video lighting technology can generate multiple versions of the same scene under different lighting conditions through data augmentation, greatly improving the model's robustness. In virtual reality (VR) and the metaverse, video lighting technology is also fundamental to building dynamic and realistic light and shadow interactions.

[0109] Despite its promising applications, video lighting remains an extremely challenging task. It inherits all the challenges of image lighting (such as ill-posed problems) and adds a highly complex temporal dimension. Temporal consistency is the most critical and significant challenge. Simply applying image lighting techniques frame-by-frame to video will result in catastrophic flickering artifacts due to minute prediction differences between adjacent frames. Secondly, data scarcity is severe, making it difficult to obtain paired multi-illuminated real-world video datasets that maintain absolute consistency between content and motion. Furthermore, video lighting tasks must address the complexity of dynamic scenes (such as moving cameras and objects), the ill-conditioned nature of video intrinsic decomposition (temporally consistent decoupling of geometry, materials, and lighting), the high real-time requirements of augmented reality applications, and the complex modeling challenges of global illumination (such as soft shadows and indirect bounces).

[0110] Solving the aforementioned problems to advance research in video lighting technology has profound theoretical and practical significance. First, video lighting technology is not merely a filter task; it compels academia to study how to understand the 4-dimensional (3-dimensional space and 1-dimensional time) world from 2D video. A perfect video lighting model is essentially a world model that understands physical optics. Second, it will significantly lower the barrier to entry for professional film and television production, unleashing unprecedented freedom of light and shadow for new media such as short videos and live streaming. Third, it is also a key enabling technology for achieving immersion in augmented reality / virtual reality and the metaverse, representing the last visual mile connecting reality and virtuality. Finally, as one of the most challenging tasks in controllable video generation, research on video lighting technology will drive AI generation models from "being able to generate" to "being able to understand physical laws and precisely control them."

[0111] The mainstream solutions for video lighting (and its foundation—image lighting) and their limitations can be broadly categorized into the following four types:

[0112] The first category is video lighting methods guided by physical priors. These methods attempt to simulate the physical processes of lighting in real-world scenarios. They typically require estimating or reconstructing the geometry of the scene (such as normal maps and depth maps), then setting virtual light sources and rendering based on optical models. However, these methods are limited in that they restrict the types of lighting effects and struggle to represent natural light or lighting in complex areas. Furthermore, these methods essentially add lighting, making it difficult to effectively remove original shadows. And because 3D geometry estimation is inherently challenging, estimation errors can easily lead to a texture-like appearance and a lack of realism.

[0113] The second category is video lighting methods based on simplified parameter decoupling. These methods attempt to bypass complex 3D reconstruction by analyzing images to extract simplified descriptors such as light intensity, direction, or low-order spherical harmonics, and then directly modify pixel values. The problem with these methods is that the lighting modeling is too simplistic, making it difficult to capture rich phenomena such as soft shadows and specular highlights, resulting in poor applicability. These methods can also only handle simple lighting effects, and due to the lack of physical constraints, they are prone to damaging the original image structure, leading to severe distortion or artifacts.

[0114] The third type of approach employs a two-stage deep learning model for video lighting, separating illumination estimation from image rendering. This is a common deep learning paradigm, where one network estimates the illumination representation (such as spherical harmonic coefficients), and another network renders the image based on these parameters. The challenge of this approach lies in the fact that illumination decoupling is inherently an ill-posed problem, and the estimated parameters may be inaccurate. Furthermore, when modifying lighting, the rendering network can easily smooth out or lose high-frequency details such as skin texture and hair strands. The spherical harmonic coefficients are abbreviated as SH Coefficients.

[0115] The fourth type of approach utilizes powerful diffusion models as generators for video lighting. This type of method treats lighting as a conditional image generation task, progressively generating the target image from noise under the guidance of conditions such as text, normal maps, or light intensity. The inherent bottleneck of this approach is that traditional control methods (text, normal maps, etc.) are often insufficient to accurately and unambiguously model complex lighting effects. Furthermore, diffusion models are highly sensitive to the consistency of guiding conditions; when control conditions (such as text and normal maps) conflict, the diffusion model struggles to effectively integrate them, easily leading to chaotic or unexpected results.

[0116] To address the aforementioned technical problems, this embodiment provides a method for processing video lighting effects. The algorithm framework diagram of this method is shown below. Figure 3 As shown, the process of this method includes two parallel control signal extraction branches and a core video generation module.

[0117] The first branch is the lighting effect decoupling path, which includes a SigLIP visual encoder (which can be referred to as module 1) and a lighting effect decoupler (which can be referred to as module 2). The input to the SigLIP visual encoder is a pair of reference images showing changes in the target lighting effect. This pair of reference images includes a first reference image (img_a) without the target lighting effect and a second reference image (img_A) with the target lighting effect added. The function of the SigLIP visual encoder is to utilize the powerful visual representation capabilities of SigLIP to encode the first and second reference images into high-dimensional features, respectively, and output first lighting effect feature information and second lighting effect feature information (i.e., feature information before and after lighting). Then, the first and second lighting effect feature information are fed into the lighting effect decoupler.

[0118] The light effect decoupler can receive first light effect feature information and second light effect feature information. Through comparison and analysis, it decouples and extracts the light effect embedding information (E) purely caused by changes in illumination. light And it outputs the light effect embedded information as a precise conditional control signal.

[0119] The second branch is the auxiliary semantic guidance pathway, which includes the Vision-Language Model (VLM) extraction module (which can be called module 4). This module takes a second reference image as input and uses the capabilities of the visual language model to analyze the second reference image (which can be called the target effect image) and extract the light effect description text of the second reference image (e.g., "soft sidelight" or "twilight atmosphere").

[0120] For example, the preset prompt could be: Please describe in detail the lighting effects and atmosphere in the second reference image. Focus on the direction, intensity, and color of the light source, the characteristics of the shadows, the highlight areas, and the overall contrast and mood. Provide a detailed lighting analysis report in natural language. The preset prompt can be called a guiding text keyword.

[0121] The light effect description text can be input into the contrastive language-image pre-trained text encoder (CLIP TextEncoder) to encode the light effect description text, resulting in light effect text features. These light effect text features can then be used as one of the control signals and fed into VACE to guide video generation.

[0122] The outputs from the two branches, along with the target video to be lit, converge at the VACE DiT diffusion generation module (which can be referred to as module 3). This module is the core video processor, receiving three inputs: an input video frame sequence (i.e., the target video to be lit), lighting effect embedding information from module 2, and lighting effect description text from module 4. The function of module 3 is to perform diffusion generation processing on the input video frame sequence based on the VACE architecture, utilizing both precise embedded guidance and auxiliary text guidance, to stably and temporally apply the decoupled lighting effect to the target video. The output of this module is the resulting video frame sequence after lighting.

[0123] The SigLip (Sigmoid Loss for Language-Image Pre-Training) visual encoder is a model for visual-language pre-training. Similar to the Contrastive Language-Image Pre-Training (CLIP) model, the core objective of the SigLip visual encoder is to learn powerful visual representations, encoding images into feature vectors that capture their rich semantic information. SigLip's main innovation lies in the loss function used in its pre-training phase. Traditional Contrastive Language-Image Pre-Training models employ contrastive learning methods, typically using the Information Noise Contrastive Estimation (InfoNCE) loss function. This function relies on distinguishing correct image-text pairs (positive samples) from a large number of incorrect pairs (negative samples) within a batch, and normalizes this loss using a softmax function. While this method is effective, it is very sensitive to the batch size, requiring very large batches to contain enough negative samples, and the temperature hyperparameter needs to be carefully tuned.

[0124] Before being input into the SigLIP visual encoder, the reference image pair needs to be preprocessed, such as resizing it to a size acceptable to the SigLIP visual encoder (e.g., 384x384 pixels) and segmenting it into multiple image patches to fit its underlying VisionTransformer (ViT) architecture. The output of the SigLip visual encoder is first and second lighting effect feature information. The lighting effect feature information is image feature vectors or image embeddings, which are high-dimensional vectors (e.g., 768-dimensional) designed to capture the rich semantic information and visual content of the reference image pair.

[0125] A light effect adapter can be specifically designed for light effect control. The core function of this adapter is to receive and process combined feature information from the SigLip visual encoder. Specifically, the adapter receives the concatenated light effect feature information, obtained by concatenating the first and second light effect feature information extracted from the first and second reference images, respectively. Through the adapter's internal structure, the concatenated light effect feature information undergoes in-depth processing and analysis to precisely decouple and generate light effect embedding information (which can be called a light effect condition vector).

[0126] Among them, the spliced ​​light effect feature information is an image patch embedding sequence, which retains more spatial information than using a single global embedding vector for each reference image.

[0127] The core structure of the lighting adapter consists of five stacked lighting adaptation units. Each layer includes a self-attention module, a cross-attention module, and a feedforward network module, performing self-attention, cross-attention, and feedforward operations. This iterative structure allows the lighting adapter to progressively deepen its understanding of changes in illumination.

[0128] Within each lighting effect adaptation unit, the self-attention module allows different parts of the second lighting effect feature information corresponding to the second reference image to interact with each other. This helps to integrate features related to the consistent effect produced by new illumination in the second reference image. It can identify the patterns and structure of the target lighting effect state itself, potentially highlighting the effect of a globally applied effect or a consistent shadow direction.

[0129] The cross-attention module is a key module for extracting differences introduced by lighting effects. Here, the second lighting effect feature information corresponding to the second reference image serves as the query vector, while the first lighting effect feature information corresponding to the first reference image, which provides key and value vectors, is attended to. This step allows the lighting effect adapter to compare the image's state after lighting with its state before lighting. By attending to both the first and second lighting effect feature information, it can identify which aspects have changed due to lighting and which aspects are part of the underlying scene content (and should exist in both the first and second lighting effect feature information). This mechanism directly facilitates the decoupling of the target lighting effect from the reference image content.

[0130] The network structure design of the light effect adapter involves concatenating the first light effect feature information embedding_a and the second light effect feature information embedding_A, i.e., the concatenated light effect feature information Zconcat = concat(embedding_a, embedding_A), as the input to the light effect adapter. The shape of the concatenated light effect feature information Zconcat is (2N, D), where N represents the feature length and D represents the feature dimension.

[0131] When the dimension D of the stitched light effect feature information Zconcat is inconsistent with the working dimension 768 of the light effect adapter (D!=768), the first light effect feature information embedding_a is initially projected through a linear layer to obtain a projection feature proj_a with a unified dimension of 768 and a shape of (N, 768); the second light effect feature information embedding_A is initially projected through a linear layer to obtain a projection feature proj_A with a unified dimension of 768 and a shape of (N, 768); when the dimension D of the stitched light effect feature information Zconcat is consistent with the working dimension 768 of the light effect adapter, the first light effect feature information embedding_a is used as the projection feature proj_a, and the second light effect feature information embedding_A is used as the projection feature proj_A.

[0132] The projected feature proj_a is assigned the value Ha and used as the key vector and value vector, and the projected feature proj_A is assigned the value HA and used as the query vector of the sequence to be processed; then it enters the cyclic processing process of the light effect adaptation unit (including self-attention module, cross-attention module and feedforward network module) which includes five iterations.

[0133] In each iteration, the self-attention module first performs multi-head self-attention computation (MultiHeadSelfAttention (HA)) on the query vector HA to obtain the self-attention output feature attn_output_self, so that the first light effect feature information can interact with each other and complete feature extraction. Then, the self-attention output feature is residually connected with the original query feature HA, and updated to obtain the new query feature HA=LayerNorm(HA+attn_output_self) through layer normalization (LayerNorm).

[0134] In the cross-attention module, firstly, multi-head self-attention (MultiHeadCrossAttention) is performed on the query vector HA to obtain the multi-head self-attention output feature attn_output_cross, so that the second light effect feature information pays attention to the first light effect feature information to discover the difference / correspondence; then, the multi-head self-attention output feature is residually connected with the original query feature HA, and updated through layer normalization (LayerNorm) to obtain the new query feature HA=LayerNorm(HA+attn_output_cross).

[0135] In the feedforward network module, the new query feature HA, updated by the cross-attention module, is input into the feedforward network module (FeedForward) composed of a multi-layer perceptron (MLP) for further feature transformation, resulting in the feedforward network output feature ff_output. This output feature is then residually connected to HA and layer normalization is performed to update the feature HA = LayerNorm(HA + ff_output). FeedForward typically consists of two linear layers with a non-linear activation function (e.g., GELU) in between. GELU stands for Gaussian Error Linear Unit.

[0136] The feature sequence HA after loop processing can be averaged using the sequence dimension (AveragePooling(HA, dimension=sequence_dim)) to obtain a 768-dimensional pooled output feature (pooled_output). Alternatively, a pre-added special classification token ([CLS] token) can be used as an alternative, and this corresponding feature can be output as a 768-dimensional light effect feature vector. The 768-dimensional pooled output feature or light effect feature vector can then be used as the output of the light effect adapter. Here, dimension represents the dimensionality, and sequence_dim represents the sequence dimension.

[0137] The VACE DiT diffusion generation module is built upon powerful pre-trained diffusion transformer (DiT) video models, such as the Lightweight Transformer eXtended Video (LTX-Video) model. This backbone diffusion transformer model is frozen throughout the lighting process to preserve its powerful video generation capabilities and temporal consistency. The U-Net architecture of the diffusion transformer model contains multiple transformer blocks that process the latent variables of the video and gradually restore details during denoising.

[0138] First, all information input to the VACE DiT diffusion generation module needs to be uniformly formatted by the VACE Video Condition Unit (VCU):

[0139] Video frame sequence (V before ): The sequence of input video frames to be illuminated (target video) is used as the main visual input of the video conditional unit.

[0140] Mask (M): Since the lighting effect is to be applied to the entire target video, the mask sequence is set to all-ones, indicating that all pixels in the target video are areas to be changed.

[0141] Light effect embedded information (E light The light effect embedding information from module 2 is processed into a condition token acceptable to VACE.

[0142] Light effect description text (Tight): The light effect description text from Module 4 will be converted into light effect text features by the text encoder, which will serve as the text condition token for VACE.

[0143] Conceptual decoupling: VACE's conceptual decoupling mechanism distinguishes between the areas to be preserved and the areas to be changed based on the mask M.

[0144] Since M is all 1s, the target video V before All frames will be treated as F c (ReactiveFrames, i.e., frames to be changed), F k (lnactiveFrames, i.e., frames that remain unchanged) will be empty.

[0145] This setup can match the video-to-video (V2V) task paradigm, which involves transforming the visual style (lighting) of the target video while preserving its original motion and content.

[0146] Secondly, perform multimodal conditional injection (ContextAdapterTuning):

[0147] F after VCU formatting c (i.e. V) before The VACE context embedder, a trainable component, is responsible for encoding high-dimensional video frame sequences into a more compact and semantically rich contextual feature.

[0148] The context features output by the context embedder, along with the light effect embedding information from Module 2 and the light effect description text from Module 4, are fed into the context blocks in the VACE architecture for fusion processing. The context blocks are responsible for effectively fusing these multimodal conditional signals—context features, light effect embedding information, and light effect description text—and injecting their output as an additional signal into the intermediate layers of the frozen DiT backbone model. The context blocks typically consist of multiple transformers and employ a cross-attention mechanism internally. The context blocks are trainable during the training phase, while the backbone DiT model remains frozen. This means that only lightweight adapter layers need to be trained to understand new light effect conditions, significantly improving training efficiency and convergence speed.

[0149] Next, video diffusion denoising is performed:

[0150] Latent space processing: target video V before First, the variable is compressed into a low-dimensional latent space using a variational autoencoder (VAE) or similar component, and then random noise is added to form a noisy latent variable z. t .

[0151] DiT Core Denoising: The DiT backbone model receives the noisy latent variable z. t And injected conditional information. In multiple denoising steps, DiT progressively predicts and removes noise. During this process, the injected lighting effect embedding information and lighting effect description text continuously guide DiT, enabling it to not only restore the content and motion of the original video during denoising, but also simultaneously render the target lighting effect onto the target video.

[0152] Temporal Consistency: As a video diffusion model, VACE's DiT architecture inherently includes a temporal attention mechanism. This means that at each step of denoising, the model considers not only the spatial information of the current frame but also the temporal information of adjacent frames. This inherent temporal modeling capability is crucial for ensuring stable and continuous lighting effects in the output video and avoiding flicker.

[0153] VAE Encoding and Output: After the diffusion denoising process is complete, the generated noise-free latent variables will be converted from latent space back to pixel space by a variational autoencoder decoder. Finally, module 3 outputs the illuminated video frame sequence V. after These frame sequences are similar to V in both content and motion. before The lighting effects are consistent, but have been precisely and temporally modified to the target lighting effects defined by the second reference image.

[0154] This embodiment proposes a video lighting effect processing method based on decoupled lighting effect injection. It decouples the target lighting effect by using reference image pairs before and after lighting effect addition (before and after lighting), and guides the video diffusion model to perform lighting together with the lighting effect description text. This method has a wider range of functions and more controllable lighting effects.

[0155] The precise control, broad flexibility, and high-fidelity fusion quality of the aforementioned video's lighting effect processing method bring significant practical value to downstream applications. Firstly, this method abandons the traditional reliance on vague and inaccurate control signals such as text or normal maps, achieving a key breakthrough in the precision of lighting effect control. By using paired reference images before and after lighting to decouple and extract the target lighting effect, it obtains richer, more intuitive, and unambiguous lighting information than traditional guidance methods such as text or normal maps. Text descriptions struggle to accurately convey the spatial distribution, intensity, and color details of complex lighting effects, while geometric information such as normal maps may be inaccurate in prediction. These factors can easily lead to insufficient lighting effect expression or contradictory guidance conditions, resulting in chaotic generation results. The aforementioned method, using image pairs as guidance, significantly improves the accuracy of lighting effect decoupling and the precision of control. Compared to textual descriptions that struggle to accurately convey spatial distribution, or normal maps that may be inherently unpredictable, reference image pairs offer richer, more intuitive, and unambiguous lighting information. This effectively avoids the inherent limitations of other control methods in expressing lighting effects, allowing for more robust capture and reproduction of target lighting effects. The resulting application value is that it provides professional creators with precise, WYSIWYG control, significantly lowering the barrier to entry and uncertainty in lighting effect adjustments, and ensuring stability and reliability during commercial deployment.

[0156] The aforementioned video lighting effect processing method boasts significant advantages in terms of application scenarios and flexibility. Users only need to provide the target video to be processed and a pair of reference images that can demonstrate the desired changes in target lighting effects to achieve the transfer of lighting effects. Because this method does not rely on specific 3D scene reconstruction, complex physical parameter settings, or predefined simple light source types, it can theoretically transfer any observed complex lighting effects, whether subtle changes in natural light or dramatic artistic lighting, greatly expanding its application scope and creative freedom. The resulting application value is revolutionary: on the one hand, in film and film and visual effects post-production, it can eliminate the high-cost 3D reconstruction and physically based rendering processes; on the other hand, it greatly expands the application scope and creative freedom, enabling ordinary users to achieve professional-grade lighting effects in their videos by providing reference examples, and providing powerful technical empowerment for popular applications such as video editing software and short video platforms.

[0157] The aforementioned video lighting effect processing method leverages a robust video diffusion model architecture to ensure that the merged target lighting effect and the target video are both natural and realistic, maintaining high temporal consistency—that is, ensuring high-fidelity fusion with spatiotemporal consistency. Spatially, the diffusion model's powerful generation capabilities effectively avoid the texture-mapped or unrealistic feel caused by inaccurate geometry or material estimations in traditional physically based rendering methods, and also overcome the severe distortions caused by simple pixel adjustment methods. The diffusion model's progressive denoising process helps to smoothly and harmoniously integrate the target lighting effect into the structure and texture of the target image, avoiding the texture-mapped feel caused by inaccurate geometry estimations in traditional physically based rendering and the severe distortions caused by simple pixel adjustment methods, generating a visual effect with natural light and shadow transitions and good detail retention. More importantly, in the temporal dimension, its inherent temporal modeling capabilities ensure that the applied lighting effect is stable and continuous between video frames, effectively solving the flickering and artifact problems caused by traditional frame-by-frame processing methods. This high fidelity in spatiotemporal consistency is key to the successful application of the video. It enables the output target video to meet the quality standards required for professional film and television production and immersive augmented reality / virtual reality applications, ensuring the realism, usability, and temporal consistency of the target video.

[0158] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0159] Based on the same inventive concept, this application also provides an image light effect processing apparatus for implementing the image light effect processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image light effect processing apparatus embodiments provided below can be found in the limitations of the image light effect processing method described above, and will not be repeated here.

[0160] In one exemplary embodiment, such as Figure 4 As shown, an image light effect processing device is provided, wherein:

[0161] The reference image pair acquisition module 402 is used to acquire a reference image pair that displays the change of the target light effect; the target light effect is the light effect to be added to the target image; the reference image pair includes a first reference image without the target light effect and a second reference image with the target light effect added.

[0162] The light effect feature information acquisition module 404 is used to obtain first light effect feature information based on the first reference image and second light effect feature information based on the second reference image;

[0163] The light effect embedding information acquisition module 406 is used to decouple the first light effect feature information and the second light effect feature information to obtain the light effect embedding information generated by adding the target light effect.

[0164] The light effect addition processing module 408 is used to perform light effect addition processing on the target image according to the light effect embedding information and the light effect description text of the second reference image, so as to obtain a target image with the target light effect added.

[0165] In one embodiment, the light effect feature information acquisition module 404 is further configured to: divide the first reference image into multiple image blocks, combine the multiple image blocks to obtain a first reference image block sequence; and input the first reference image block sequence into a visual encoder to obtain first light effect feature information.

[0166] In one embodiment, the light effect embedding information acquisition module 406 is further configured to: decouple the first light effect feature information and the second light effect feature information through the first-layer light effect adaptation unit of the light effect adapter to obtain the first-layer enhanced initial light effect embedding information; decouple the previous-layer enhanced initial light effect embedding information through the non-first-layer light effect adaptation unit of the light effect adapter to obtain the non-first-layer enhanced initial light effect embedding information; and obtain the light effect embedding information generated by adding the target light effect based on the last-layer enhanced initial light effect embedding information.

[0167] In one embodiment, the light effect embedding information acquisition module 406 is further configured to: perform feature interaction processing on the second light effect feature information through the self-attention module to obtain the self-attention processed light effect feature information; perform alignment and fusion processing on the self-attention processed light effect feature information and the first light effect feature information through the cross-attention module to obtain cross-attention feature information, so as to obtain the initial light effect embedding information generated by adding the target light effect; and perform nonlinear transformation and feature enhancement on the initial light effect embedding information through the feedforward network module to obtain the first-layer enhanced initial light effect embedding information.

[0168] In one embodiment, the device further includes a light effect description text acquisition module, configured to: input the second reference image and a preset prompt word into a visual language model to obtain the result output by the visual language model; the preset prompt word is used to instruct the visual language model to describe the lighting effect of the second reference image; and obtain the light effect description text of the second reference image based on the result output by the visual language model.

[0169] In one embodiment, the light effect addition processing module 408 is further configured to: encode the light effect description text to obtain light effect text features; and perform light effect addition processing on the target image according to the light effect embedding information and the light effect text features to obtain a target image with the target light effect added.

[0170] In one embodiment, the light effect addition processing module 408 is further configured to: perform context encoding on the target image using a context embedder in the diffusion generation model to obtain image context features; fuse the light effect embedding information, the light effect text features, and the image context features using a context block in the diffusion generation model to obtain light effect condition features; perform latent space compression and noise addition processing on the target image using a variational autoencoder in the diffusion generation model to obtain noisy latent variables; and perform multi-step iterative denoising on the noisy latent variables using a diffusion backbone network in the diffusion generation model under the guidance of the light effect condition features to obtain a target image with the added target light effect.

[0171] The various modules in the aforementioned image light effect processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0172] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to image lighting effect processing. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an image lighting effect processing method.

[0173] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0174] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0175] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0176] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0177] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0178] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0179] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0180] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing the light effect of an image, characterized in that, The method includes: Obtain a pair of reference images that display the changes in the target lighting effect; the target lighting effect is the lighting effect to be added to the target image; the pair of reference images includes a first reference image without the target lighting effect and a second reference image with the target lighting effect added. First light effect feature information is obtained based on the first reference image, and second light effect feature information is obtained based on the second reference image; The first light effect feature information and the second light effect feature information are decoupled to obtain the light effect embedding information generated by adding the target light effect; Based on the light effect embedding information and the light effect description text of the second reference image, the target image is processed to add light effects, resulting in a target image with the added target light effects.

2. The method according to claim 1, characterized in that, The step of obtaining the first light effect feature information based on the first reference image includes: The first reference image is divided into multiple image blocks, and the multiple image blocks are combined to obtain a first reference image block sequence; The first reference image block sequence is input into the visual encoder to obtain the first light effect feature information.

3. The method according to claim 1, characterized in that, The step of decoupling the first light effect feature information and the second light effect feature information to obtain the light effect embedding information generated by adding the target light effect includes: The first-layer light effect adaptation unit of the light effect adapter decouples the first light effect feature information and the second light effect feature information to obtain the first-layer enhanced initial light effect embedding information. The non-first-layer light effect adaptation unit of the light effect adapter decouples the enhanced initial light effect embedding information of the previous layer to obtain the enhanced initial light effect embedding information of the non-first layer. Based on the initial light effect embedding information after the final layer enhancement, the light effect embedding information generated by adding the target light effect is obtained.

4. The method according to claim 3, characterized in that, Each layer of the light effect adaptation unit includes a self-attention module, a cross-attention module, and a feedforward network module. The first-layer light effect adaptation unit of the light effect adapter decouples the first light effect feature information and the second light effect feature information to obtain the first-layer enhanced initial light effect embedding information, including: The second light effect feature information is processed by the self-attention module to obtain the light effect feature information after self-attention processing. The cross-attention module aligns and fuses the self-attention processed light effect feature information and the first light effect feature information to obtain cross-attention feature information, thereby obtaining the initial light effect embedding information generated by adding the target light effect. The feedforward network module performs nonlinear transformation and feature enhancement on the initial light effect embedding information to obtain the first-layer enhanced initial light effect embedding information.

5. The method according to claim 1, characterized in that, Obtain the light effect description text of the second reference image, including: The second reference image and preset prompt words are input into the visual language model to obtain the output of the visual language model; the preset prompt words are used to instruct the visual language model to describe the lighting effect of the second reference image. Based on the output of the visual language model, the light effect description text of the second reference image is obtained.

6. The method according to claim 1, characterized in that, The step of adding light effects to the target image based on the light effect embedding information and the light effect description text of the second reference image to obtain a target image with the added target light effects includes: The light effect description text is encoded to obtain light effect text features; Based on the light effect embedding information and the light effect text features, the target image is processed to add light effects, resulting in a target image with the added target light effects.

7. The method according to claim 6, characterized in that, The step of adding light effects to the target image based on the light effect embedding information and the light effect text features to obtain a target image with the added target light effects includes: The target image is context-encoded using a context embedder in the diffusion generation model to obtain image context features; By using the context block in the diffusion generation model, the light effect embedding information, the light effect text features, and the image context features are fused to obtain the light effect condition features; The target image is subjected to latent space compression and noise addition processing by using the variational autoencoder in the diffusion generation model to obtain noisy latent variables; By using the diffusion backbone network in the diffusion generation model, the noisy latent variables are denoised in multiple steps under the guidance of the light effect condition features, resulting in a target image with the added target light effect.

8. An image light effect processing device, characterized in that, The device includes: The reference image pair acquisition module is used to acquire a reference image pair that displays the change of the target light effect; the target light effect is the light effect to be added to the target image; the reference image pair includes a first reference image without the target light effect and a second reference image with the target light effect added. The light effect feature information acquisition module is used to obtain first light effect feature information based on the first reference image and second light effect feature information based on the second reference image; The light effect embedding information acquisition module is used to decouple the first light effect feature information and the second light effect feature information to obtain the light effect embedding information generated by adding the target light effect. The light effect addition processing module is used to perform light effect addition processing on the target image according to the light effect embedding information and the light effect description text of the second reference image, so as to obtain a target image with the target light effect added.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images