Method for reconstructing illumination of driver video in cockpit based on diffusion model

Abstract

The application belongs to the technical field of intelligent automobiles, and particularly relates to a driver video light reconstruction method in a cabin based on a diffusion model. The method first constructs a light condition graph with continuous gradual change characteristics of brightness and darkness according to an input video resolution and a target light scene, and sets parameters such as light source position and intensity. Then, a video frame sequence is extracted and encoded into a latent space representation, retaining driver actions, postures and cabin structures, while fusing text semantics and light coding to form a multi-modal generation condition. Next, a timing motion module and a light control model are introduced into a basic diffusion model, giving consideration to the reconstruction of timing consistency and light direction, shadow and hue in the denoising generation process. Finally, the generated latent sequence is decoded into RGB frames and synthesized into a video. The application can efficiently generate realistic video samples under various light conditions such as strong light, dim light, backlight, shadow and reflection, while keeping the original behavior content and spatial structure stable.

Description

Technical Field

[0001] This invention belongs to the field of intelligent vehicle technology, specifically a method for reconstructing driver video illumination in the cockpit based on a diffusion model. Background Technology

[0002] With the development of intelligent connected vehicle technology, Driver Monitoring Systems (DMS) have become an important component in ensuring human-machine co-driving safety. DMS typically relies on cockpit cameras to collect visual information such as driver facial expressions, eye states, head posture, and body movements to identify the driver's state. To ensure the stability of DMS algorithms in real road environments, test data not only needs to cover different drivers and different action types, but also needs to fully cover complex and changing cockpit lighting conditions, such as strong daylight, side window incident light, backlight, tunnel brightness variations, low nighttime illumination, dashboard reflections, and local shadows.

[0003] However, acquiring video data under different lighting conditions presents significant challenges in practical engineering. Real-world vehicle data acquisition is highly dependent on weather, time, road conditions, and vehicle driving status, making it difficult to accurately reproduce specific lighting conditions. Obtaining a rich sample of lighting data typically requires repeated data collection under different times, locations, and weather conditions, significantly increasing time, manpower, and financial costs. Furthermore, extreme scenarios such as strong backlighting, short-term glare, and sudden changes in lighting at tunnel entrances and exits exhibit strong randomness, making it difficult to reliably acquire data using conventional methods. This results in a shortage of complex lighting samples, impacting the robustness testing of the DMS algorithm under extreme lighting conditions.

[0004] Existing data augmentation methods often employ techniques such as brightness adjustment, contrast variation, color perturbation, or filter overlay to simulate lighting changes. However, these methods primarily operate at the two-dimensional pixel level, making it difficult to realistically represent the spatial relationships between light source direction, light-dark transitions, facial shadows, vehicle interior occlusion, and localized reflections. In continuous video processing, frame-by-frame augmentation may also introduce problems such as brightness jumps, background flickering, and unstable details in people, failing to meet the requirements of DMS video testing for temporal continuity and visual realism. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a cockpit-based driver video illumination reconstruction method based on a diffusion model. By introducing a controllable illumination condition map, textual semantic constraints, foreground constraints of the original cockpit video, and a time-series consistency generation mechanism, the target illumination distribution is injected as conditional information into the diffusion generation process. While maintaining the continuity of driver actions, postures, and cockpit structure, the method reconstructs the illumination direction, intensity, shadows, and overall brightness relationships of the video image, thereby efficiently generating video data under various complex illumination conditions. This provides reliable support for DMS algorithm testing, complex scene verification, and system sensitive sample expansion.

[0006] The technical solution of this invention is described below in conjunction with the accompanying drawings:

[0007] This invention provides a method for reconstructing cockpit-based pilot video illumination based on a diffusion model, comprising the following steps:

[0008] S1. Construct a lighting condition map that matches the target video;

[0009] Based on the type of cockpit lighting to be generated, the position, shape, size, brightness, and offset direction of the light source are set to form a spatial lighting distribution map consisting of a black background and bright spots, and a naturally gradient lighting boundary is generated through blurring.

[0010] S2. Construct multimodal generation conditions;

[0011] First, the input video is subjected to frame sequence extraction, and the video frames are encoded into foreground latent representations to preserve the driver's attitude, actions, and cockpit structure. At the same time, a text encoder is used to convert positive and negative cue words into semantic constraints, and combined with the lighting condition map, a unified conditional input containing content, semantics, and lighting information is formed.

[0012] S3. Generate re-illuminated video frames based on the temporal diffusion model and illumination control model;

[0013] The system loads a video motion module into the basic diffusion model to enhance the temporal consistency between consecutive frames, and introduces an illumination control model to constrain the diffusion denoising process to the target illumination conditions. Based on the multimodal generation conditions, the model gradually denoises and generates new video frame latent representations in the latent space, reconstructing the light source direction, brightness relationship, shadow distribution and overall illumination effect of the image.

[0014] S4. Decode the generated frames and synthesize the target video;

[0015] The system inputs the latent sequence of the re-illuminated video into the VAE decoder, restores it into continuous RGB image frames, and performs temporal arrangement and video synthesis according to the original video or preset frame rate to generate a cockpit monitoring test video under the target illumination conditions.

[0016] Furthermore, the specific method of S1 is as follows:

[0017] S11. Read the cockpit video of the pilot to be processed, and obtain the basic parameters of video resolution, frame rate, and frame count. Assume the input video is: (1)

[0018] In the formula, V represents the input video; I t This represents the t-th frame; T represents the total number of video frames; the size of each frame is: (2)

[0019] In the formula, Let H represent the set of real numbers; H represent the image height; W represent the image width; and 3 represent the RGB three channels. The constructed target lighting condition map L maintains the same size as the input video frame, i.e.: (3)

[0020] S12. Based on the lighting scene to be simulated, set the position, shape, size, and intensity parameters of the target light source. The target light source parameters are expressed as follows:

[0021] (4)

[0022] In the formula, and This indicates the coordinate position of the center of the light source in the lighting condition diagram. Indicates the radius of the light spot or the area of ​​influence. Indicates light intensity;

[0023] The center position of the light source is set according to the image size normalization:

[0024] (5)

[0025] In the formula, and These represent the relative position coefficients of the light source in the horizontal and vertical directions, respectively, with values ​​ranging from [0,1].

[0026] S13. After determining the light source parameters, construct the initial lighting condition diagram. The initial lighting condition map uses a combination of a black background and white light spots. The black areas represent weak light areas with a normalized light intensity of 0, while the bright areas represent normalized light intensity greater than or equal to a preset threshold. The target light source area, in which For any pixel in the image The initial light intensity is defined as: (6)

[0027] In the formula, Represents pixels The initial illumination intensity at the location; when the pixel is within the radius of the light source. Indoors, impart light intensity ,in When the pixel is located When the background is exposed, the value is assigned to 0, thus forming a black background with white light spots;

[0028] If you need to simulate elliptical lighting or oblique incident light, expand the circular light spot into an elliptical light spot: (7)

[0029] In the formula, and These represent the semi-axis lengths of the elliptical light spot in the horizontal and vertical directions, respectively;

[0030] S14. Since real cockpit lighting typically exhibits continuous attenuation, the initial lighting map is blurred using Gaussian blurring. Convert to smooth lighting condition map :

[0031] (8)

[0032] In the formula, The standard deviation is expressed as Gaussian kernel, This represents the convolution operation. Controlling the softness of the lighting boundary, The larger the size, the wider the light diffusion range and the smoother the transition between light and dark;

[0033] Subsequently, the blurred illumination map was normalized: (9)

[0034] In the formula, Represents the pixels in the final lighting condition map. The normalized illumination intensity has a value range of [0,1]. The normalized illumination condition map serves as the illumination control input for the subsequent diffusion model, and is used to characterize the target light source direction, illumination range, intensity distribution, and brightness variation relationship.

[0035] Furthermore, the specific method of S2 is as follows:

[0036] S21. Extract frames from the input video to obtain a continuous video frame sequence: (10)

[0037] In the formula, This indicates that the input video is available. This represents the image of frame T. This represents the total number of video frames. To ensure that subsequent input conditions remain consistent in spatial dimension, each frame is uniformly adjusted to a preset size and normalized. (11)

[0038] In the formula, This indicates size adjustment and normalization operations. Indicates the first preprocessed... Frame driver image;

[0039] S22. Input the preprocessed video frames into the VAE encoder to obtain the foreground latent representation for each frame: (12)

[0040] In the formula, Indicates VAE encoder, Indicates the first Foreground latent features of the driver's image in frame 1; foreground latent representation of all video frames is as follows: (13);

[0041] S23. Set positive and negative text prompts based on the target generation effect; positive prompts are used to describe the semantics of the image that are desired to be retained or generated, and negative prompts are used to suppress the semantics of the image that are not desired to be retained or generated; let the positive prompts be... Negative prompt words are The CLIP text encoder yielded the following results:

[0042] (14)

[0043] (15)

[0044] In the formula, Indicates a text encoder. Indicates a positive semantic condition. Indicates a negative semantic condition;

[0045] S24. Obtain the target lighting condition map from S1. Input the conditional encoding module to obtain the potential lighting conditions:

[0046] (16)

[0047] In the formula, This indicates a lighting condition encoder, implemented by a VAE encoder; The target illumination potential conditions are represented. Illumination potential conditions are used to characterize the position, direction, intensity, illumination range and light-dark transition relationship of the target light source, providing controllable illumination constraints for subsequent diffusion models.

[0048] S25. Representing the foreground potential of the video. Textual semantic conditions , and potential lighting conditions The input condition fusion module forms unified multimodal generation conditions:

[0049] (17)

[0050] In the formula, This represents the multimodal conditional fusion function. Indicates the first The diffusion generation conditions corresponding to the frame, This represents the lighting condition weighting coefficient, with a value range of [0,1], used to control the intensity of the influence of the target lighting conditions on the generated result; when The lighting conditions are highly constrained, resulting in a significant lighting reconstruction effect in the generated video; At that time, the lighting conditions are less constrained, and the generated results tend to retain the original video lighting.

[0051] For the entire video segment, the multimodal generation conditions are expressed as follows:

[0052] (18)

[0053] The condition set includes driver video content constraints, text semantic constraints, and target illumination constraints, which serve as input conditions for subsequent temporal diffusion models to reconstruct video illumination.

[0054] Furthermore, the specific method of S3 is as follows:

[0055] S31. Load the pre-trained base diffusion model, and set it as follows:

[0056] (19)

[0057] in, This represents a diffusion denoising network. and These represent the VAE encoder and decoder, respectively. Building upon this, a temporal motion module is introduced, enabling the diffusion model to process consecutive video frames. (20)

[0058] in, This indicates the loading or fusion operation of the timing motion module. Indicates the parameters of the motion module. This represents a diffusion model with temporal modeling capabilities. This step is used to enhance the consistency of character poses, facial states, and cockpit backgrounds between adjacent frames, reducing flickering and jumps caused by frame-by-frame generation.

[0059] S32. Based on the temporal diffusion model, a lighting control model is further added to ensure that the diffusion denoising process is constrained by the target lighting conditions; let the parameters of the lighting control model be... The fused re-illumination diffusion model is then expressed as:

[0060] (twenty one)

[0061] In the formula, This indicates the loading operation of the lighting control module. This represents a diffusion model that simultaneously possesses the ability to generate time-consistent data and the ability to reconstruct illumination.

[0062] S33. In order to generate new lighting effects while preserving the original driver actions and cockpit structure, the system uses the foreground latent representation obtained in S2. Construct noisy latent variables; let the total number of diffusion steps be... denoising intensity is The initial denoising step is then expressed as:

[0063] (twenty two)

[0064] For the first Noise is added to the latent variables of the foreground of the frame to obtain the initial latent variables to be denoised: (twenty three)

[0065] In the formula, Indicates the first The noise scheduling coefficient corresponding to the step; , representing Gaussian noise; noise reduction intensity Used to control the balance between preserving the original video content and reconstructing the lighting. The larger the value, the more obvious the change in generated lighting; The smaller the size, the more likely it is to preserve the original image;

[0066] S34. During the diffusion sampling process, the first Frame in each denoising step's latent variables Time step And the multimodal generation conditions obtained from S2 Input the relight diffusion network to predict the noise component in the current latent variables:

[0067] (twenty four)

[0068] In the formula, This represents a re-illumination diffusion model that integrates a temporal motion module and an illumination control module. The model represents the first time. The noise predicted in the first step; to enhance the response of the generated results to text semantics and lighting conditions, the noise prediction results under positive and negative conditions are calculated separately, and a condition-guided form is adopted:

[0069] (25)

[0070] In the formula, This represents the noise prediction result under positive conditions. This represents the noise prediction result under negative conditions, where s represents the conditional guidance intensity. This represents the final noise estimate after fusing positive and negative constraints; subsequently, the sampler updates the latent variables in reverse based on the noise estimate:

[0071] (26)

[0072] In the formula, Indicates the first The sampling update function for each step;

[0073] S35. After the multi-step temporally guided denoising in S34, the final re-illumination latent representation can be obtained for each frame:

[0074] (27)

[0075] In the formula, This represents the complete denoising generation process jointly performed by the re-illumination diffusion model and the sampler. Indicates the first The initial noisy latent variables of the frame, This indicates the multimodal generation conditions for the corresponding frame. Indicates the first The final relighting latent variable generated for the frame;

[0076] Arrange all generated frames in chronological order according to the original video to obtain the relighting potential sequence of the entire video:

[0077] (28)

[0078] In the formula, This represents the final output relit video latent sequence. This indicates the total number of video frames.

[0079] Furthermore, the specific method of S4 is as follows:

[0080] S41. Input the re-illuminated video latent sequence obtained in S3 into the VAE decoder, restore the generated results in the latent space to RGB image frames, and use the VAE decoder... Decode the latent variables for each frame:

[0081] = (29)

[0082] in, Indicates the decoded result of the first... Frame relighting image;

[0083] S42. Perform pixel normalization, cropping, scaling, or format conversion on the image frames to meet the requirements of video compositing; let the post-processing function be... Then the first The image after frame processing is represented as follows:

[0084]

[0085] In the formula, and These represent the height and width of the target video frame, respectively. This represents the re-illuminated image frame after post-processing.

[0086] S43. Rearrange all post-processed image frames according to the chronological order of the original driver video to form a continuous sequence of re-illuminated image frames:

[0087] (31)

[0088] In the formula, This represents the sequence of re-illuminated video frames to be synthesized. Let the original video frame rate be f, and the output video use the same frame rate.

[0089] (32)

[0090] Subsequently, the system encodes and synthesizes the re-illuminated image frame sequence according to the original frame rate to obtain the cockpit video of the pilot under the target illumination conditions:

[0091] (33)

[0092] In the formula, This represents the video synthesis function. This indicates the final generated video.

[0093] The beneficial effects of this invention are as follows:

[0094] 1) This invention effectively reduces the cost of acquiring complex lighting video data within the cockpit. By constructing a controllable lighting condition map and combining it with a diffusion model to reconstruct the lighting of the original driver video, videos under various lighting conditions, such as strong light, low light, backlight, local shadows, and interior reflections, can be quickly generated without the need for actual vehicle data acquisition. Compared to traditional pixel-level enhancement methods such as brightness adjustment and filter overlay, this invention can more realistically reconstruct the direction of the light source, the transition between light and dark, the distribution of shadows, and the overall color tone, making the generated video closer to the real cockpit lighting environment.

[0095] 2) This invention also improves the temporal continuity and testability of the generated video while maintaining the stability of the original driver's actions, posture, facial state, and cockpit structure. By introducing a video motion module and a lighting control model, the generated results can reduce flickering, jumps, and structural deformation problems caused by frame-by-frame processing, making it suitable for robustness verification of the DMS algorithm in complex lighting scenarios. Attached Figure Description

[0096] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0097] Figure 1 This is a flowchart of the present invention;

[0098] Figure 2 This is a diagram of the input video;

[0099] Figure 3 This is a schematic diagram of the light spot;

[0100] Figure 4 This is a schematic diagram of the target lighting conditions;

[0101] Figure 5 This is a schematic diagram of a video reconstructed using lighting. Detailed Implementation

[0102] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0103] Example 1

[0104] See Figure 1 This invention provides a method for reconstructing cockpit-based pilot video illumination based on a diffusion model, comprising the following steps:

[0105] S1. Construct a lighting condition map that matches the target video;

[0106] First, based on the cockpit lighting type to be generated, the position, shape, size, brightness, and offset direction of the light source are set to form a spatial lighting distribution map consisting of a black background and bright spots. Then, a naturally gradation lighting boundary is generated through blurring. This lighting condition map is used to characterize the direction, illumination range, and intensity variations of the target light source, providing controllable conditions for subsequent video lighting reconstruction using a diffusion model. The specific method is as follows:

[0107] S11. First, read the cockpit video of the pilot to be processed and obtain its basic parameters such as video resolution, frame rate, and frame count; assuming the input video is:

[0108] (1)

[0109] In the formula, V represents the input video, and I t This represents the t-th frame, where T represents the total number of frames in the video. The size of each frame is:

[0110] (2)

[0111] In the formula, H represents the image height, W represents the image width, and 3 represents the RGB three channels. The constructed target lighting condition map L maintains the same size as the input video frame, that is:

[0112] (3)

[0113] This setting ensures that the subsequent lighting conditions correspond one-to-one with the original driver video in terms of spatial position, avoiding the offset and distortion of the lighting area due to inconsistent size.

[0114] S12. Based on the lighting scene to be simulated, set the position, shape, size, and intensity parameters of the target light source. The target light source parameters are expressed as follows:

[0115] (4)

[0116] In the formula, and This indicates the coordinate position of the center of the light source in the lighting condition diagram. Indicates the radius of the light spot or the area of ​​influence. Indicates light intensity.

[0117] The center position of the light source is set according to the image size normalization:

[0118] (5)

[0119] In the formula, and These represent the relative position coefficients of the light source in the horizontal and vertical directions, respectively, with values ​​ranging from [0,1].

[0120] S13. After determining the light source parameters, construct the initial lighting condition diagram. The image uses a combination of a black background and white light spots, where the black areas represent low-light areas and the bright areas represent the target light source areas. For any pixel in the image... Its initial light intensity is defined as:

[0121] (6)

[0122] In the formula, Represents pixels The initial illumination intensity at the location. When this pixel is within the radius of the light source. When the light source is inside the light source, it is given a higher brightness; when it is outside the light source range, it is assigned a value of 0, thus forming a black background with white light spots.

[0123] If you need to simulate elliptical lighting or oblique incident light, expand the circular light spot into an elliptical light spot:

[0124] (7)

[0125] in, and These represent the semi-axis lengths of the elliptical light spot in the horizontal and vertical directions, respectively.

[0126] S14. Because real cockpit lighting typically exhibits continuous attenuation, the initial lighting map needs to be blurred. Gaussian blurring can be used to... Convert to smooth lighting condition map :

[0127] (8)

[0128] In the formula, The standard deviation is expressed as Gaussian kernel, This represents the convolution operation. Control the softness of the lighting boundaries. The larger the size, the wider the light diffusion range and the smoother the transition between light and dark.

[0129] Subsequently, the blurred illumination map was normalized:

[0130] (9)

[0131] In the formula, Represents the pixels in the final lighting condition map. The normalized illumination intensity is defined as [0,1]. The normalized illumination condition map serves as the illumination control input for the subsequent diffusion model, characterizing the target light source direction, illumination range, intensity distribution, and brightness variation.

[0132] S2. Construct multimodal generation conditions;

[0133] First, the input video is processed by extracting frame sequences and encoding the video frames into foreground latent representations to preserve the driver's posture, actions, and cockpit structure. Simultaneously, a text encoder converts positive and negative cue words into semantic constraints, which, combined with the illumination condition map obtained in S1, form a unified conditional input containing content, semantics, and illumination information. This provides constraints for the subsequent diffusion model to generate relit video frames. The specific method is as follows:

[0134] S21. First, extract frames from the input video to obtain a continuous video frame sequence:

[0135] (10)

[0136] In the formula, This indicates that the input video is available. This represents the image of frame T. This represents the total number of video frames. To ensure that subsequent input conditions remain consistent in spatial dimension, each frame is uniformly adjusted to a preset size and normalized.

[0137] (11)

[0138] In the formula, This indicates size adjustment and normalization operations. and These represent the height and width of the target image, respectively. Indicates the first preprocessed... Frame driver image;

[0139] This step is used to ensure that the video frames match the lighting condition map in terms of spatial dimensions.

[0140] S22. Input the preprocessed video frames into the VAE encoder to obtain the foreground latent representation for each frame:

[0141] (12)

[0142] In the formula, Indicates VAE encoder, Indicates the first Foreground latent features of the driver's image in each frame. The foreground latent representation of all video frames can be expressed as:

[0143] (13)

[0144] This foreground potential is used to preserve the driver's identity features, facial features, body posture, movement changes, and cockpit interior structure in the original video, so that the subsequent lighting reconstruction results do not deviate from the original video content.

[0145] S23. Set positive and negative text prompts based on the target generated effect. Positive prompts describe the semantics of the scene that you want to retain or generate, while negative prompts suppress the semantics of the scene that you do not want to retain or generate. Let the positive prompt be... Negative prompt words are The CLIP text encoder yielded the following results:

[0146] (14)

[0147] (15)

[0148] In the formula, Indicates a text encoder. Indicates a positive semantic condition. This indicates a negative semantic condition. This step is used to provide semantic constraints for the diffusion model, ensuring that the generated results meet the content and visual style requirements of the cockpit-based pilot video.

[0149] S24. Obtain the target lighting condition map from S1. Input the conditional encoding module to obtain the potential lighting conditions:

[0150] (16)

[0151] In the formula, This indicates a lighting condition encoder, implemented by a VAE encoder; This represents the target illumination potential conditions, which characterize the position, direction, intensity, illumination range, and light-dark transition relationship of the target light source, providing controllable illumination constraints for subsequent diffusion models.

[0152] S25. Representing the foreground potential of the video. Textual semantic conditions , and potential lighting conditions The input condition fusion module forms unified multimodal generation conditions:

[0153] (17)

[0154] In the formula, This represents the multimodal conditional fusion function. Indicates the first The diffusion generation conditions corresponding to the frame, This represents the lighting condition weighting coefficient, used to control the intensity of the influence of the target lighting conditions on the generated result. If A larger value results in a more noticeable lighting reconstruction effect in the generated video; if... If the value is smaller, the generated result is more likely to retain the original video lighting.

[0155] For the entire video segment, the multimodal generation conditions are expressed as follows:

[0156] (18)

[0157] This set of conditions includes constraints on driver video content, text semantics, and target illumination, which serve as input conditions for subsequent temporal diffusion models to reconstruct video illumination.

[0158] S3. Generate re-illuminated video frames based on the temporal diffusion model and illumination control model;

[0159] The system loads a video motion module into the basic diffusion model to enhance temporal consistency between consecutive frames and introduces an illumination control model to constrain the diffusion denoising process to target illumination conditions. Subsequently, based on the multimodal generation conditions formed by S2, the model progressively denoises and generates new latent representations of video frames in the latent space. While maintaining the stability of the driver's actions, posture, and cockpit structure, it reconstructs the light source direction, brightness relationships, shadow distribution, and overall illumination effect of the image. The specific method is as follows:

[0160] S31. First, load the pre-trained base diffusion model, let it be:

[0161] (19)

[0162] In the formula, This represents a diffusion denoising network. and These represent the VAE encoder and decoder, respectively. Building upon this, a temporal motion module is introduced, enabling the diffusion model to process consecutive video frames.

[0163] (20)

[0164] In the formula, This indicates the loading or fusion operation of the timing motion module. Indicates the parameters of the motion module. This represents a diffusion model with temporal modeling capabilities. This step is used to enhance the consistency of character poses, facial states, and cockpit backgrounds between adjacent frames, reducing flickering and jumps caused by frame-by-frame generation.

[0165] S32. Based on the temporal diffusion model, a lighting control model is further added to ensure that the diffusion denoising process is constrained by the target lighting conditions. Let the parameters of the lighting control model be... The fused re-illumination diffusion model can then be expressed as:

[0166] (twenty one)

[0167] In the formula, This indicates the loading operation of the lighting control module. This represents a diffusion model that simultaneously possesses temporally consistent generation capabilities and lighting reconstruction capabilities. During the generation process, this model not only references textual semantics and the original video foreground, but also adjusts the light source direction, brightness distribution, shadow relationships, and overall color tone of the image based on potential lighting conditions.

[0168] S33. In order to generate new lighting effects while preserving the original driver actions and cockpit structure, the system uses the foreground latent representation obtained in S2. Construct noisy latent variables. Let the total number of diffusion steps be... denoising intensity is The initial denoising step can then be expressed as:

[0169] (twenty two)

[0170] For the first Noise is added to the latent variables of the foreground of the frame to obtain the initial latent variables to be denoised:

[0171] (twenty three)

[0172] It is in the middle, Indicates the first The noise scheduling coefficient corresponding to the step; , representing Gaussian noise. Noise reduction intensity. Used to control the balance between preserving the original video content and reconstructing the lighting. The larger the value, the more obvious the change in generated lighting; The smaller the size, the more likely it is to preserve the original image.

[0173] S34. During the diffusion sampling process, the first Frame in each denoising step's latent variables Time step And the multimodal generation conditions obtained from S2 Input the relight diffusion network to predict the noise component in the current latent variables:

[0174] (twenty four)

[0175] In the formula, This represents a re-illumination diffusion model that integrates a temporal motion module and an illumination control module. The model represents the first time. The noise predicted in the first step. To enhance the response of the generated results to text semantics and lighting conditions, noise prediction results under positive and negative conditions are calculated separately, and a condition-guided form is adopted:

[0176] (25)

[0177] In the formula, This represents the noise prediction result under positive conditions. This represents the noise prediction result under negative conditions, where s represents the conditional guidance intensity. This represents the final noise estimate after fusing positive and negative constraints. Subsequently, the sampler updates the latent variables in reverse based on this noise estimate:

[0178] (26)

[0179] In the formula, Indicates the first The sampling update function iterates from high-noise steps to low-noise steps, gradually removing noise from the latent variables. Under the joint constraints of foreground content, text semantics, and target lighting conditions, it reconstructs the light source direction, brightness relationship, shadow distribution, and overall lighting effect in the video frames. Simultaneously, due to the introduction of a temporal motion module in the model, adjacent frames can share motion consistency constraints during denoising, thereby reducing problems such as abrupt changes in character outlines, background flickering, and lighting discontinuities.

[0180] S35. After the multi-step temporally guided denoising in S34, the final re-illumination latent representation can be obtained for each frame:

[0181] (27)

[0182] In the formula, This represents the complete denoising generation process jointly performed by the re-illumination diffusion model and the sampler. Indicates the first The initial noisy latent variables of the frame, This indicates the multimodal generation conditions for the corresponding frame. Indicates the first The final relighting latent variable generated for the frame.

[0183] Arrange all generated frames in chronological order according to the original video to obtain the relighting potential sequence of the entire video:

[0184] (28)

[0185] In the formula, This represents the final output relit video latent sequence. This indicates the total number of video frames. The latent sequence preserves the original driver's actions, posture changes, and cockpit space structure in terms of content, and visually presents the lighting direction, shadow changes, brightness levels, and overall tone guided by the target lighting conditions. Furthermore, due to the temporal consistency constraints employed during generation, the latent sequence exhibits good continuity between adjacent frames, making it suitable for subsequent decoding to generate cockpit video of the driver under the target lighting conditions.

[0186] S4. Decode the generated frames and synthesize the target video;

[0187] The system inputs the re-illuminated video latent sequence obtained in step three into the VAE decoder, reconstructs it into continuous RGB image frames, and then arranges and synthesizes them temporally according to the original video or a preset frame rate to generate a cockpit driver monitoring test video under the target lighting conditions. This video, while maintaining the continuity of the driver's actions, posture, and cockpit structure, presents the reconstructed lighting direction, brightness relationships, and shadow effects. It can be used for DMS algorithm testing and verification in complex lighting scenes. The specific method is as follows:

[0188] S41. Input the re-illuminated video latent sequence obtained in S3 into the VAE decoder, restore the generated results in the latent space to RGB image frames, and use the VAE decoder... Decode the latent variables for each frame:

[0189] = (29)

[0190] In the formula, Indicates the decoded result of the first... Frame relighting image. After this step, the system converts the latent representation generated by the diffusion model into a visualized cockpit image frame of the pilot.

[0191] S42. Because the decoded image frames may have differences in pixel range, size deviation, or edge anomalies, they need to undergo pixel normalization, cropping, scaling, or format conversion to meet the requirements of video compositing. Let the post-processing function be... Then the first The image after frame processing can be represented as:

[0192]

[0193] in, and These represent the height and width of the target video frame, respectively. This represents the re-illuminated image frame after post-processing. This step ensures that all generated frames maintain consistency in resolution, channel format, and pixel range, preventing issues such as inconsistent image size or display abnormalities during subsequent video compositing.

[0194] S43. Rearrange all post-processed image frames according to the chronological order of the original driver video to form a continuous sequence of re-illuminated image frames:

[0195] (31)

[0196] In the formula, This represents the sequence of re-illuminated video frames to be synthesized. Let the original video frame rate be f, and the output video use the same frame rate.

[0197] (32)

[0198] Subsequently, the system encodes and synthesizes the re-illuminated image frame sequence according to the original frame rate to obtain the cockpit video of the pilot under the target illumination conditions:

[0199] (33)

[0200] In the formula, This represents the video synthesis function. This represents the final generated video. The video maintains the same temporal sequence as the input video, preserves the driver's actions, posture, and cockpit structure, and visually presents the reconstructed light source direction, shadow distribution, brightness levels, and overall lighting effects. It can be used for testing and verifying the DMS algorithm under complex lighting conditions.

[0201] Example 2

[0202] This embodiment uses a cockpit video of the driver as input, performs controlled lighting reconstruction on the video, and generates a cockpit video of the driver under the target lighting conditions. The input video is a video of the driver sitting in the driver's seat of the vehicle, which includes the driver's face, upper body posture, and cockpit background structure. (See reference...) Figure 2 The system first imports the video to be processed through the video reading module and extracts a sequence of consecutive frames according to preset parameters. In this embodiment, the video resolution is set to... The frame rate was set to 8fps, and the number of frames read was set to 24, forming a 3-second continuous video frame sequence of the driver. This video frame sequence is used to provide driver identification features, head posture, facial state, body movements, and cockpit space structure, providing original content constraints for subsequent lighting reconstruction.

[0203] After obtaining the input video parameters, the system constructs a target lighting condition map with the same size as the video frame. Specifically, this is done by drawing a black background map and setting white circular light spots within it. Black areas represent low-light or no-light areas, and white light spots represent the areas where the target light source is located. The position, size, brightness, and offset of the light spots can be adjusted according to the target lighting scene. For example, in this embodiment, the light spots are set in the upper right corner of the image to simulate side-incident light from a car window. (See [reference]). Figure 3 To avoid overly harsh light source boundaries, the system further applies Gaussian blur to the initial illumination map, creating a soft brightness distribution in the white light spots that gradually decreases from the center outwards. This results in a target illumination condition map with continuous light-dark transition characteristics. (See reference...) Figure 4 The lighting condition map is then input into the VAE encoder and converted into lighting potential conditions for subsequent lighting control in the diffusion model.

[0204] Simultaneously, the system performs latent space encoding on the input video frames. Each frame is converted into a foreground latent representation by a VAE encoder. This foreground latent representation is used to constrain the generated result to maintain the original driver's actions, posture, and cockpit structure. The system also encodes prompts using a text encoder to form semantic generation conditions. In this embodiment, positive prompts are set to "a man sitting in a car, with rich facial details, daylight, and realistic lighting," to guide the model in generating a driver inside the car, a clear face, and realistic lighting effects. Negative prompts are set to blurry, low-quality, facial distortion, and unrealistic lighting, to suppress abnormal generation results. Subsequently, the system fuses the foreground latent representation, text semantic conditions, and target lighting latent conditions through a lighting condition fusion module to form multimodal generation conditions for the diffusion model. The lighting control weight is set to 0.5 to balance the degree of preservation of the original video content with the intensity of the target lighting reconstruction.

[0205] During the generation phase, the system loads a basic diffusion model and embeds a temporal motion module and a lighting control model onto it. In this embodiment, the basic diffusion model adopts a realistic model based on the Stable Diffusion 1.5 architecture to provide the ability to generate characters, cabin environment, and realistic image details. The temporal motion module uses the AnimateDiff motion module to enhance the temporal consistency between consecutive frames and reduce issues such as face jumps, background flickering inside the vehicle, and discontinuous motion caused by frame-by-frame generation. The lighting control model uses the IC-Light relighting module to make the diffusion denoising process respond to the aforementioned target lighting condition map. During sampling, the system inputs the multimodal generation conditions into the sampler and performs stepwise denoising in the latent space. In this embodiment, the sampling step count is set to 20, the cue word guidance intensity is 8.0, the sampler is Euler, and the denoising intensity is 0.65. The denoising intensity controls the relationship between the degree of preservation of the original video content and the magnitude of lighting reconstruction; a lower value results in a more accurate representation of the original video, while a higher value indicates more significant changes in target lighting. In this embodiment, a value of 0.65 is used, which can generate more obvious local bright light, dark shadow and overall color tone changes while maintaining the stability of the driver's posture and the cockpit structure.

[0206] After multiple steps of time-guided denoising, the system obtains the latent sequence of the relit video. This latent sequence is further input into the VAE decoder and decoded into consecutive RGB image frames. The decoded image frames are then processed by size alignment, pixel range normalization, and format conversion. They are then arranged according to the temporal order of the original video and synthesized at the original frame rate of 8fps to form the cockpit driver video under the target lighting conditions. (See reference...) Figure 5 The final video retains the original driver's posture, expression, movements, and cockpit background structure in terms of content, and visually presents the light source direction, shadow distribution, brightness levels, and overall lighting effects guided by the target lighting condition map.

[0207] In summary, this patent proposes a cockpit video illumination reconstruction method based on a diffusion model. By constructing a controllable illumination condition map and combining video foreground constraints, textual semantic constraints, a temporal diffusion model, and an illumination control model, it achieves video generation under complex lighting scenarios while maintaining the continuous stability of driver actions, postures, and cockpit structure. This method can effectively reduce the cost of real-vehicle data acquisition, expand the DMS test sample library, and improve the robustness verification capability of the algorithm in scenarios such as strong light, low light, backlight, and local shadows.

[0208] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A cockpit-based driver video illumination reconstruction method based on a diffusion model, characterized in that, Includes the following steps: S1. Construct a lighting condition map that matches the target video; Based on the type of cockpit lighting to be generated, the position, shape, size, brightness and offset direction of the light source are set to form a spatial lighting distribution map composed of a black background and bright spots, and a naturally gradient lighting boundary is generated through blurring. S2. Construct multimodal generation conditions; First, the input video is subjected to frame sequence extraction, and the video frames are encoded into foreground latent representations to preserve the driver's attitude, actions, and cockpit structure. At the same time, a text encoder is used to convert positive and negative cue words into semantic constraints, and combined with the lighting condition map, a unified conditional input containing content, semantics, and lighting information is formed. S3. Generate re-illuminated video frames based on the temporal diffusion model and illumination control model; The system loads a video motion module into the basic diffusion model to enhance the temporal consistency between consecutive frames, and introduces an illumination control model to constrain the diffusion denoising process to the target illumination conditions. Based on the multimodal generation conditions, the model gradually denoises and generates new latent representations of video frames in the latent space, reconstructing the light source direction, brightness relationship, shadow distribution and overall lighting effect of the image; S4. Decode the generated frames and synthesize the target video; The system inputs the latent sequence of the re-illuminated video into the VAE decoder, restores it into continuous RGB image frames, and performs temporal arrangement and video synthesis according to the original video or preset frame rate to generate a cockpit monitoring test video under the target illumination conditions.

2. The cockpit-based driver video illumination reconstruction method based on a diffusion model according to claim 1, characterized in that, The specific method of S1 is as follows: S11. Read the cockpit video of the pilot to be processed, and obtain the basic parameters of video resolution, frame rate, and frame count. Assume the input video is: （1） In the formula, V represents the input video; I t This represents the t-th frame; T represents the total number of video frames; the size of each frame is: （2） In the formula, H represents the image height, W represents the image width, and 3 represents the RGB three channels. The constructed target lighting condition map L maintains the same size as the input video frame, that is: （3） S12. Based on the lighting scene to be simulated, set the position, shape, size, and intensity parameters of the target light source. The target light source parameters are expressed as follows: （4） In the formula, and This indicates the coordinate position of the center of the light source in the lighting condition diagram. Indicates the radius of the light spot or the area of ​​influence. Indicates light intensity; The center position of the light source is set according to the image size normalization: （5） In the formula, and These represent the relative position coefficients of the light source in the horizontal and vertical directions, respectively, with values ​​ranging from [0,1]. S13. After determining the light source parameters, construct the initial lighting condition diagram. The initial lighting condition map uses a combination of a black background and white light spots, where black areas represent low-light areas and bright areas represent target light source areas; for any pixel in the image... The initial light intensity is defined as: （6） In the formula, Represents pixels The initial illumination intensity at the location; when the pixel is within the radius of the light source. Indoors, impart light intensity ,in When the pixel is located When the background is exposed, the value is assigned to 0, thus forming a black background with white light spots; If you need to simulate elliptical lighting or oblique incident light, expand the circular light spot into an elliptical light spot: （7） In the formula, and These represent the semi-axis lengths of the elliptical light spot in the horizontal and vertical directions, respectively; S14. Since real cockpit lighting typically exhibits continuous attenuation, the initial lighting map is blurred using Gaussian blurring. Convert to smooth lighting condition map : （8） In the formula, The standard deviation is expressed as Gaussian kernel, This represents the convolution operation. Controlling the softness of the lighting boundary, The larger the size, the wider the light diffusion range and the smoother the transition between light and dark; Subsequently, the blurred illumination map was normalized: （9） In the formula, Represents the pixels in the final lighting condition map. The normalized illumination intensity has a value range of [0,1]. The normalized illumination condition map serves as the illumination control input for the subsequent diffusion model, and is used to characterize the target light source direction, illumination range, intensity distribution, and brightness variation relationship.

3. The cockpit-based driver video illumination reconstruction method based on a diffusion model according to claim 1, characterized in that, The specific method of S2 is as follows: S21. Extract frames from the input video to obtain a continuous video frame sequence: （10） In the formula, This indicates that the input video is being processed. This represents the image of frame T. This represents the total number of video frames. To ensure that subsequent input conditions remain consistent in spatial dimension, each frame is uniformly adjusted to a preset size and normalized. （11） In the formula, This indicates size adjustment and normalization operations. and These represent the height and width of the target image, respectively. Indicates the first preprocessed step Frame driver image; S22. Input the preprocessed video frames into the VAE encoder to obtain the foreground latent representation for each frame: （12） In the formula, Indicates VAE encoder, Indicates the first Foreground latent features of the driver's image in the frame; The foreground potential representation of all video frames is: （13）； S23. Set positive and negative text prompts based on the target generation effect; positive prompts are used to describe the semantics of the image that are desired to be retained or generated, and negative prompts are used to suppress the semantics of the image that are not desired to be retained or generated; let the positive prompts be... Negative prompt words are The results obtained using the CLIP text encoder are as follows: （14） （15） In the formula, Indicates a text encoder. Indicates a positive semantic condition. Indicates a negative semantic condition; S24. Obtain the target lighting condition map from S1. Input the conditional encoding module to obtain the potential lighting conditions: （16） In the formula, This indicates a lighting condition encoder, implemented by a VAE encoder; The target illumination potential conditions are represented. Illumination potential conditions are used to characterize the position, direction, intensity, illumination range and light-dark transition relationship of the target light source, providing controllable illumination constraints for subsequent diffusion models. S25. Representing the foreground potential of the video. Textual semantic conditions , and potential lighting conditions The input condition fusion module forms unified multimodal generation conditions: （17） In the formula, This represents the multimodal conditional fusion function. Indicates the first The diffusion generation conditions corresponding to the frame, This represents the lighting condition weighting coefficient, with a value range of [0,1], used to control the intensity of the influence of the target lighting conditions on the generated result; when The lighting conditions are highly constrained, resulting in a significant lighting reconstruction effect in the generated video; At that time, the lighting conditions are less constrained, and the generated results tend to retain the original video lighting. For the entire video segment, the multimodal generation conditions are expressed as follows: （18） The condition set includes driver video content constraints, text semantic constraints, and target illumination constraints, which serve as input conditions for subsequent temporal diffusion models to reconstruct video illumination.

4. The cockpit-based driver video illumination reconstruction method based on a diffusion model according to claim 1, characterized in that, The specific method of S3 is as follows: S31. Load the pre-trained base diffusion model, and set it as follows: （19） in, This represents a diffusion denoising network. and These represent the VAE encoder and decoder, respectively; a temporal motion module is introduced to enable the diffusion model to process consecutive video frames. （20） in, This indicates the loading or fusion operation of the timing motion module. Indicates the parameters of the motion module. This represents a diffusion model with temporal modeling capabilities; S32. Based on the temporal diffusion model, a lighting control model is further added to ensure that the diffusion denoising process is constrained by the target lighting conditions; let the parameters of the lighting control model be... The fused re-illumination diffusion model is then expressed as: （21） In the formula, This indicates the loading operation of the lighting control module. This represents a diffusion model that simultaneously possesses the ability to generate time-consistent data and the ability to reconstruct illumination. S33. In order to generate new lighting effects while preserving the original driver actions and cockpit structure, the system uses the foreground latent representation obtained in S2. Construct noisy latent variables; let the total number of diffusion steps be... denoising intensity is The initial denoising step is then expressed as: （22） For the Noise is added to the latent variables of the foreground of the frame to obtain the initial latent variables to be denoised: （23） In the formula, Indicates the first The noise scheduling coefficient corresponding to the step; , representing Gaussian noise; noise reduction intensity Used to control the balance between preserving the original video content and reconstructing the lighting. The larger the value, the more obvious the change in generated lighting; The smaller the size, the more likely it is to preserve the original image; S34. During the diffusion sampling process, the first Frame in each denoising step's latent variables Time step And the multimodal generation conditions obtained from S2 Input the relight diffusion network to predict the noise component in the current latent variables: （24） In the formula, This represents a re-illumination diffusion model that integrates a temporal motion module and an illumination control module. The model represents the first time. The noise predicted in the first step; to enhance the response of the generated results to text semantics and lighting conditions, the noise prediction results under positive and negative conditions are calculated separately, and a condition-guided form is adopted: （25） In the formula, This represents the noise prediction result under positive conditions. This represents the noise prediction result under negative conditions, where s represents the conditional guidance intensity. This represents the final noise estimate after fusing positive and negative constraints; subsequently, the sampler updates the latent variables in reverse based on the noise estimate: （26） In the formula, Indicates the first The sampling update function for each step; S35. After the multi-step temporally guided denoising in S34, the final re-illumination latent representation can be obtained for each frame: （27） In the formula, This represents the complete denoising generation process jointly performed by the re-illumination diffusion model and the sampler. Indicates the first The initial noisy latent variables of the frame, This indicates the multimodal generation conditions for the corresponding frame. Indicates the first The final relighting latent variable generated for the frame; Arrange all generated frames in chronological order according to the original video to obtain the relighting potential sequence of the entire video: （28） In the formula, This represents the final output relit video latent sequence. This indicates the total number of video frames.

5. The cockpit-based driver video illumination reconstruction method based on a diffusion model according to claim 1, characterized in that, The specific method of S4 is as follows: S41. Input the re-illuminated video latent sequence obtained in S3 into the VAE decoder, restore the generated results in the latent space to RGB image frames, and use the VAE decoder... Decode the latent variables for each frame: = （29） in, Indicates the decoded result of the first... Frame relighting image; S42. Perform pixel normalization, cropping, scaling, or format conversion on image frames to meet video compositing requirements; let the post-processing function be... Then the first The image after frame processing is represented as follows: In the formula, and These represent the height and width of the target video frame, respectively. This represents the re-illuminated image frame after post-processing. S43. Rearrange all post-processed image frames according to the chronological order of the original driver video to form a continuous sequence of re-illuminated image frames: （31） In the formula, This represents the sequence of re-illuminated video frames to be synthesized. Let the original video frame rate be f, and the output video use the same frame rate. （32） Subsequently, the system encodes and synthesizes the re-illuminated image frame sequence according to the original frame rate to obtain the cockpit video of the pilot under the target illumination conditions: （33） In the formula, This represents the video synthesis function. This indicates the final generated video.

Images