Background replacement and video re-lighting method based on text description

Abstract

The invention discloses a text description-based background replacement and video relighting method. The method comprises the following steps of: obtaining a 3D rendering video; obtaining a normal video of an original real video to be processed to obtain a pixel point set of each video frame; randomly selecting a simulation point light source in the pixel point set, randomly selecting a color corresponding to the simulation point light source, and reapplying the color to each video frame to obtain a simulation relighting video; training the constructed video acquisition model through the 3D rendering video and the simulated relighting video to acquire an optimal video generation model; therefore, a video acquisition process of background replacement and video relighting based on text description is realized. The problems that an existing method based on image re-lighting model construction cannot well utilize the continuous time sequence association relation between video frames and cannot effectively achieve the modeling effect of video frame time sequence association are solved; in addition, the field lacks a high-quality video re-lighting data set, and the re-lighting capability of the model is seriously limited.

Description

Technical Field

[0001] This invention relates to the field of video relighting technology, and in particular to a method for background replacement and video relighting based on text description. Background Technology

[0002] Thanks to advancements in diffusion models, image relighting can now receive text or images as conditions to achieve good relighting effects. With the rapid development of video generation and editing, researchers are attempting to extend image relighting to the video domain. This scheme belongs to the field of video relighting and aims to adjust the lighting of dynamic foregrounds in videos according to different background environments.

[0003] Currently, video relighting methods mainly focus on applying the latest methods in the field of image relighting, such as IC-Light (L. Zhang, A. Rao, and M. Agrawala. Scaling in-the-wild training for diffusion-based illumination harmonization and editing by imposing consistent lighttransport. In The Thirteenth International Conference on Learning Representations, 2025.), to the video domain.

[0004] Light-A-Video (Y. Zhou, J. Bu, P. Ling, P. Zhang, T. Wu, Q. Huang, J. Li, X. Dong, Y. Zang, Y. Cao, A. Rao, J. Wang, and L. Niu. Light-a-video: Training-free videorelighting via progressive light fusion, 2025. URL https: / / arxiv.org / abs / 2502.08590.) is a training-free video relighting method based on IC-Light's frame-by-frame relighting approach. It employs consistent illumination attention and stabilizes background light source generation through inter-frame enhancement interactions. Simultaneously, it introduces a progressive light fusion module to ensure smooth illumination transitions. RelightVid (Y. Fang, Z. Sun, S. Zhang, T. Wu, Y. Xu, P. Zhang, J. Wang, G. Wetzstein, and D. Lin. Relightvid:

[0005] The Temporal-consistent diffusion model for video relighting (URL: https: / / arxiv.org / abs / 2501.16330) uses IC-Light as the base model and extends it to a 3D U-Net, introducing a spatiotemporal attention layer to capture the spatiotemporal dependencies between video frames. At the same time, a dataset containing Blender rendering data and simulated real data was constructed to train the model.

[0006] However, these models are all based on image relighting models, which cannot make good use of the continuous temporal correlation between video frames and cannot effectively achieve the modeling effect of temporal correlation of video frames. In addition, the lack of high-quality video relighting datasets in this field severely limits the relighting capabilities of the models. Summary of the Invention

[0007] This invention provides a method for background replacement and video relighting based on text description to overcome the above-mentioned technical problems.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] A method for background replacement and video relighting based on text description includes the following steps:

[0010] S1: Construct 3D rendering video data;

[0011] S2: Obtain the original real video to be processed, and perform video normal extraction processing on the original real video to obtain the normal video; and obtain the pixel set of each video frame based on the normal video;

[0012] A predetermined number of pixels are randomly selected from the pixel set as simulated point light sources in the background environment pixels. The color corresponding to the simulated point light source is randomly selected and mapped to its corresponding pixel position to obtain the illumination map corresponding to each video frame. The obtained illumination maps are then used to replace the video frames in the original real video to obtain the simulated relighting video.

[0013] S3: Use 3D rendered video and simulated relighting video as training video data for the original video, and train the constructed video acquisition model based on text description for background replacement and video relighting to obtain the optimal video acquisition model; and use the optimal video acquisition model to realize the video acquisition process based on text description for background replacement and video relighting.

[0014] Furthermore, S2 specifically includes the following steps:

[0015] S21: Obtain the original real video to be processed, and perform video normal extraction processing on the original real video based on the normal extraction model NormalCrafter to obtain the normal video;

[0016] And obtain the pixel set P of each video frame based on the normal video;

[0017] S22: Randomly select a preset number of pixels p from the pixel set P as simulated point light sources in the background environment pixels, and randomly select the light source color I corresponding to the simulated point light source. p By calculating the light source color I between each simulated point light source p Simulated lighting is used to obtain simulated relit video frames; and the simulated relit video frames are then reapplied to each video frame to obtain the lighting map corresponding to each video frame.

[0018] The lighting maps were then applied to the original real video to obtain a simulated relighting video.

[0019] Furthermore, the light source color I between each simulated point light source in S22 p The formula for calculating simulated illumination is:

[0020]

[0021] In the formula: I υ θ represents the simulated lighting at pixel υ in the lighting map. υ,p> Indicates the relative angle between pixel υ and pixel p; I p This indicates the color corresponding to the simulated point light source.

[0022] Furthermore, S3 specifically includes the following steps:

[0023] S31: Use 3D rendered video and simulated relighting video as training video data of the original video, define 3D rendered video as the first training data, and define simulated relighting video as the second training data.

[0024] S32: Construct a video acquisition model for background replacement and video relighting based on text description, which includes a mask video acquisition module, a foreground video acquisition module, an input module, a stitching module, a target video description module, a video encoding module, and a relighting video generation module;

[0025] The masked video acquisition module is used to obtain the foreground masked video frame image of the training video data based on a preset mask extraction model.

[0026] The foreground video acquisition module is used to perform an element-wise multiplication operation on the foreground mask video frame image and the video frames in the training video data to obtain the original foreground video frame image.

[0027] The input module is used to transmit the original foreground video frame image and the preset video noise image to the stitching module;

[0028] The stitching module is used to perform element-wise summation on the original foreground video frame image and the video noise image to obtain each optimized video frame image, and then stitch the optimized video frame images according to the frame sequence of the desired target video to obtain the stitched video.

[0029] The target video description module is used to obtain the target description of the desired target video based on a multimodal large model, and to vectorize the target description based on a text encoder to obtain a text description vector.

[0030] The video encoding module is used to perform vectorization processing on the spliced ​​video based on the vaeencoder encoder to obtain the video encoding vector;

[0031] The relighting video generation module is used to perform vector attention interaction operation between the text description vector and the video encoding vector through the Attention module, and to obtain the simulated relighting video based on the output of the Attention module; and the relighting video generation module includes a subject model for learning the relighting effect and a style model for isolating the style of 3D rendered video data.

[0032] S33: Fix the model parameters of the main model, input the first training data into the style model to train the model and obtain the model in the first training stage;

[0033] S34: Keep the parameter weights of the style model unchanged in the first training phase model, and input the first training data into the main model and the style model respectively, and input the second training data into the main model to train the model and obtain the second training phase model.

[0034] S35: Based on the constructed model loss function, obtain the current training loss between the simulated relighting video output by the model in the second training stage and the desired target video, and determine whether the current training loss meets the preset loss threshold.

[0035] If so, the second training model obtained at this time will be used as the optimal video acquisition model;

[0036] Otherwise, the model weight parameters of the main model are adaptively adjusted based on the backpropagation method, and step S33 is repeated.

[0037] By using the optimal video acquisition model, the video acquisition process of background replacement and video relighting based on text description is realized.

[0038] Furthermore, the formula for constructing the model loss function in S34 is as follows:

[0039]

[0040] In the formula: F represents the style model; Θ represents the trainable parameters of the iterative training process; X represents the stitched video; T represents the text description vector of the desired target video; t represents the time step; and represents the noise vector in the preset video noise image.

[0041] Furthermore, S1 specifically includes the following steps:

[0042] S11: Collect and acquire the materials needed to render the video;

[0043] Furthermore, the materials include at least: a set of character materials, a set of motion materials, a set of camera movement methods, and a set of scene materials; the scene material set includes real-world indoor and outdoor environment materials.

[0044] S12: Based on the character material preset from the character material set, randomly select any one action material and camera movement mode from the action material set and the camera movement mode set;

[0045] S13: Based on randomly selected motion materials and camera movement, 3D rendering is performed on different scene materials in the scene material set using Unreal Engine UE5 to obtain 3D video pairs with the same foreground characters but different backgrounds and lighting; the 3D video pairs are the 3D rendered video data.

[0046] This invention provides a method for background replacement and video relighting based on text description, with the following beneficial effects:

[0047] 1. Normal video is obtained by extracting video normals from the original real video; pixel set of each video frame is obtained based on the normal video; a preset number of pixels in the pixel set are randomly selected as simulated point light sources in the background environment pixels, and the color corresponding to the simulated point light source is randomly selected and reapplied to each video frame to obtain the lighting map corresponding to each video frame; the lighting map is then applied to the original real video to obtain simulated relighting video; a large-scale, high-quality video relighting data is constructed for subsequent training, and the constructed simulated relighting video data can achieve a relighting effect with consistent foreground, high-quality background and blended lighting.

[0048] 2. By constructing a video acquisition model based on text description for background replacement and video relighting, and combining it with the proposed data adaptive training method, the influence of 3D data style in 3D rendering data is avoided when training the video acquisition model with simulated relighting videos, which greatly improves the relighting effect of the model. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of the background replacement and video relighting method based on text description of the present invention;

[0051] Figure 2 This is a technical roadmap diagram for obtaining 3D rendered video in this embodiment;

[0052] Figure 3 This is a block diagram illustrating the technical route for obtaining simulated re-lighting video in this embodiment;

[0053] Figure 4 This is a schematic diagram of the video acquisition model for background replacement and video relighting based on text description constructed in this embodiment.

[0054] Figure 5 This is a screenshot of the 3D rendering data constructed in this embodiment.

[0055] Figure 6 This is a screenshot from a video simulating real-world data of relighting in this embodiment;

[0056] Figure 7 This is a simulation diagram of the reasoning result of the method described in this embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] This embodiment provides a method for background replacement and video relighting based on text description, such as... Figure 1 As shown, the specific steps include:

[0059] S1: Construct 3D rendered video data, such as Figure 2 As shown, the specific steps include:

[0060] S11: Collect and acquire the materials needed to render the video;

[0061] Furthermore, the materials include at least: a set of character materials, a set of motion materials, a set of camera movement methods, and a set of scene materials; the scene material set includes real-world indoor and outdoor environment materials.

[0062] S12: Based on the character material preset from the character material set, randomly select any one action material and camera movement mode from the action material set and the camera movement mode set; wherein the camera movement mode can be determined by setting the direction and distance of the camera movement in the preset rendering script, and the camera movement mode is, for example, from top to bottom, from left to right, from near to far, or follow, etc.

[0063] S13: Based on randomly selected motion footage and camera movement, 3D rendering is performed on different scene footage in the scene footage set using Unreal Engine UE5 to obtain 3D video pairs with the same foreground character but different backgrounds and lighting; these 3D video pairs are the 3D rendered video data, such as... Figure 5 As shown; in this embodiment, the lighting is different in different scene materials and in different positions of the same scene. The lighting can be changed by changing the position of the character; the method of 3D rendering of different scene materials in the scene material set based on Unreal Engine UE5 is a well-known existing technology and will not be described in detail here.

[0064] S2: Obtain the original real video to be processed, and perform video normal extraction processing on the original real video to obtain the normal video; and obtain the pixel set of each video frame based on the normal video;

[0065] A predetermined number of pixels are randomly selected from the pixel set as simulated point light sources in the background environment pixels. The color corresponding to the simulated point light source is randomly selected and mapped to its corresponding pixel position to obtain the illumination map corresponding to each video frame. The obtained illumination maps are then used to replace the video frames in the original real video to obtain the simulated relighting video.

[0066] like Figure 3 As shown, the specific steps include:

[0067] S21: Obtain the original real video to be processed, and perform video normal extraction processing on the original real video based on the normal extraction model NormalCrafter to obtain the normal video; where video normal refers to a kind of texture detail in the video frame image, which represents the concavity and convexity of the image at each point. The pixel value at each point, in the preset three-dimensional coordinate system, has three components that represent a vector, that is, the direction of the normal at that point; video normal is used to characterize the texture details of objects in the image;

[0068] And obtain the pixel set P of each video frame based on the normal video;

[0069] S22: Randomly select a preset number of pixels p from the pixel set P as simulated point light sources in the background environment pixels, and randomly select the light source color I corresponding to the simulated point light source. p By calculating the light source color I between each simulated point light source p Simulated lighting is used to obtain simulated relit video frames; and the simulated relit video frames are then reapplied to each video frame to obtain the lighting map corresponding to each video frame.

[0070] And the light source colors I between each simulated point light source p The formula for calculating simulated illumination is:

[0071]

[0072] In the formula: I υ θ represents the simulated lighting at pixel υ in the lighting map. υ,p> Indicates the relative angle between pixel υ and pixel p; I p This indicates the color corresponding to the simulated point light source;

[0073] The lighting maps were then applied to the original real video to obtain simulated relighting video, such as... Figure 6 As shown;

[0074] S3: Use 3D rendered video and simulated relighting video as training video data for the original video, and train the constructed video acquisition model for background replacement and video relighting based on text description to obtain the optimal video acquisition model; and realize the video acquisition process for background replacement and video relighting based on text description through the optimal video acquisition model.

[0075] Specifically, the following steps are included:

[0076] S31: Use 3D rendered video and simulated relighting video as training video data of the original video, define 3D rendered video as the first training data, and define simulated relighting video as the second training data.

[0077] S32: Construct a video acquisition model for background replacement and video relighting based on text description, such as... Figure 4 As shown, it includes a mask video acquisition module, a foreground video acquisition module, an input module, a stitching module, a target video description module, a video encoding module, and a relighting video generation module;

[0078] The masked video acquisition module is used to obtain the foreground masked video frame image of the training video data based on the preset mask extraction model RMBG-2.0;

[0079] The foreground video acquisition module is used to perform an element-wise multiplication operation on the foreground mask video frame image and the video frames in the training video data to obtain the original foreground video frame image.

[0080] The input module is used to transmit the original foreground video frame image and the preset video noise image to the stitching module;

[0081] The stitching module is used to perform element-wise summation on the original foreground video frame image and the video noise image to obtain each optimized video frame image, and then stitch the optimized video frame images according to the frame sequence of the desired target video to obtain the stitched video.

[0082] The target video description module is used to obtain the target description of the desired target video based on the multimodal large model Qwen2.5-VL-7B-Instruct, and to vectorize the target description based on the text encoder to obtain the text description vector.

[0083] The video encoding module is used to perform vectorization processing on the spliced ​​video based on the vaeencoder encoder to obtain the video encoding vector;

[0084] The re-lighting video generation module is used to perform vector attention interaction operation on the text description vector and the video encoding vector through the Attention module, and obtain the simulated re-lighting video based on the output of the Attention module; and the re-lighting video generation module includes a subject model and a style model.

[0085] The main model is specifically the video controllable generation model Wan2.1-Fun-1.3B-Control, and the style model is the LoRA model of the main model; the function of the style model is to isolate the style of the 3D data; the function of the main model is to learn the relighting effect of the video simulation.

[0086] S33: Fix the model parameters of the main model, input the first training data into the style model to train the model and obtain the model in the first training stage;

[0087] S34: The parameter weights of the style model in the first training phase model are fixed. Simultaneously, the first training data is input into the main model and the style model respectively, and the second training data is input into the main model for model training to obtain the second training phase model. In this embodiment, based on training video data, foreground mask video frame images, and text description vectors, the style model in the first training model is frozen during the training process of obtaining the second training model. This means the style model is not used, thus isolating the 3D data style of the 3D rendered video. By training the main model's ability to simulate relighting, a relighting effect with consistent foreground, high-quality background, and blended lighting is achieved. Specifically, the data construction methods in steps S1 and S2 obtain high-quality 3D rendered videos and simulated relighting videos as training data for the relighting model. The advantages of 3D rendered video data are its diversity, adherence to physical laws, and close resemblance to real-world lighting. The disadvantage is that the style of the 3D rendered video data is inconsistent with... The real world has its differences; the disadvantage of simulated lighting videos is that the quality of simulated lighting videos is low and does not quite match the lighting of natural scenes, while the advantage is that the data style matches the real world. In order to make full use of the advantages of 3D rendered video data and avoid introducing unrealistic styles from 3D data, this method proposes a data-adaptive model structure and training method. That is, the training phase in this embodiment is divided into two stages: In the first stage, only 3D rendered video data is used to train the style model part of the constructed video acquisition model (i.e., the LoRA model of the main body model), aiming to learn the style of 3D data; In the second stage, 3D rendered video data and simulated lighting videos acquired by S2 are used simultaneously to train the main body model part of the video acquisition model. During the training process, 3D rendered video data is processed by the style model while simulated lighting video data is not processed by the style model, thereby isolating the style of 3D data and learning only its natural lighting effect.

[0088] S35: Based on the constructed model loss function, obtain the current training loss between the simulated relighting video output by the model in the second training stage and the desired target video, and determine whether the current training loss meets the preset loss threshold.

[0089] Specifically, the formula for constructing the model loss function is as follows:

[0090]

[0091] In the formula: F represents the style model LORA; Θ represents the trainable parameters of the iterative training process; X represents the stitched video; T represents the text description vector of the desired target video; t represents the time step; and represents the noise vector in the preset video noise image.

[0092] If so, the second training model obtained at this time will be used as the optimal video acquisition model;

[0093] Otherwise, the model weight parameters of the main model are adaptively adjusted based on the backpropagation method, and step S33 is repeated.

[0094] By using the optimal video acquisition model, the video acquisition process of background replacement and video relighting based on text description is realized.

[0095] Compared with the prior art, the beneficial effects of the method described in this embodiment are as follows:

[0096] 1. Normal video is obtained by extracting video normals from the original real video; pixel set of each video frame is obtained based on the normal video; a preset number of pixels in the pixel set are randomly selected as simulated point light sources in the background environment pixels, and the color corresponding to the simulated point light source is randomly selected and reapplied to each video frame to obtain the lighting map corresponding to each video frame; the lighting map is then applied to the original real video to obtain simulated relighting video; a large-scale, high-quality video relighting data is constructed for subsequent training, and the constructed simulated relighting video data can achieve a relighting effect with consistent foreground, high-quality background and blended lighting.

[0097] 2. By constructing a video acquisition model for background replacement and video relighting based on text description, and in order to learn only the high-quality relighting effects of the rendered data and avoid introducing unrealistic styles from the 3D data, the method described in this embodiment combines the proposed data adaptive training method. This avoids the influence of 3D data styles in the 3D rendering data when training the video acquisition model with simulated relighting videos, greatly improving the model's relighting effect. Figure 7 The figure shown is a simulation comparison of the input and output results of the optimal video acquisition model.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for background replacement and video relighting based on text description, characterized in that, Specifically, the following steps are included: S1: Construct 3D rendering video data; S2: Obtain the original real video to be processed, and perform video normal extraction processing on the original real video to obtain the normal video; and obtain the pixel set of each video frame based on the normal video; A predetermined number of pixels are randomly selected from the pixel set as simulated point light sources in the background environment pixels. The color corresponding to the simulated point light source is randomly selected and mapped to its corresponding pixel position to obtain the illumination map corresponding to each video frame. The obtained illumination maps are then used to replace the video frames in the original real video to obtain simulated relighting video. S3: Use 3D rendered video and simulated relighting video as training video data for the original video, and train the constructed video acquisition model based on text description for background replacement and video relighting to obtain the optimal video acquisition model; and use the optimal video acquisition model to realize the video acquisition process based on text description for background replacement and video relighting.

2. The video generation method based on text description for background replacement and video light modulation according to claim 1, characterized in that, S2 specifically includes the following steps: S21: Obtain the original real video to be processed, and perform video normal extraction processing on the original real video based on the normal extraction model NormalCrafter to obtain the normal video; And obtain the pixel set P of each video frame based on the normal video; S22: Randomly select a preset number of pixels p from the pixel set P as simulated point light sources in the background environment pixels, and randomly select the light source color I corresponding to the simulated point light source. p By calculating the light source color I between each simulated point light source p Simulated lighting is used to obtain simulated relit video frames; and the simulated relit video frames are then reapplied to each video frame to obtain the lighting map corresponding to each video frame. The lighting maps were then applied to the original real video to obtain a simulated relighting video.

3. The video generation method based on text description for background replacement and video light modulation according to claim 2, characterized in that, The light source color I between each simulated point light source in S22 p The formula for calculating simulated illumination is: In the formula: I υ θ represents the simulated lighting at pixel υ in the lighting map. υ,p> Indicates the relative angle between pixel υ and pixel p; I p This indicates the color corresponding to the simulated point light source.

4. The video generation method based on text description for background replacement and video light modulation according to claim 3, characterized in that, S3 specifically includes the following steps: S31: Use 3D rendered video and simulated relighting video as training video data of the original video, define 3D rendered video as the first training data, and define simulated relighting video as the second training data. S32: Construct a video acquisition model for background replacement and video relighting based on text description, which includes a mask video acquisition module, a foreground video acquisition module, an input module, a stitching module, a target video description module, a video encoding module, and a relighting video generation module; The masked video acquisition module is used to obtain the foreground masked video frame image of the training video data based on a preset mask extraction model. The foreground video acquisition module is used to perform an element-wise multiplication operation on the foreground mask video frame image and the video frames in the training video data to obtain the original foreground video frame image. The input module is used to transmit the original foreground video frame image and the preset video noise image to the stitching module; The stitching module is used to perform element-wise summation on the original foreground video frame image and the video noise image to obtain each optimized video frame image, and then stitch the optimized video frame images according to the frame sequence of the desired target video to obtain the stitched video. The target video description module is used to obtain the target description of the desired target video based on a multimodal large model, and to vectorize the target description based on a text encoder to obtain a text description vector. The video encoding module is used to perform vectorization processing on the spliced ​​video based on the vaeencoder encoder to obtain the video encoding vector; The relighting video generation module is used to perform vector attention interaction operation between the text description vector and the video encoding vector through the Attention module, and to obtain the simulated relighting video based on the output of the Attention module; and the relighting video generation module includes a subject model for learning the relighting effect and a style model for isolating the style of 3D rendered video data. S33: Fix the model parameters of the main model, input the first training data into the style model to train the model and obtain the model in the first training stage; S34: Keep the parameter weights of the style model unchanged in the first training phase model, and input the first training data into the main model and the style model respectively, and input the second training data into the main model to train the model and obtain the second training phase model. S35: Based on the constructed model loss function, obtain the current training loss between the simulated relighting video output by the model in the second training stage and the desired target video, and determine whether the current training loss meets the preset loss threshold. If so, the second training model obtained at this time will be used as the optimal video acquisition model; Otherwise, the model weight parameters of the main model are adaptively adjusted based on the backpropagation method, and step S33 is repeated. By using the optimal video acquisition model, the video acquisition process of background replacement and video relighting based on text description is realized.

5. The video generation method based on text description for background replacement and video light modulation according to claim 4, characterized in that, The formula for constructing the model loss function in S34 is as follows: In the formula: F represents the style model; Θ represents the trainable parameters of the iterative training process; X represents the stitched video; T represents the text description vector of the desired target video; t represents the time step; and represents the noise vector in the preset video noise image.

6. The video generation method based on text description for background replacement and video light modulation according to claim 1, characterized in that, S1 specifically includes the following steps: S11: Collect and acquire the materials needed to render the video; Furthermore, the materials include at least: a set of character materials, a set of motion materials, a set of camera movement methods, and a set of scene materials; the scene material set includes real-world indoor and outdoor environment materials. S12: Based on the character material preset from the character material set, randomly select any one action material and camera movement mode from the action material set and the camera movement mode set; S13: Based on randomly selected motion materials and camera movement, 3D rendering is performed on different scene materials in the scene material set using Unreal Engine UE5 to obtain 3D video pairs with the same foreground characters but different backgrounds and lighting; the 3D video pairs are the 3D rendered video data.

Images