Dynamic scene re-operation mirror video generation method and system based on diffusion model

Abstract

The invention discloses a dynamic scene re-operation mirror video generation method and system based on a diffusion model, and belongs to the technical field of computer vision and video generation. A diffusion generation architecture with a control branch is adopted, and the core is composed of an embedded layer, a main branch and a control branch. In the control branch, the output of each sub-block is added with the output of the corresponding block of the main branch after being processed by the zero initial linear layer, and the sum is input into the next block of the main branch. During training, generating a rendered video by using the target video and the reference video in the same scene; and inputting the target video latent variable after noise addition into the control branch, inputting the splicing result of the target video, the reference video and the rendering video latent variable into the main branch, and simultaneously providing the text latent variable of the reference video for the two branches as a condition. During generation, the model finally generates a target video latent variable through step-by-step denoising and decodes the target video latent variable into a target track video, and it is ensured that the motion of a moving object in a scene of a generated video and a reference video is consistent at the same time.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and video generation technology, specifically relating to a method and system for generating dynamic scene re-motion video based on a diffusion model. Background Technology

[0002] In film and television production, virtual reality, and other fields, generating videos of the same dynamic scene from different camera perspectives is a key requirement. Existing methods mainly fall into two categories: Trajectory Crafter injects point cloud projections from the new camera perspective into the generative model, filling in gaps in the point cloud and injecting reference video into the model via cross-attention to generate a video of the original scene from the new perspective. However, the completion effect of this method is affected by the quality of the point cloud, and the effect of injecting reference video information via cross-attention is poor, making it difficult to ensure that the character movements in the newly generated video are consistent with the original scene. ReCamera master concatenates the target video and the reference video as input to the generative model, and inputs the camera's intrinsic and extrinsic parameters into the network. This method can better ensure that the character movements in the newly generated video are consistent with the original scene, but due to the lack of point cloud projection information, simply inputting the camera's intrinsic and extrinsic parameters makes it difficult to accurately control camera movement.

[0003] Traditional methods rely on simultaneous multi-camera shooting or complex 3D scene reconstruction, which suffers from high cost and poor flexibility. While existing deep learning-based generative methods can synthesize videos from new perspectives, they face the following technical bottlenecks:

[0004] 1. Insufficient consistency of dynamic scenes: The movements of moving objects in the generated video are difficult to keep in strict temporal alignment with the reference video;

[0005] 2. Low efficiency of multimodal conditional fusion: Text, camera trajectory and video features are difficult to coordinate effectively, resulting in semantic deviations in the generated content;

[0006] 3. Low accuracy of cross-camera trajectory conversion: Point cloud projection errors can easily cause geometric distortion or object misalignment in the generated video.

[0007] To address the aforementioned issues, this invention proposes an innovative method that ensures geometric consistency through point cloud reprojection and combines multimodal conditional injection with an improved diffusion model architecture to achieve high-quality cross-camera trajectory video generation. Summary of the Invention

[0008] To address the aforementioned issues, this invention proposes a dynamic scene re-motion video generation method and system based on a diffusion model. By combining point cloud reprojection, multimodal conditional injection, and an improved diffusion model, high-quality videos of dynamic scenes from the target's perspective are generated based on the input reference video, target camera trajectory, and text description. Simultaneously, the temporal consistency of moving objects within the scene and the accuracy of their geometric structures are ensured. This method is applicable to scenarios such as virtual film production, autonomous driving simulation, and multi-view video generation.

[0009] The present invention adopts the following technical solution:

[0010] In a first aspect, this invention proposes a method for generating dynamic scene re-motion video based on a diffusion model, comprising the following steps:

[0011] A diffusion model with a control branch is constructed as a generative model, consisting of an embedding layer, a main branch, and a control branch. The main branch is a pre-trained DiT model, and the control branch contains a pre-trained DiT model and a linear layer with weights initialized to zero. The output of each Transformer block of the pre-trained DiT model in the control branch is processed by the linear layer with weights initialized to zero and then added to the output of the corresponding Transformer block of the pre-trained DiT model in the main branch. The result of the addition is used as the input of the next Transformer block in the main branch.

[0012] During the training phase, two videos with different camera trajectories in the same dynamic scene are used as the target video and reference video, respectively. The reference video provides the point cloud, and the target video provides the camera trajectory to generate the rendered video. The latent variables of the target video, reference video, and rendered video are obtained, as well as the text latent variables of the reference video. The latent variables of the target video after progressively adding noise are used as the input of the control branch. The concatenation result of the latent variables of the target video, reference video, and rendered video is used as the input of the main branch. The text latent variables are used as the conditional input of the main branch and the control branch. The generative model is trained with ∈-x0 as the prediction target. During training, the pre-trained DiT model in the control branch is frozen. ∈ represents Gaussian noise, and x0 represents the target video.

[0013] In the generation phase, point clouds are provided by the reference video, and a rendered video is generated given the target trajectory. Pure noise variables are used as input to the control branch, and the concatenation result of pure noise variables, latent variables of the reference video, and latent variables of the rendered video is used as input to the main branch. Text latent variables are used as conditional inputs to the main branch and the control branch. The model gradually denoises and generates latent variables of the target video, and decodes to obtain the target video with the given target trajectory.

[0014] Furthermore, the video generation process for different camera trajectories in the same dynamic scene includes:

[0015] Load the scene file, dynamic object model and its corresponding animation file, and give the initial position and orientation of the camera;

[0016] At the initial position and orientation of the camera, acquire the depth map and normal map of the scene. Use the normal map to filter planar regions where the angle between the normal and the Z-axis is <10°. Obtain the depth values ​​of the pixels in the planar regions from the depth map, and calculate the candidate positions of the dynamic object model in the scene using the ray propagation formula.

[0017] Placing a dynamic object model at any candidate location and binding it with an animation file is considered as a dynamic scene.

[0018] Randomly select a pixel in the depth map and take a random depth value that is less than the pixel's depth value. Combine this with the ray propagation formula to calculate the camera position for the next keyframe. Introduce a random deflection angle within ±10 degrees as the camera orientation for the next keyframe, based on the alignment with the dynamic object model. Repeat the process of randomly selecting pixels in the depth map multiple times to generate the camera positions and orientations of several keyframes, forming a camera trajectory. Generate 10-20 camera trajectories for each dynamic scene.

[0019] In this dynamic scene, render a video for each camera trajectory.

[0020] Furthermore, the pre-trained DiT models in the main branch and control branch are initialized from the Wanxiang model.

[0021] Furthermore, the process of generating rendered videos during the training phase includes:

[0022] Obtain the reference camera trajectory corresponding to the reference video and extract the point cloud from the reference video;

[0023] Obtain the target camera trajectory corresponding to the target video;

[0024] Based on the target camera trajectory, reference camera trajectory, and reference video point cloud, the coordinate transformation formula is used. The reference video point cloud is projected onto the target camera trajectory to obtain the rendered video; where P target For the target video point cloud, T target For the target camera trajectory, For the inverse transform of the reference camera trajectory, P ref For reference video point clouds, P target For the target video point cloud;

[0025] Convert the target video point cloud into a rendered video.

[0026] Furthermore, WanVAE is used to encode the target video, reference video, and rendered video. The encoding results are then embedded to encode feature vectors that match the input dimensions of the pre-trained DiT model, which are used as latent variables for the target video, reference video, and rendered video, respectively. The rendered video needs to be downsampled before encoding, reducing its length and width to 1 / 4 of the original size.

[0027] Furthermore, the method for obtaining the textual latent variables of the reference video is as follows:

[0028] Semantic annotation is performed on the reference video to obtain descriptive text information related to the video;

[0029] Text information is input into the T5 model for encoding, and the input text is mapped into a fixed-dimensional feature vector to obtain text latent variables.

[0030] Furthermore, during the training phase, the latent variable x of the target video at time step t is calculated using the noise addition formula. t =(1-σ t )×x0+σ t ×∈, where σ t Let t be the noise intensity at time step t.

[0031] Furthermore, the process of generating and rendering the video during the generation phase includes:

[0032] Input the reference video into the VGGT model to generate reference video point cloud and reference camera trajectory;

[0033] Given the target camera trajectory;

[0034] Based on the target camera trajectory, reference camera trajectory, and reference video point cloud, the reference video point cloud is projected onto the target camera trajectory using a coordinate transformation formula to obtain the rendered video. The formula is the same as above.

[0035] Furthermore, during the training phase, the model outputs predicted noise, which represents the component x that should be removed from the noise variable input at the current time step. pred To approximate the original image, it is represented as:

[0036] X t-1 =X t +x pred +∈

[0037] Among them, X t X represents the noise variable of the input at time step t. t-1 The noise variables after denoising are iterated step by step until the latent variables of the target video are restored, and the target video with a given target trajectory is decoded.

[0038] Secondly, this invention proposes a dynamic scene re-motion video generation system based on a diffusion model, comprising:

[0039] The video generation model module uses a diffusion model with a control branch as the generation model. It consists of an embedding layer, a main branch, and a control branch. The main branch is a pre-trained DiT model, and the control branch contains a pre-trained DiT model and a linear layer with weights initialized to zero. The output of each Transformer block of the pre-trained DiT model in the control branch is processed by the linear layer with weights initialized to zero and then added to the output of the corresponding Transformer block of the pre-trained DiT model in the main branch. The result of the addition is used as the input of the next Transformer block in the main branch.

[0040] The model training module utilizes two videos with different camera trajectories in the same dynamic scene as the target video and reference video, respectively. The reference video provides the point cloud, and the target video provides the camera trajectory, generating a rendered video. It acquires the latent variables of the target video, reference video, and rendered video, as well as the textual latent variables of the reference video. The latent variables of the target video, after progressively adding noise, are used as input to the control branch. The concatenated result of the latent variables of the target video, reference video, and rendered video is used as input to the main branch. The textual latent variables are used as conditional inputs to both the main branch and the control branch. The model is trained with ∈-x0 as the prediction target, and the pre-trained DiT model in the control branch is frozen during training. ∈ represents Gaussian noise, and x0 represents the target video.

[0041] The re-motion video generation module uses a reference video to provide point clouds and generates a rendered video given a target trajectory. Pure noise variables are used as input to the control branch, and the concatenation result of pure noise variables, reference video latent variables, and rendered video latent variables is used as input to the main branch. Text latent variables are used as conditional inputs to the main branch and control branch. The model gradually denoises and generates target video latent variables, and decodes to obtain the target video with the given target trajectory.

[0042] Compared with existing technologies, the advantages of this invention are:

[0043] 1. By injecting reference videos into the model, the generated target video and the reference video have good consistency in scene, especially in the consistency of human movements.

[0044] 2. Injecting the rendered video after point cloud reprojection into the model increases the geometric prior of the network input. By using point cloud reprojection information, the camera's ability to control camera movement is enhanced, making the generated video's camera movement effect more accurate.

[0045] 3. The model contains a control branch. Freezing the pre-trained DiT model weights in the control branch prevents the model's generation ability from decreasing with fine-tuning. The generation results of this method are superior to those of other methods.

[0046] This invention ensures that the generated video and the reference video have consistent motion of moving objects in the scene at the same time, providing a complete and effective solution for multi-view video generation of dynamic scenes, and has important application value and innovative significance. Attached Figure Description

[0047] Figure 1 This is an overall framework diagram of the present invention;

[0048] Figure 2 This invention uses Blender to render videos with different camera movements in the same dynamic scene;

[0049] Figure 3-5 This is a video demonstration of the rewind camera technique of the present invention. Detailed Implementation

[0050] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0051] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0052] like Figure 1 As shown, the present invention proposes a dynamic scene re-motion video generation method based on a diffusion model, which mainly includes the following steps:

[0053] Step 1: Dataset Rendering

[0054] To obtain videos of different camera movements in the same dynamic scene for training the model of this invention, this invention uses Blender rendering to obtain data. The rendering process of this invention consists of the following steps:

[0055] 1. Human model and scene acquisition and initial camera settings:

[0056] Download a human model with standard skeletal rigging and its corresponding skeletal animation files from the Mixamo platform, ensuring the model format is compatible with Blender (e.g., FBX format). Simultaneously, download suitable Blender scene files from professional resource websites, covering scenes of different styles (e.g., indoor, outdoor) and scales. In each Blender scene, select a camera placement position, controlling the camera's height above the ground to approximately 2 meters. This height provides a comprehensive view of the scene and characters while conforming to conventional visual observation habits. The camera position will serve as the initial camera position for subsequent video rendering, laying the foundation for the overall video's shooting perspective.

[0057] 2. Ground inspection:

[0058] At the current camera position, using Blender's rendering settings, the depth map and normal map rendering passes are enabled to render the scene and obtain high-precision depth map and normal map data. The depth map records the distance information of each pixel in the scene from the camera, while the normal map stores the normal direction information of each pixel's surface. In the normal map, pixels whose normal direction makes an angle of less than 10° with the positive z-axis direction (i.e., the direction towards the sky) are selected. The areas containing these pixels are determined to be planar areas where characters can be placed, effectively avoiding situations where characters are placed in unreasonable positions such as vertical walls.

[0059] Based on the camera's extrinsic parameters (including the camera's position and rotation information in the world coordinate system), the camera's optical center (rays_o) corresponding to each pixel on the ground can be accurately obtained. This parameter represents the starting position of the ray corresponding to each pixel in 3D space. Combining the camera's intrinsic parameters (including focal length, principal point coordinates, etc.) and extrinsic parameters, the ray propagation direction (rays_d) is calculated through linear algebra operations. To ensure consistency and stability of the calculation, rays_d is a normalized unit vector. According to the principle of ray propagation, any point P on the ray can be represented by the formula P = rays_o + rays_d * d, where d represents the depth value of the pixel in the planar region, obtained from the depth map. The position calculated by this formula is the location where the ray originates from the camera and intersects the ground; this position will be used as a candidate position for placing the character.

[0060] 3. Placement of animated characters:

[0061] Based on the ground location information calculated in the preceding ground detection steps, a human model is placed at any of these candidate locations. To enrich the character's movement in the video, a skeletal animation is randomly selected from the downloaded skeletal animation collection. Using Blender's skeletal rigging and animation constraint functions, the human model is precisely bound to the selected skeletal animation. By adjusting parameters such as weights and keyframes, it is ensured that the character model can accurately reproduce the dynamic movements defined by the skeletal animation, thereby generating a dynamic character with vivid movements in the scene.

[0062] 4. Camera trajectory generation:

[0063] The camera position is set manually in the scene as the initial camera position, and the direction in which the camera center is pointed at the person is taken as the initial orientation. During the rendering of different camera movement videos of the same scene, the initial camera position and orientation are kept unchanged to ensure the visual continuity between different camera movement videos.

[0064] In the depth map generated during the ground detection step, a pixel d' is randomly selected using a random number generation algorithm. The next camera keyframe position P is calculated using the formula P = rays_o + rays_d * d. To prevent the camera from passing through walls or the ground unreasonably, d in the formula is taken as a random number less than the depth value d' of the pixel, thus ensuring that the camera position is within a reasonable spatial range. After determining the camera keyframe position, the camera orientation is set to face the subject, and a random deflection angle within ±10 degrees is added as the orientation of the camera keyframe. This ensures that the camera always focuses on the subject while introducing a degree of randomness to enrich the camera's shooting perspective. Multiple camera keyframes are generated using this method, and these keyframes are then connected sequentially to form a complete camera motion trajectory.

[0065] 5. Video rendering:

[0066] For the same scene, 10-20 different camera motion trajectories are randomly selected for rendering. Each rendering starts from the scene state at time t=0, strictly maintaining the same initial camera position and orientation for each rendered video, and ensuring the initial orientation is always aligned with the added dynamic characters in the scene, guaranteeing a consistent starting perspective across different camera movements. During rendering, Blender's EEVEE engine is enabled, and based on the video's purpose and quality requirements, the rendering resolution is set to 480×832, rendering sampling rate to 32, and anti-aliasing level to 64 for efficient scene rendering. Each camera trajectory generates 85 frames of high-quality dynamic scene camera movement video. Each video renders an RGB image and a depth map; additionally, camera intrinsic and extrinsic parameters are saved after rendering.

[0067] Step 2: Model Construction

[0068] This invention proposes a diffusion model with a control branch as a generative model. It comprises two branches: a control branch and a main branch. The main branch is a pre-trained DiT model, while the control branch includes the pre-trained DiT model and linear layers with weights initialized to zero. The output of each Transformer block of the pre-trained DiT model in the control branch is processed by the linear layers with zero weights and then added to the output of the corresponding Transformer block of the pre-trained DiT model in the main branch. The sum is used as the input to the next Transformer block in the main branch.

[0069] In one specific embodiment of the present invention, the control branch is based on the original Wanxiang model, whose structure mainly consists of an embedding layer, 30 Transformer blocks, and an output layer. The control branch uses the pre-trained weights of the Wanxiang model. The main branch also includes 30 Transformer blocks. The main branch is similar in structure to the original Wanxiang model, but adds a rendering embedding layer for encoding the latent variables of the rendered video. In this embodiment, the rendering embedding layer is a fully connected neural network that maps the latent variables of the rendered video to feature vectors that match the input dimension of the pre-trained DiT model. The input of the control branch is only the latent variables of the target video. The output of each Transformer block in the control branch is added to the corresponding Transformer block of the main branch through a zero linear layer, and the resulting input is used as the input of the next Transformer block. The zero linear layer is a linear transformation layer with weights initialized to zero. During training, it learns parameters to adjust the fusion method of features in the control branch and the main branch.

[0070] Step 3: Point cloud reprojection to generate rendered video

[0071] For several videos rendered in the same scene, two different videos are randomly selected. One video is designated as the target video, and the other is used as the reference video. After obtaining the target camera trajectory, the coordinate transformation formula is applied based on the target camera trajectory, the reference camera trajectory, and the reference video point cloud. The reference video point cloud is projected onto the target camera trajectory to obtain the rendered video; where P target For the target video camera trajectory point cloud, T target For the target camera trajectory, For the inverse transform of the reference camera trajectory, P ref For reference video point clouds, P targetThe target video point cloud is generated and then converted into a rendered video. The coordinate transformation formula is based on the principle of rigid body transformation in three-dimensional space, and the point cloud is transformed in different coordinate systems through matrix multiplication.

[0072] Step 4: Text annotation and encoding

[0073] VideoLLaMA was used to semantically annotate the reference video, obtaining descriptive text information related to the video. This text information was then input into a T5 model for encoding, yielding textual latent variables. The T5 model is a pre-trained language model based on the Transformer architecture. Through self-supervised learning on a large-scale text corpus, it can map the input text into fixed-dimensional feature vectors, i.e., textual latent variables.

[0074] Step 5: Video Encoding

[0075] The reference video is encoded using WanVAE and projected through an embedding layer onto a feature vector that matches the input dimension of the pre-trained DiT model, yielding the latent variables of the reference video. The original length of the reference video is 81 frames, with a resolution of 480×832. WanVAE can compress the original video to 1 / 4 of its original length in the temporal dimension and to 1 / 8 of its original width and height, resulting in 21 frames of 60×104 latent variables, each with 16 channels. The WanVAE model is a variational autoencoder, comprising an encoder and a decoder structure. The encoder compresses video frames into low-dimensional feature vectors using a convolutional neural network, while the decoder restores the feature vectors to video frame images through deconvolution operations. The encoder uses a convolutional neural network for feature extraction, and the decoder uses a deconvolutional neural network for feature restoration. The convolutional neural network contains multiple convolutional layers and RMS-norm layers to extract local features from the video frames; the deconvolutional neural network contains multiple upsample layers to restore the low-dimensional feature maps to high-resolution video frame images through upsample operations.

[0076] The target video is also encoded using WanVAE and projected through an embedding layer onto a feature vector that matches the input dimension of the pre-trained DiT model to obtain the latent variables of the target video.

[0077] Encoding the rendered video requires a downsampling operation, reducing both its length and width to one-quarter of their original size. This downsampling operation employs bilinear interpolation to maintain image smoothness while reducing video resolution. The downsampled rendered video is then input into the WanVAE model for encoding and projected through an embedding layer onto a feature vector that matches the input dimensions of the pre-trained DiT model, yielding the latent variables of the rendered video.

[0078] Step Six: Model Training

[0079] During the training phase, two videos with different camera trajectories in the same dynamic scene are used as the target video and reference video, respectively. The reference video provides the point cloud, and the target video provides the camera trajectory to generate the rendered video. The latent variables of the target video, reference video, and rendered video are obtained, as well as the text latent variables of the reference video. The latent variables of the target video after progressively adding noise are used as the input of the control branch, and the concatenation result of the latent variables of the target video, reference video, and rendered video is used as the input of the main branch. The text latent variables are used as the conditional input of the main branch and the control branch. The generative model is trained with ∈-x0 as the prediction target. During training, the pre-trained DiT model in the control branch is frozen; ∈ represents Gaussian noise, and x0 represents the target video.

[0080] In this embodiment, two videos of the same scene with different camera trajectories from the dataset are randomly selected. One video is chosen as the target video, and the other as the reference video. The latent variables of the target video, the reference video, and the rendered video are concatenated along the token dimension and used as input to the subsequent model. By concatenating latent variables from different sources, rich contextual information is provided to the model, enabling it to understand the changes in the dynamic scene at various times and the pose information of the target camera. The model uses the denoised target video as its training objective and employs a stream-based noise scheduling strategy. During training, a time step t is randomly selected from 0 to 1000, and the image x at time step t is calculated using the noise addition formula. t =(1-σ t )×x0+σ t ×∈, where x0 is the latent variable of the target video, x t Let ∈ be the latent variable of the target video after adding noise, ∈ be random Gaussian noise, and σ be a predefined noise table. t Let be the noise intensity at time step t, with a value ranging from 0 to 1. The network input consists of time step t and the noisy latent variable x of the video. t The output is x pred This refers to the result of the network prediction. The target of the network prediction is ∈-x0. The result of the network prediction, x... pred The MSE loss is calculated based on the network's predicted target and optimized using the Adam optimizer. During network training, the control branch needs to be frozen, and only the main branch and zero linear layers are trained. The network is trained using 4 H100 sheets with a batch size of 2, for a total of 20,000 training steps.

[0081] Step 7: Model Prediction

[0082] After model training, the input includes a reference video, the target camera trajectory, and relevant text information. The reference video provides point clouds, and a rendered video is generated given the target trajectory. Pure noise variables are used as input to the control branch, and the concatenation of pure noise variables, latent variables from the reference video, and latent variables from the rendered video is used as input to the main branch. Text latent variables are used as conditional inputs to both the main and control branches. The model progressively denoises and generates latent variables for the target video, ultimately decoding to obtain the target video given the target trajectory. The model trained in the previous step can generate videos of the same scene captured under the target camera trajectory, ensuring that the actions of moving objects in the scene are consistent with those in the reference video at the same time.

[0083] In one specific embodiment of the present invention, the specific process is as follows:

[0084] For a reference video containing moving objects, the VGGT (Visual Geometry Grounded Transformer) model is used to process it. VGGT is a 3D vision model based on a large-scale feedforward Transformer, capable of directly predicting key 3D attributes of a scene from single-view, multi-view, or even hundreds of images. It divides the input image into blocks using a DINO model, converting them into image tokens. Then, it employs an alternating attention structure, alternately using frame-level self-attention and global self-attention to fuse single-frame details and cross-frame global information. Finally, it predicts the camera intrinsic and extrinsic parameters for each frame using a camera parameter head and predicts the point cloud map using a dense prediction head. The VGGT model can estimate the depth of each frame of the reference video, align the depths between different frames, and generate a reference point cloud for each frame. Simultaneously, the VGGT model outputs the camera trajectory for each frame. Then, based on the target camera trajectory, a rendered video is generated through point cloud reprojection.

[0085] In the generation of the target video, a 50-step Diffusion denoising process is employed to achieve high-quality video generation. This process first uses random noise x that conforms to a Gaussian distribution. t As the starting point for the diffusion of the current noise frame, in the inverse denoising stage, the current noise frame is input into the control branch, while the current noise frame, reference video, and rendered video are input into the main branch. The model will output predicted noise, which represents the noise generated from x. t Components to be removed from x pred To approximate the original image. Based on this prediction, using the formula...

[0086] x t-1 =x t +x pred +∈

[0087] The denoised image is calculated iteratively until a clear image x0 is restored. If it is in an intermediate step, i.e. t > 0, an appropriate amount of Gaussian noise ∈ is added to maintain the randomness of diffusion. In the final step, a clean image is directly output.

[0088] To verify the effectiveness of the present invention, Figure 2 An experiment was conducted on a certain dataset. The figure shows videos rendered from different perspectives under the same dynamic scene.

[0089] This invention also selected three videos and generated camera movement results for these videos from different perspectives. The camera movement angle selected in this invention is a rightward rotation of the camera. The results are as follows: Figure 3-5 As shown, in each example, the top row is the original video, and the bottom row is the video after re-movement.

[0090] This embodiment also provides a dynamic scene re-motion video generation system based on a diffusion model, including:

[0091] The video generation model module uses a diffusion model with a control branch as the generation model. It consists of an embedding layer, a main branch, and a control branch. The main branch is a pre-trained DiT model, and the control branch contains a pre-trained DiT model and a linear layer with weights initialized to zero. The output of each Transformer block of the pre-trained DiT model in the control branch is processed by the linear layer with weights initialized to zero and then added to the output of the corresponding Transformer block of the pre-trained DiT model in the main branch. The result of the addition is used as the input of the next Transformer block in the main branch.

[0092] The model training module utilizes two videos with different camera trajectories in the same dynamic scene as the target video and reference video, respectively. The reference video provides the point cloud, and the target video provides the camera trajectory, generating a rendered video. It acquires the latent variables of the target video, reference video, and rendered video, as well as the textual latent variables of the reference video. The latent variables of the target video, after progressively adding noise, are used as input to the control branch. The concatenated result of the latent variables of the target video, reference video, and rendered video is used as input to the main branch. The textual latent variables are used as conditional inputs to both the main branch and the control branch. The model is trained with ∈-x0 as the prediction target, and the pre-trained DiT model in the control branch is frozen during training. ∈ represents Gaussian noise, and x0 represents the target video.

[0093] The re-motion video generation module uses a reference video to provide point clouds and generates a rendered video given a target trajectory. Pure noise variables are used as input to the control branch, and the concatenation result of pure noise variables, reference video latent variables, and rendered video latent variables is used as input to the main branch. Text latent variables are used as conditional inputs to the main branch and control branch. The model gradually denoises and generates target video latent variables, and decodes to obtain the target video with the given target trajectory.

[0094] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the remaining modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0095] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0096] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for generating dynamic scene re-motion video based on a diffusion model, characterized in that, Including the following steps: A diffusion model with a control branch is constructed as a generative model, consisting of an embedding layer, a main branch, and a control branch. The main branch is a pre-trained DiT model, and the control branch contains a pre-trained DiT model and a linear layer with weights initialized to zero. The output of each Transformer block of the pre-trained DiT model in the control branch is processed by the linear layer with weights initialized to zero and then added to the output of the corresponding Transformer block of the pre-trained DiT model in the main branch. The result of the addition is used as the input of the next Transformer block in the main branch. During the training phase, two videos with different camera trajectories in the same dynamic scene are used as the target video and the reference video, respectively. The reference video provides the point cloud and the target video provides the camera trajectory to generate the rendered video. Obtain the latent variables of the target video, reference video, and rendered video, as well as the text latent variables of the reference video; The latent variables of the target video after progressively adding noise are used as the input to the control branch. The concatenation result of the latent variables of the target video, reference video, and rendered video is used as the input to the main branch. The latent variables of the text are used as the conditional inputs to the main branch and the control branch. The generative model is trained with ∈-x0 as the prediction target. During training, the pre-trained DiT model in the control branch is frozen; ∈ represents Gaussian noise, and x0 represents the target video. During the generation phase, point clouds are provided by reference videos, and rendered videos are generated given the target trajectory. The pure noise variable is used as the input to the control branch, and the concatenation result of the pure noise variable, the reference video latent variable, and the rendered video latent variable is used as the input to the main branch. The text latent variable is used as the conditional input to the main branch and the control branch. The model gradually denoises to generate the target video latent variable, and decodes it to obtain the target video with the given target trajectory.

2. The dynamic scene re-motion video generation method based on a diffusion model according to claim 1, characterized in that, The video generation process for different camera trajectories in the same dynamic scene includes: Load the scene file, dynamic object model and its corresponding animation file, and give the initial position and orientation of the camera; At the initial position and orientation of the camera, acquire the depth map and normal map of the scene. Use the normal map to filter planar regions where the angle between the normal and the Z-axis is <10°. Obtain the depth values ​​of the pixels in the planar regions from the depth map, and calculate the candidate positions of the dynamic object model in the scene using the ray propagation formula. Placing a dynamic object model at any candidate location and binding it with an animation file is considered as a dynamic scene. Randomly select a pixel in the depth map and take a random depth value that is less than the pixel's depth value. Combine this with the ray propagation formula to calculate the camera position for the next keyframe. Introduce a random deflection angle within ±10 degrees as the camera orientation for the next keyframe, based on the alignment with the dynamic object model. Repeat the process of randomly selecting pixels in the depth map multiple times to generate the camera positions and orientations of several keyframes, forming a camera trajectory. Generate 10-20 camera trajectories for each dynamic scene. In this dynamic scene, render a video for each camera trajectory.

3. The dynamic scene re-motion video generation method based on a diffusion model according to claim 1, characterized in that, The pre-trained DiT models in the main branch and control branch are obtained by initializing the Wanxiang model.

4. The dynamic scene re-motion video generation method based on a diffusion model according to claim 1, characterized in that, The process of generating and rendering videos during the training phase includes: Obtain the reference camera trajectory corresponding to the reference video and extract the point cloud from the reference video; Obtain the target camera trajectory corresponding to the target video; Based on the target camera trajectory, reference camera trajectory, and reference video point cloud, the reference video point cloud is projected onto the target camera trajectory using a coordinate transformation formula to obtain the rendered video. Convert the target video point cloud into a rendered video.

5. The dynamic scene re-motion video generation method based on a diffusion model according to claim 1, characterized in that, The target video, reference video, and rendered video are encoded using WanVAE. The encoded results are then embedded to create feature vectors that match the input dimensions of the pre-trained DiT model, which are used as latent variables for the target video, reference video, and rendered video, respectively. Before encoding, the rendered video needs to be downsampled to reduce its length and width to 1 / 4 of their original values.

6. The dynamic scene re-motion video generation method based on a diffusion model according to claim 5, characterized in that, The method for obtaining the textual latent variables of the reference video is as follows: Semantic annotation is performed on the reference video to obtain descriptive text information related to the video; Text information is input into the T5 model for encoding, and the input text is mapped into a fixed-dimensional feature vector to obtain text latent variables.

7. The dynamic scene re-motion video generation method based on a diffusion model according to claim 1, characterized in that, During the training phase, the latent variable x of the target video at time step t is calculated using the noise addition formula. t =(1-σ t )×x0+σ t ×∈, where σ t Let t be the noise intensity at time step t.

8. The dynamic scene re-motion video generation method based on a diffusion model according to claim 1, characterized in that, The process of generating and rendering a video during the generation phase includes: Input the reference video into the VGGT model to generate reference video point cloud and reference camera trajectory; Given the target camera trajectory; Based on the target camera trajectory, reference camera trajectory, and reference video point cloud, the reference video point cloud is projected onto the target camera trajectory using a coordinate transformation formula to obtain the rendered video.

9. The dynamic scene re-motion video generation method based on a diffusion model according to claim 1, characterized in that, During the training phase, the model outputs predicted noise, which represents the component x that should be removed from the noise variable input at the current time step. pred To approximate the original image, it is represented as: X t-1 =X t +x pred +∈ Among them, X t X represents the noise variable of the input at time step t. t-1 The noise variables after denoising are iterated step by step until the latent variables of the target video are restored, and the target video with a given target trajectory is decoded.

10. A dynamic scene re-motion video generation system based on a diffusion model, characterized in that, include: The video generation model module uses a diffusion model with a control branch as the generation model. It consists of an embedding layer, a main branch, and a control branch. The main branch is a pre-trained DiT model, and the control branch contains a pre-trained DiT model and a linear layer with weights initialized to zero. The output of each Transformer block of the pre-trained DiT model in the control branch is processed by the linear layer with weights initialized to zero and then added to the output of the corresponding Transformer block of the pre-trained DiT model in the main branch. The result of the addition is used as the input of the next Transformer block in the main branch. The model training module is used to generate a rendered video by using two videos with different camera trajectories in the same dynamic scene as the target video and the reference video, respectively. The reference video provides the point cloud and the target video provides the camera trajectory. Obtain the latent variables of the target video, reference video, and rendered video, as well as the text latent variables of the reference video; The latent variables of the target video after progressively adding noise are used as the input to the control branch. The concatenation result of the latent variables of the target video, reference video, and rendered video is used as the input to the main branch. The latent variables of the text are used as the conditional inputs to the main branch and the control branch. The generative model is trained with ∈-x0 as the prediction target. During training, the pre-trained DiT model in the control branch is frozen; ∈ represents Gaussian noise, and x0 represents the target video. The re-motion camera video generation module uses a reference video to provide point clouds and generates a rendered video given the target trajectory. The pure noise variable is used as the input to the control branch, and the concatenation result of the pure noise variable, the reference video latent variable, and the rendered video latent variable is used as the input to the main branch. The text latent variable is used as the conditional input to the main branch and the control branch. The model gradually denoises to generate the target video latent variable, and decodes it to obtain the target video with the given target trajectory.

Images