A video generation model training method and device, and a storage medium

Abstract

The present disclosure relates to the technical field of computer, and particularly relates to a video generation model training method and device and a storage medium, comprising: controlling a video generation model to output a video by using a camera pose of a sample video, and supervising and fine-tuning the video generation model by a loss between the output video and the sample video; generating a group containing multiple generated videos based on the same set of input conditions by using the video generation model which is supervised and fine-tuned; converting each generated video into a three-dimensional Gaussian point cloud, and rendering each three-dimensional Gaussian point cloud into a rendered video; determining a reward score of each generated video based on consistency between each rendered video and a corresponding generated video, and optimizing the video generation model based on the reward score of each generated video in the same group. According to the embodiment of the present disclosure, high-quality 3D scene videos can be generated without relying on a large amount of labeled data.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus and storage medium for training a video generation model. Background Technology

[0002] With the rapid development of computer graphics and machine learning technologies, 3D video generation technology has been widely applied in fields such as virtual reality, game development, and architectural visualization. Existing 3D video generation methods primarily rely on supervised learning. Supervised learning depends on training with large-scale text-video pairs, aiming to enable the model to generate high-quality, high-fidelity video content based on the input text description.

[0003] However, this supervised training method, which relies on two-dimensional images and video frames, has inherent limitations and cannot effectively guarantee the realism of the generated videos in the three-dimensional physical world. Generated videos often appear plausible in two dimensions, but when viewed from a three-dimensional perspective, the shapes of objects may be unreasonably distorted or their trajectories may violate physical laws, resulting in poor video quality. Summary of the Invention

[0004] In view of this, this disclosure proposes a video generation model training technology scheme.

[0005] According to one aspect of this disclosure, a method for training a video generation model is provided, the video generation model being used to generate 3D videos, comprising:

[0006] The video generation model is controlled by the camera pose of the sample video to output the video, and the loss between the output video and the sample video is used to supervise and fine-tune the video generation model.

[0007] Using the supervised fine-tuned video generation model, a group containing multiple generated videos is generated based on the same set of input conditions;

[0008] Each of the generated videos is converted into a 3D Gaussian point cloud, and each of the 3D Gaussian point clouds is rendered into a rendered video.

[0009] Based on the consistency between each rendered video and its corresponding generated video, a reward score is determined for each generated video, and the video generation model is optimized based on the reward scores of each generated video in the same group.

[0010] In one possible implementation, the step of using the camera pose of the sample video to control the video generation model to output the video includes:

[0011] Determine the camera pose of the sample video;

[0012] The camera pose is encoded into a control signal, and the control signal is injected into the denoising process of the video generation model to obtain the output video.

[0013] In one possible implementation, determining the reward score for each generated video based on the consistency between each rendered video and its corresponding generated video includes:

[0014] Calculate the peak signal-to-noise ratio (PSNR) and / or structural similarity (SSIM) between the generated video and the corresponding rendered video to evaluate the geometric consistency and object structural continuity between the generated video and the rendered video;

[0015] The reward score is determined based on the PSNR and / or SSIM.

[0016] In one possible implementation, determining the reward score for each generated video based on the consistency between each rendered video and its corresponding generated video includes:

[0017] Obtain the depth information corresponding to each frame of the rendered video;

[0018] Based on the depth information of the i-th frame in the rendered video, as well as the camera pose of the i-th frame and the camera pose of the (i+1)-th frame, each pixel of the i-th frame in the generated video is projected into the image coordinate system of the (i+1)-th frame through a 3D projection transformation to obtain the projected image.

[0019] Determine the inter-frame similarity between the projected image and the (i+1)th frame in the generated video, and determine the reward score based on the similarity.

[0020] In one possible implementation, determining the reward score for each generated video includes:

[0021] The reward score is determined by weighting the PSNR, SSIM, and / or inter-frame similarity in the reward function.

[0022] In one possible implementation, the method further includes:

[0023] The weights are dynamically determined based on the application scenario of the video generation model.

[0024] In one possible implementation, dynamically determining the weights based on the application scenario of the video generation model includes:

[0025] Based on the text prompts input by the user, the application scenario of the video generation model is determined, and the weights are dynamically determined based on the application scenario.

[0026] In one possible implementation, the method further includes:

[0027] The sample video and its camera pose are input into the reconstruction model LRM to obtain the three-dimensional Gaussian point cloud corresponding to the sample video.

[0028] The three-dimensional Gaussian point cloud is rendered to obtain a reconstructed rendered video;

[0029] Calculate the loss between the reconstructed rendered video and the sample video as the ground truth, and update the parameters of the reconstruction model.

[0030] According to another aspect of this disclosure, a video generation model training apparatus is provided, the video generation model being used to generate three-dimensional videos, comprising:

[0031] The fine-tuning module is used to control the output video of the video generation model by using the camera pose of the sample video, and to perform supervised fine-tuning of the video generation model by comparing the loss between the output video and the sample video.

[0032] The group generation module is used to generate a group containing multiple generated videos based on the same set of input conditions using the video generation model after supervised fine-tuning.

[0033] The video rendering module is used to convert each of the generated videos into three-dimensional Gaussian point clouds, and to render each of the three-dimensional Gaussian point clouds into a video rendering module.

[0034] The model optimization module is used to determine the reward score of each generated video based on the consistency between each rendered video and the corresponding generated video, and to optimize the video generation model based on the reward scores of each generated video in the same group.

[0035] In one possible implementation, the fine-tuning module is used for:

[0036] Determine the camera pose of the sample video;

[0037] The camera pose is encoded into a control signal, and the control signal is injected into the denoising process of the video generation model to obtain the output video.

[0038] In one possible implementation, the model optimization module is used for:

[0039] Calculate the peak signal-to-noise ratio (PSNR) and / or structural similarity (SSIM) between the generated video and the corresponding rendered video to evaluate the geometric consistency and object structural continuity between the generated video and the rendered video;

[0040] The reward score is determined based on the PSNR and / or SSIM.

[0041] In one possible implementation, the model optimization module is used for:

[0042] Obtain the depth information corresponding to each frame of the rendered video;

[0043] Based on the depth information of the i-th frame in the rendered video, as well as the camera pose of the i-th frame and the camera pose of the (i+1)-th frame, each pixel of the i-th frame in the generated video is projected into the image coordinate system of the (i+1)-th frame through a 3D projection transformation to obtain the projected image.

[0044] Determine the inter-frame similarity between the projected image and the (i+1)th frame in the generated video, and determine the reward score based on the similarity.

[0045] In one possible implementation, the model optimization module is used for:

[0046] The reward score is determined by weighting the PSNR, SSIM, and / or inter-frame similarity in the reward function.

[0047] In one possible implementation, the device further includes:

[0048] The weight determination module is used to dynamically determine the weights based on the application scenario of the video generation model.

[0049] In one possible implementation, the weight determination module is used to determine the application scenario of the video generation model based on the text prompts input by the user, and dynamically determine the weights based on the application scenario.

[0050] In one possible implementation, the apparatus further includes a reconstruction model training module for:

[0051] The sample video and its camera pose are input into the reconstruction model LRM to obtain the three-dimensional Gaussian point cloud corresponding to the sample video.

[0052] The three-dimensional Gaussian point cloud is rendered to obtain a reconstructed rendered video;

[0053] Calculate the loss between the reconstructed rendered video and the sample video as the ground truth, and update the parameters of the reconstruction model.

[0054] According to another aspect of this disclosure, a video generation model training apparatus is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0055] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0056] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0057] In this embodiment, the video generation model is controlled to output video by utilizing the camera pose of sample videos. Supervised fine-tuning of the video generation model is performed using the loss between the output video and the sample video, enabling the model to acquire preliminary camera control capabilities. This allows the model to explore various possibilities from a starting point with basic control capabilities. Then, using the supervised fine-tuned video generation model, a group containing multiple generated videos is generated based on the same set of input conditions. The model is optimized based on the reward scores of each generated video in the same group, allowing it to explore multiple possible output results in parallel. This allows the model to freely explore video generation, learn how to generate more realistic videos, and generalize the generated videos to unseen scenes.

[0058] A reward score is calculated by comparing the generated and rendered videos, and this score is used to optimize the model. This ensures that the reward score accurately and objectively reflects the intrinsic quality of the generated video at the 3D geometric level. A high score means that the generated video is very close to its ideal 3D state, and the model is encouraged to optimize in that direction; a low score indicates defects in its 3D structure, and the model is guided to avoid such outputs. In this way, the video generation model can be effectively driven to build strong 3D consistency constraints internally, thus generating high-quality 3D scene videos that remain stable, realistic, and continuous under dynamic viewpoint switching without relying on massive amounts of labeled data.

[0059] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0060] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0061] Figure 1 A flowchart illustrating a video generation model training method according to an embodiment of this disclosure is shown.

[0062] Figure 2A block diagram of a video generation model training apparatus according to an embodiment of the present disclosure is shown.

[0063] Figure 3 This is a block diagram illustrating an apparatus for training a video generation model according to an exemplary embodiment. Detailed Implementation

[0064] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0065] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0066] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0067] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0068] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0069] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0070] 3D video is a type of video content that provides a three-dimensional visual effect. Compared to traditional two-dimensional (2D) video, 3D video offers viewers a richer sense of depth and an immersive experience. For example, 3D video can be binocular stereoscopic video, which simulates the visual effect of human eyes by capturing and displaying images from two perspectives. Each perspective corresponds to the left and right eyes respectively, and the two images are presented to the left and right eyes separately through special display devices (such as 3D glasses or glasses-free 3D displays), thereby creating a sense of depth.

[0071] Figure 1 A flowchart illustrating a video generation model training method according to an embodiment of this disclosure is shown. Figure 1 As shown, the video generation model is used to generate 3D video, and the method includes:

[0072] In step S11, the camera pose of the sample video is used to control the output video of the video generation model, and the video generation model is fine-tuned under supervision by the loss between the output video and the sample video.

[0073] Sample videos are used to train the video generation model and can be captured using real shooting equipment. The camera pose of the sample video consists of parameters used to record the position and orientation of the shooting equipment in three-dimensional space. These parameters typically include rotation matrices and translation vectors, used to precisely describe the camera's viewpoint and position information at the time each frame is captured. For example, when shooting the same static scene from multiple perspectives, each frame corresponds to a set of pose data to characterize the shooting viewpoint of that frame.

[0074] Video generation models are used to generate 3D videos. These models can be AI-based generative models used to synthesize video sequences based on input conditions. One type of video generation model is a diffusion model, which uses a progressive denoising process to transform random noise into coherent image frames that meet conditional constraints, thus assembling a video.

[0075] When supervising the fine-tuning of video generation models, existing labeled data can be used to further adjust the pre-trained model to adapt it to a specific task. In this process, a loss function can be constructed by comparing the model output with real samples, and this function can guide the update of model parameters, thereby improving the model's performance on the target task.

[0076] For example, the difference between the output video and the real video in pixel space or feature space can be calculated to form a loss signal, which drives the model to better learn the mapping relationship between the control signal and the output content.

[0077] In this step, the camera pose information corresponding to the sample video can be used as a control signal to guide the video generation model to produce video output. By encoding the camera pose into a conditional vector that the model can recognize and inputting it into the video generation model, the model is controlled to output video.

[0078] Subsequently, a loss function is constructed by identifying the differences between the generated video and the original sample video. This loss function reflects the degree of deviation between the generated content and the expected target, and then updates the model parameters based on the gradient descent principle, enabling the model to gradually learn to generate video content from the corresponding perspective based on the input pose. For example, pose information can be injected into the video generation model through network structures such as ControlNet to guide the denoising process of the video generation model, and the model can be optimized through mean squared error or perceptual loss to make its output as consistent as possible with the real video.

[0079] This step utilizes a small amount of sample data to enable the video generation model to learn and understand the mapping relationship between the camera pose control signal and the output video content, laying the foundation for the subsequent generalization of the model to unseen videos and effectively reducing the dependence on large-scale, high-quality labeled data.

[0080] In step S12, the video generation model, after being supervised and fine-tuned, is used to generate a group containing multiple generated videos based on the same set of input conditions.

[0081] The supervised fine-tuning video generation model has the ability to generate videos from corresponding viewpoints based on camera pose. Building upon this, by fixing the input conditions and repeatedly calling the model for forward inference, a group consisting of multiple generated videos can be obtained.

[0082] The same set of input conditions here can be text descriptions and camera control signals that remain unchanged during model generation. These conditions together constitute the specific constraints of the generation task, ensuring that the video generation model synthesizes content under the same set of semantic and viewpoint planning. For example, a text prompt describing the scene, combined with a preset camera pose parameter, can constitute a set of generation conditions.

[0083] During the generation process, by changing the sampling noise or random seed, the model can explore different generation paths while following the same control signal, thereby introducing reasonable diversity into the output. A set of generation results is obtained by multiple independent samplings under the same input conditions.

[0084] During the generation process, by introducing different random initial states or noise sequences, the video generation model can produce multiple video outputs with subtle differences in content details, texture representation, or local motion. These outputs together constitute a candidate set for comparison and evaluation.

[0085] For example, a video generation model can generate multiple video versions with slight differences in the lighting and shadows on the building surface or the amplitude of leaf swaying, based on the same text description and camera trajectory, due to differences in initial noise.

[0086] In step S13, each of the generated videos is converted into a three-dimensional Gaussian point cloud, and each of the three-dimensional Gaussian point clouds is rendered into a rendered video.

[0087] 3D Gaussian point clouds are a data format used to represent the structure of 3D scenes. They discretely model continuous 3D space using a large number of Gaussian distributions with specific attributes such as position, color, transparency, scale, and rotation. This representation method can efficiently store and render the geometric and appearance information of complex scenes. For example, an indoor scene can be represented as thousands of tiny, ellipsoidal colored points, each representing a tiny surface area in the scene.

[0088] In this step, the video generated by the video generation model can be converted into a 3D Gaussian point cloud. This conversion process can be based on a pre-trained large-scale reconstruction model that can infer the consistent 3D structure behind multiple video images and output it as a 3D Gaussian point cloud. Specific training methods can be found in the possible implementations provided in this disclosure, and will not be elaborated here.

[0089] Subsequently, using the obtained 3D Gaussian point cloud as input, a renderer is employed to regenerate a new rendered video based on the camera pose sequence corresponding to the original generated video. This rendered video is derived from the inferred 3D geometry.

[0090] In step S14, based on the consistency between each rendered video and the corresponding generated video, the reward score of each generated video is determined, and the video generation model is optimized based on the reward scores of each generated video in the same group.

[0091] 3D consistency refers to the characteristic of a generated video maintaining stable and consistent geometric structure and object form across different viewpoints. This consistency is reflected in the fact that when the viewpoint changes, objects in the scene do not exhibit unreasonable deformation, flickering, or floating phenomena, and their spatial relationships and surface textures remain coherent and realistic. For example, in a video with good 3D consistency, as the camera rotates around an object, the object's outline, size, and surface pattern will show smooth changes that conform to the laws of perspective.

[0092] If the 3D consistency of the generated video output by the video generation model is high, it means that the multi-view information of the generated video is self-consistent. Therefore, when converting it to a 3D Gaussian point cloud representation, the resulting point cloud model will have a clear structure and few noise points, resulting in a very clear video rendered from this high-quality point cloud. Conversely, if the 3D consistency of the generated video is low, there will be geometric conflicts between different viewpoints. This will generate a large number of floating noise points in the converted point cloud model, leading to a significant deterioration in the final rendered video quality.

[0093] Therefore, the 3D consistency quality of each generated video can be evaluated by comparing the consistency between the original generated video and its corresponding rendered video. Specifically, the differences between the two videos at the feature level, such as pixel intensity, structural information, or depth information, can be calculated and quantified into a comprehensive reward score. If the consistency between the generated and rendered videos is low, it indicates that the quality of the generated video is not high; if the consistency is high, it indicates that the quality of the generated video is high.

[0094] The reward score is a quantitative evaluation metric used to measure the performance of a generated video across a specific quality dimension. A higher score indicates better three-dimensional consistency of the generated video. This score is calculated using a predefined reward function and is used to guide the direction of model parameter updates.

[0095] When optimizing video generation models based on reward scores, a group-relative strategy optimization method can be used. This method compares the reward scores obtained by different videos within the same group to calculate the relative advantage of each generated result. This advantage reflects the superiority or inferiority of a specific generation strategy relative to the group average.

[0096] During the optimization process, the parameters of the video generation model are adjusted to make the model more inclined to adopt generation strategies that produce high reward scores. For example, if a video receives a high reward score due to its good 3D consistency, the optimization algorithm will guide the model parameters to adjust in a direction that produces similar output characteristics, thereby gradually improving the overall generation quality of the model.

[0097] Once optimized, the video generation model can be used to generate videos. When generating a video, you can input a text prompt and a camera pose parameter. The video generation model can then generate a video that matches the camera pose parameter based on the text prompt.

[0098] In this embodiment, the camera pose of the sample video is used to control the output video of the video generation model. The loss between the output video and the sample video is used to perform supervised fine-tuning of the video generation model, enabling the video generation model to acquire preliminary camera control capabilities. Then, using the supervised fine-tuned video generation model, a group containing multiple generated videos is generated based on the same set of input conditions. Each generated video is converted into a 3D Gaussian point cloud, and each 3D Gaussian point cloud is rendered as a rendered video. Based on the consistency between each rendered video and the corresponding generated video, the reward score of each generated video is determined. The video generation model is then optimized based on the reward scores of each generated video in the same group. By using the camera pose of sample videos to control the output video of the video generation model, and by using the loss between the output video and the sample video, the video generation model is fine-tuned under supervision. This gives the video generation model preliminary camera control capabilities, allowing it to start from a point with basic control capabilities and explore various possibilities. Then, using the fine-tuned video generation model, based on the same set of input conditions, a group containing multiple generated videos is generated. The model is then optimized based on the reward scores of each generated video in the same group, enabling it to explore multiple possible output results in parallel. This allows the model to freely explore video generation, learn how to generate more realistic videos, and generalize the generated videos to unseen scenes.

[0099] If the 3D consistency of the generated video output by the video generation model is high, it means that the multi-view information of the generated video is self-consistent. Therefore, when converting it to a 3D Gaussian point cloud representation, the resulting point cloud model will have a clear structure and few noise points, resulting in a very clear video rendered from this high-quality point cloud. Conversely, if the 3D consistency of the generated video is low, there will be geometric conflicts between different viewpoints. This will generate a large number of floating noise points in the converted point cloud model, leading to a significant deterioration in the final rendered video quality.

[0100] Therefore, a reward score is obtained by calculating the difference between the generated video and the rendered video, and the model is optimized accordingly. This ensures that the reward score accurately and objectively reflects the intrinsic quality of the generated video at the 3D geometric level. A high score means that the generated video is very close to its ideal 3D state, and the model is encouraged to optimize in this direction; a low score indicates that its 3D structure has defects, and the model is guided to avoid such outputs. In this way, the video generation model can be effectively driven to build strong 3D consistency constraints internally, thus generating high-quality 3D scene videos that remain stable, realistic, and continuous under dynamic viewpoint switching without relying on massive amounts of labeled data.

[0101] In one possible implementation, controlling the video generation model to output video using the camera pose of the sample video includes: determining the camera pose of the sample video; encoding the camera pose into a control signal; and injecting the control signal into the denoising process of the video generation model to obtain the output video.

[0102] In this implementation, when determining the camera pose of the sample video, camera parameter information corresponding to each frame can be extracted from the raw video data. These parameters are typically obtained directly through computer vision algorithms or device sensor data and are used to accurately describe the viewpoint, position, and orientation relationships during shooting. For example, the Structure from Motion (SFM) method can be used to calculate the camera's rotation matrix and translation vector from multiple frames of the video sequence.

[0103] Control signals refer to encoded camera pose data that can be recognized and processed by neural networks. These signals convert spatial geometric information into vector representations in the model's latent space, thus serving as conditional inputs to the generation process.

[0104] Therefore, the camera pose parameters of each frame can be parsed from the sample video. These parameters are then converted into continuous vector representations through an encoding network (such as ControlNet) to form a conditional control signal. Subsequently, when the video generation model performs denoising operations, this control signal is fused with the noise features of the current time step, influencing the model's prediction behavior through a conditional modulation mechanism.

[0105] For example, in the latent diffusion model, the pose encoding vector and the temporal step encoding can be input into the attention layer together, so that the denoising process takes into account both the temporal dimension and spatial constraints, and finally outputs a video sequence that is consistent with the specified camera trajectory.

[0106] In this embodiment, the camera pose of the sample video is determined; the camera pose is encoded into a control signal, and the control signal is injected into the denoising process of the video generation model to obtain the output video. Thus, the video generation model can learn the correlation between the camera pose and the video content, achieving the goal of controlling the generated content through geometric parameters. This conditional injection-based control method enables the generated video to have accurate viewpoint correspondence, laying the foundation for subsequent 3D consistency optimization. Simultaneously, this implementation maintains the original generation flexibility of the diffusion model, allowing for continuous generation from different viewpoints by adjusting the control signal.

[0107] It should be noted that this process can be achieved with a small number of video samples. In related technologies, when performing supervised fine-tuning of video generation models, it is necessary to collect a large number of sample videos in the real world, which requires a lot of manpower and resources. That is, to collect videos through real cameras. If the video generation model needs to be trained to have good generalization ability, tens of thousands or even tens of millions of videos are required.

[0108] In this embodiment of the disclosure, during supervised fine-tuning, only a small number of sample videos are needed for the model to learn to control video generation using camera poses, which can be as few hundred or a few thousand videos. Subsequently, reinforcement learning can be used to allow the video generation model to freely explore video generation, enabling the model to learn more information and generalize the generated videos to unseen scenes.

[0109] In one possible implementation, determining the reward score for each generated video based on the consistency between each rendered video and its corresponding generated video includes: calculating the peak signal-to-noise ratio (PSNR) and / or structural similarity (SSIM) between the generated video and its corresponding rendered video to evaluate the geometric consistency and object structural continuity between the generated video and the rendered video; and determining the reward score based on the PSNR and / or SSIM.

[0110] When evaluating the consistency between the rendered video and the generated video, it can be based on peak signal-to-noise ratio (PSNR) and / or structural similarity (SSIM).

[0111] Peak signal-to-noise ratio (PSNR) is a metric for evaluating the quality of video reconstruction. It assesses fidelity by calculating the mean square error between the original and reconstructed signals. This metric is measured in decibels (dB); a higher value indicates less distortion and reflects the degree of accurate matching at the pixel level. For example, in video quality assessment, PSNR can be used to quantify the difference in geometric information between the generated and reference videos.

[0112] Structural similarity (SSIM) is a quality assessment metric based on the characteristics of the human visual system. It comprehensively compares the similarity between two images across three dimensions: brightness, contrast, and structure. Unlike PSNR, SSIM can capture the preservation of structural information in an image. Therefore, SSIM can detect the integrity of object edges and the degree of texture detail reproduction in generated videos.

[0113] Therefore, when evaluating the consistency between the rendered video and the generated video, the generated video can first be frame-aligned with the corresponding rendered video to ensure consistency in timestamps and spatial locations. Then, the PSNR and / or SSIM metrics are calculated for each frame. When calculating PSNR, geometric consistency is evaluated by calculating the mean square error between pixel values ​​of two frames; when calculating SSIM, the continuity of object structure is evaluated by comparing the structural features of local image patches.

[0114] The metrics calculated frame by frame can be aggregated into an overall score for the entire video sequence using temporal pooling. For example, the average or minimum value of all frame metric values ​​can be taken as the final consistency measure.

[0115] When determining reward scores based on these metrics, a weighted combination approach can be used to map PSNR and / or SSIM values ​​to other relevant metric values ​​to a unified reward value. Since PSNR and SSIM reflect video quality from different perspectives, their combination can more comprehensively evaluate three-dimensional consistency. For example, a positive reward can be given when both PSNR and SSIM reach high values, while the reward score can be reduced accordingly when either metric significantly decreases.

[0116] In this implementation, the geometric consistency and structural continuity of the generated and rendered videos are evaluated by calculating the Peak Signal-to-Noise Ratio (PSNR) and / or Structural Similarity Index (SSIM) between the generated and rendered videos. A reward score is then determined based on the PSNR and / or SSIM. This provides reinforcement learning with a continuous and fine-grained reward signal, guiding the model to optimize 3D consistency. The PSNR metric ensures that the generated video closely approximates the ideal 3D reconstruction result at the geometric pixel level, while the SSIM metric guarantees that the structural integrity of the visual content is maintained. This allows for a more comprehensive evaluation of the generated video quality, enabling the model to optimize geometric accuracy and visual naturalness, ultimately producing video content with high 3D consistency.

[0117] In one possible implementation, determining the reward score for each generated video based on the consistency between each rendered video and the corresponding generated video includes: obtaining depth information corresponding to each frame of the rendered video; based on the depth information of the i-th frame in the rendered video, and the camera pose of the i-th frame and the (i+1)-th frame, projecting each pixel of the i-th frame in the generated video onto the image coordinate system of the (i+1)-th frame through a 3D projection transformation to obtain a projected image; determining the inter-frame similarity between the projected image and the (i+1)-th frame in the generated video, and determining the reward score based on the similarity.

[0118] When evaluating the consistency between the rendered video and the generated video, it can also be evaluated based on the inter-frame depth consistency loss.

[0119] Depth information is the distance data between each pixel in an image and the optical center of the camera, usually represented as a grayscale image or a distance matrix. This information reflects the three-dimensional spatial structure and relative positional relationships of objects in a scene, providing spatial geometric constraints for geometric consistency assessment. For example, approximate depth information can be inferred from a single frame image using a monocular depth estimation network, or more accurate depth information can be obtained using multi-view stereo matching algorithms.

[0120] It can acquire the rendered video sequence obtained from 3D Gaussian point cloud rendering, as well as the depth information corresponding to each frame in the rendered video. Simultaneously, it can acquire the generated video sequence to be evaluated. For two consecutive frames, given the depth information of frame i, the camera pose of frame i, and the camera pose of frame i+1, a 3D projection transformation can be used to project each pixel of frame i in the generated video onto the image coordinate system of the next frame. During this projection process, the pixels of the current frame are back-projected into 3D space using the depth information. Then, based on the camera poses of frame i and frame i+1, rotation and translation operations are performed on the pixels in 3D space. Finally, the pixels after rotation and translation are projected onto the image plane of the next frame.

[0121] After projection, the projected image is compared with the (i+1)th frame image in the generated video to determine their similarity. Various similarity metrics can be used, such as calculating the Structural Similarity Index (SSIM) or Normalized Cross-Correlation (NCC) between the two images. This comparison assesses whether the 3D structure of objects in the scene remains stable and consistent as the camera's viewpoint changes. For example, when the camera translates, static objects in the scene should exhibit corresponding parallax changes according to their depth information. If the changes in the actual generated video are inconsistent with the expected changes calculated based on depth and pose, it indicates a 3D inconsistency problem in the generated video.

[0122] The lower the similarity, the worse the three-dimensional consistency of the generated video between consecutive frames; the higher the similarity, the better the three-dimensional consistency is maintained. Therefore, a reward score can be determined based on this similarity to guide the optimization process of the video generation model.

[0123] In this embodiment, depth information corresponding to each frame of the rendered video is obtained; based on the depth information of the i-th frame in the rendered video, and the camera pose of the i-th frame and the (i+1)-th frame, each pixel of the i-th frame in the generated video is projected onto the image coordinate system of the (i+1)-th frame through a 3D projection transformation to obtain a projected image; the similarity between the projected image and the (i+1)-th frame in the generated video is determined, and a reward score is determined based on the similarity. This effectively identifies problems such as shape distortion, position jumps, or unreasonable motion trajectories that may occur when an object changes its viewpoint. By incorporating this similarity into the reward mechanism, the video generation model can be guided to learn to maintain the 3D structural stability of objects in the scene, improving the spatial coherence and visual realism of the generated video.

[0124] In one possible implementation, determining the reward score for each generated video includes: weighting the PSNR, SSIM, and / or inter-frame similarity in the reward function by weights to determine the reward score.

[0125] In this implementation, the reward score is determined using a multi-indicator weighted fusion strategy. By combining multiple complementary evaluation dimensions, a comprehensive evaluation system for the quality of the generated video is formed. Each indicator reflects the quality characteristics of the video from a different perspective, and a more comprehensive and balanced quality assessment can be achieved through reasonable weight allocation.

[0126] Peak Signal-to-Noise Ratio (PSNR) primarily reflects the pixel-level similarity between the generated and rendered videos, capturing noticeable color and brightness distortions. Structural Similarity (SSIM), on the other hand, focuses more on human visual perception, assessing the consistency in structural information preservation between two videos. Inter-frame similarity focuses on the consistency of depth information between consecutive frames, detecting the temporal stability of 3D geometric structures. These metrics each have different dimensions and numerical ranges, therefore, they typically require normalization before weighting to ensure they fall within a uniform numerical range.

[0127] It should be noted that, in addition to the indicators mentioned in this disclosure, the reward function may include other indicators, and this disclosure does not limit this.

[0128] During the weighting process, the allocation of weights can be adjusted according to the needs of specific application scenarios, as detailed in the possible implementation methods provided in this disclosure. Weights can be determined through empirical settings or performance tuning based on the validation set. The final reward score can be obtained by weighted summation of all indicators, expressed by the following formula:

[0129] Reward score = w1 × PSNR + w2 × SSIM + w3 × Inter-frame similarity + w4 × Other metrics

[0130] Where w1, w2, w3, and w4 are the weight coefficients corresponding to each indicator.

[0131] In this embodiment, a more comprehensive and stable reward signal can be generated through multi-index weighted fusion, providing clear and consistent guidance for model optimization. The complementary characteristics between different indicators help reduce evaluation bias that may be caused by a single indicator, enabling the reward score to more accurately reflect the overall quality of the generated video.

[0132] In one possible implementation, the method further includes: dynamically determining the weights based on the application scenario of the video generation model.

[0133] Application scenarios refer to the specific application areas and usage environments served by the video generation model. These scenarios may have different emphases on the quality characteristics of the generated content. Different application scenarios often focus on different quality dimensions. For example, virtual reality applications may place more emphasis on visual immersion and 3D geometric accuracy, while social media content generation may focus more on texture realism and real-time rendering efficiency. These differentiated needs directly affect the allocation of importance of various quality indicators in the final evaluation.

[0134] The process of dynamically determining weights can be implemented based on a configurable weight management mechanism. This mechanism first identifies the current application scenario type (which can be set by the user), and then assigns appropriate weight coefficients to indicators such as PSNR, SSIM, and inter-frame similarity according to the preset scenario-weight mapping relationship.

[0135] Weighting can be pre-defined based on domain knowledge or automatically learned from scene features using machine learning methods. For example, in architectural visualization scenarios, geometric consistency-related metrics might be given higher weights, while in film and television special effects scenarios, metrics related to texture realism and visual aesthetics might be more emphasized.

[0136] In practice, the application scenario can be automatically identified based on the text prompts entered by the user. After identification, the corresponding weight configuration file is called, and these weights are applied to the calculation of the reward function. This dynamic weight adjustment can be performed in real time during model inference, allowing the same model to be trained with different weights to adapt to the quality requirements of different scenarios.

[0137] The dynamic weighting mechanism works on the premise that different application scenarios have varying requirements for video quality. By analyzing the quality preferences of specific scenarios, the system can automatically adjust the evaluation focus, making the reward function better suited to actual application needs. The weight adjustment process can be viewed as an online parameter optimization, maintaining the consistency of the evaluation framework while providing flexibility to adapt to different scenarios.

[0138] In one possible implementation, dynamically determining the weights based on the application scenario of the video generation model includes: determining the application scenario of the video generation model based on text prompts input by the user, and dynamically determining the weights based on the application scenario.

[0139] In this implementation, the application scenario type of the video generation task is automatically identified by analyzing user-provided text prompts. The semantic information contained in the text prompts reflects the intended use and quality requirements of the generated content. For example, keywords such as "architectural walkthrough," "product demonstration," or "artistic creation" appearing in the prompts can indicate different application scenario categories. These scenario categories correspond to different quality assessment focuses, thus affecting the weight allocation of each indicator in the reward function.

[0140] Based on the identified application scenarios, the corresponding weights can be determined according to the mapping relationship between scenario type and weight value. For example, when the text prompt contains keywords such as "virtual reality" or "VR", the system may judge it as an immersive application scenario, and in this case, the weights of inter-frame depth consistency and geometric accuracy will be increased; while when the prompt contains expressions such as "artistic effect" or "stylized", the weight of geometric indicators can be appropriately reduced, and the weight ratio of texture quality and visual effect can be increased.

[0141] In determining application scenarios, rule-based matching or semantic similarity-based calculation methods can be used. Rule-based matching identifies the corresponding application scenario category through keyword recognition, while semantic similarity-based methods determine the corresponding scenario category by embedding prompt words into a vector space and then calculating the similarity score with each application scenario category.

[0142] In this embodiment of the disclosure, by dynamically adjusting weights based on text prompts, the video generation system can intelligently adapt to the quality requirements of different application scenarios. By understanding the user's linguistic intent, the weight allocation of evaluation indicators can be automatically optimized, enabling the generated video content to better meet the professional needs of specific scenarios while maintaining basic quality.

[0143] In one possible implementation, the method further includes: inputting a sample video and its camera pose into a reconstruction model LRM to obtain a 3D Gaussian point cloud corresponding to the sample video; rendering the 3D Gaussian point cloud to obtain a reconstructed rendered video; calculating the loss between the reconstructed rendered video and the sample video as the ground truth, and updating the parameters of the reconstruction model.

[0144] The reconstruction model can convert generated videos into 3D Gaussian point clouds and render the 3D Gaussian point clouds into rendered videos. The reconstruction model can be obtained through training. By training the reconstruction model, the model can acquire the above-mentioned conversion capabilities.

[0145] Specifically, the dataset used to train the reconstruction model contains multiple sample videos and the camera pose parameters corresponding to each video frame. By inputting the sample videos and their corresponding camera poses into the reconstruction model, the model learns the geometric relationships and visual consistency between video frames and outputs a corresponding 3D Gaussian point cloud representation, which continuously describes the geometric structure and appearance attributes of the scene using a discrete set of Gaussian distributions.

[0146] After obtaining the 3D Gaussian point cloud, it is re-rendered into a video using a differentiable renderer. The rendering process is based on the camera pose sequence of the original sample video, ensuring that the rendered video completely corresponds to the original sample video in terms of viewpoint and temporal sequence. Then, the difference loss between the rendered video and the original sample video is calculated.

[0147] Based on computational loss, the parameters of the reconstruction model are updated through backpropagation algorithm to optimize its ability to reconstruct 3D scenes from multi-view videos.

[0148] In this embodiment, by training the reconstruction model, it gains the ability to convert any video into a precise 3D Gaussian point cloud. This provides an important technical foundation for the subsequent evaluation of the video generation model. The generated video output by the video generation model can be converted into a 3D Gaussian point cloud and re-rendered by the reconstruction model to obtain a rendered video used to calculate the reward score. This makes the evaluation process of the generated video not only based on 2D visual similarity but also on 3D geometric consistency, thereby more accurately guiding the video generation model to optimize its 3D performance capabilities.

[0149] The following describes a specific implementation method provided in this disclosure, which mainly includes the following stages:

[0150] Dataset Construction Phase. Using open-source datasets or specially collected static scene videos, tools such as Colmap are employed to calculate the camera's intrinsic and extrinsic parameters for each frame, providing necessary multi-view data support for subsequent training. These datasets, containing precise pose annotations, will simultaneously serve the training of large-scale reconstruction models and video generation models.

[0151] The training phase for a large-scale reconstruction model then begins. Video sequences and their camera poses from the scene dataset are input into the LRM, which learns to convert video frames into 3D Gaussian point cloud representations. The point clouds are then re-rendered into video using a differentiable renderer, and the model parameters are updated with the perceptual loss between the rendered video and the original sample video as the optimization objective, enabling the LRM to achieve high-quality 3D reconstruction from video.

[0152] Next, supervised fine-tuning of the video generation model is performed. Using video and camera pose data from the scene dataset, pose information is encoded into control signals via the ControlNet architecture and injected into the denoising process of the diffusion model. By calculating the difference loss between the generated video and the real video, the model initially acquires the ability to generate videos with corresponding viewpoints based on camera poses.

[0153] Finally, a group-relative strategy is employed for optimization, and the video generation model is fine-tuned using reinforcement learning. Based on the same set of input conditions, the video generation model generates groups containing multiple generated videos through multiple samplings. Each generated video is then converted into a 3D Gaussian point cloud and rendered as a rendered video. Based on the consistency difference between the generated and rendered videos, a reward score is calculated, and the model parameters are updated using the GRPO algorithm, prioritizing the optimization of the 3D consistency and geometric continuity of the generated videos. The entire process adapts to the needs of different application scenarios through a dynamic weighting mechanism, ultimately achieving high-quality, highly consistent 3D static scene video generation.

[0154] Figure 2 A block diagram of a video generation model training apparatus according to an embodiment of this disclosure is shown. Figure 2 As shown, the video generation model is used to generate three-dimensional video, and the device 20 includes:

[0155] The fine-tuning module 21 is used to control the output video of the video generation model by using the camera pose of the sample video, and to perform supervised fine-tuning of the video generation model by comparing the loss between the output video and the sample video.

[0156] Group generation module 22 is used to generate a group containing multiple generated videos based on the same set of input conditions using the video generation model after supervised fine-tuning.

[0157] The video rendering module 23 is used to convert each of the generated videos into three-dimensional Gaussian point clouds, and to render each of the three-dimensional Gaussian point clouds into a video rendering module.

[0158] The model optimization module 24 is used to determine the reward score of each generated video based on the consistency between each rendered video and the corresponding generated video, and to optimize the video generation model based on the reward scores of each generated video in the same group.

[0159] In one possible implementation, the fine-tuning module is used for:

[0160] Determine the camera pose of the sample video;

[0161] The camera pose is encoded into a control signal, and the control signal is injected into the denoising process of the video generation model to obtain the output video.

[0162] In one possible implementation, the model optimization module is used for:

[0163] Calculate the peak signal-to-noise ratio (PSNR) and / or structural similarity (SSIM) between the generated video and the corresponding rendered video to evaluate the geometric consistency and object structural continuity between the generated video and the rendered video;

[0164] The reward score is determined based on the PSNR and / or SSIM.

[0165] In one possible implementation, the model optimization module is used for:

[0166] Obtain the depth information corresponding to each frame of the rendered video;

[0167] Based on the depth information of the i-th frame in the rendered video, as well as the camera pose of the i-th frame and the camera pose of the (i+1)-th frame, each pixel of the i-th frame in the generated video is projected into the image coordinate system of the (i+1)-th frame through a 3D projection transformation to obtain the projected image.

[0168] Determine the inter-frame similarity between the projected image and the (i+1)th frame in the generated video, and determine the reward score based on the similarity.

[0169] In one possible implementation, the model optimization module is used for:

[0170] The reward score is determined by weighting the PSNR, SSIM, and / or inter-frame similarity in the reward function.

[0171] In one possible implementation, the device further includes:

[0172] The weight determination module is used to dynamically determine the weights based on the application scenario of the video generation model.

[0173] In one possible implementation, the weight determination module is used to determine the application scenario of the video generation model based on the text prompts input by the user, and dynamically determine the weights based on the application scenario.

[0174] In one possible implementation, the apparatus further includes a reconstruction model training module for:

[0175] The sample video and its camera pose are input into the reconstruction model LRM to obtain the three-dimensional Gaussian point cloud corresponding to the sample video.

[0176] The three-dimensional Gaussian point cloud is rendered to obtain a reconstructed rendered video;

[0177] Calculate the loss between the reconstructed rendered video and the sample video as the ground truth, and update the parameters of the reconstruction model.

[0178] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0179] This disclosure also provides a video generation model training apparatus, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0180] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0181] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0182] Figure 3 This is a block diagram illustrating an apparatus for training a video generation model according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or terminal device. (Refer to...) Figure 3 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0183] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TMmacOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0184] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0185] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0186] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0187] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0188] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0189] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0190] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0191] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0192] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a video generation model, characterized in that, The video generation model is used to generate 3D video, including: The video generation model is controlled by the camera pose of the sample video to output the video, and the loss between the output video and the sample video is used to supervise and fine-tune the video generation model. Using the supervised fine-tuned video generation model, a group containing multiple generated videos is generated based on the same set of input conditions; Each of the generated videos is converted into a 3D Gaussian point cloud, and each of the 3D Gaussian point clouds is rendered into a rendered video. Based on the consistency between each rendered video and its corresponding generated video, the reward score of each generated video is determined, and the video generation model is optimized based on the reward scores of each generated video in the same group. The video generation model that uses the camera pose of sample videos to control the output of video includes: Determine the camera pose of the sample video; The camera pose is encoded into a control signal, and the control signal is injected into the denoising process of the video generation model to obtain the output video.

2. The method according to claim 1, characterized in that, The determination of the reward score for each generated video based on the consistency between each rendered video and its corresponding generated video includes: Calculate the peak signal-to-noise ratio (PSNR) and / or structural similarity (SSIM) between the generated video and the corresponding rendered video to evaluate the geometric consistency and object structural continuity between the generated video and the rendered video; The reward score is determined based on the PSNR and / or SSIM.

3. The method according to claim 1, characterized in that, The determination of the reward score for each generated video based on the consistency between each rendered video and its corresponding generated video includes: Obtain the depth information corresponding to each frame of the rendered video; Based on the depth information of the i-th frame in the rendered video, as well as the camera pose of the i-th frame and the camera pose of the (i+1)-th frame, each pixel of the i-th frame in the generated video is projected into the image coordinate system of the (i+1)-th frame through a 3D projection transformation to obtain the projected image. Determine the inter-frame similarity between the projected image and the (i+1)th frame in the generated video, and determine the reward score based on the similarity.

4. The method according to claim 1, characterized in that, The determination of the reward score for each generated video includes: The reward score is determined by weighting the PSNR, SSIM, and / or inter-frame similarity in the reward function.

5. The method according to claim 4, characterized in that, The method further includes: The weights are dynamically determined based on the application scenario of the video generation model.

6. The method according to claim 5, characterized in that, The application scenario based on the video generation model dynamically determines the weights, including: Based on the text prompts input by the user, the application scenario of the video generation model is determined, and the weights are dynamically determined based on the application scenario.

7. The method according to claim 1, characterized in that, The method further includes: The sample video and its camera pose are input into the reconstruction model LRM to obtain the three-dimensional Gaussian point cloud corresponding to the sample video. The three-dimensional Gaussian point cloud is rendered to obtain a reconstructed rendered video; Calculate the loss between the reconstructed rendered video and the sample video as the ground truth, and update the parameters of the reconstruction model.

8. A video generation model training device, characterized in that, The video generation model is used to generate 3D video, including: The fine-tuning module is used to control the output video of the video generation model by using the camera pose of the sample video, and to perform supervised fine-tuning of the video generation model by comparing the loss between the output video and the sample video. The group generation module is used to generate a group containing multiple generated videos based on the same set of input conditions using the video generation model after supervised fine-tuning. The video rendering module is used to convert each of the generated videos into three-dimensional Gaussian point clouds, and to render each of the three-dimensional Gaussian point clouds into a video rendering module. The model optimization module is used to determine the reward score of each generated video based on the consistency between each rendered video and the corresponding generated video, and to optimize the video generation model based on the reward scores of each generated video in the same group. The fine-tuning module is used for: Determine the camera pose of the sample video; The camera pose is encoded into a control signal, and the control signal is injected into the denoising process of the video generation model to obtain the output video.

9. A video generation model training device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images