Video generation model training method and device and storage medium

Abstract

The invention relates to the technical field of computers, in particular to a video generation model training method and device and a storage medium, and the method comprises the steps: controlling a video generation model to output a video through the camera pose of a sample video, and carrying out the supervision and fine tuning of the video generation model through the loss of the output video and the sample video; generating a group comprising a plurality of generated videos based on the same group of input conditions by using the video generation model after supervision and fine tuning; the generated videos are converted into three-dimensional Gaussian point clouds, and the three-dimensional Gaussian point clouds are rendered into rendered videos; and determining a reward score of each generated video based on the consistency between each rendered video and the corresponding generated video, and optimizing a video generation model based on the reward score of each generated video in the same group. According to the embodiment of the invention, a high-quality 3D scene video can be generated without depending on massive annotation data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer technology, and particularly relates to a video generation model training method and device and storage medium. BACKGROUND

[0002] With the rapid development of computer graphics and machine learning technology, three-dimensional (3D) video generation technology has been widely applied in virtual reality, game development, building visualization and other fields. Existing 3D video generation methods mainly rely on supervised learning. Supervised learning relies on large-scale text-video pairs for training, so that the model can generate high-quality and high-fidelity video content according to the input text description.

[0003] However, this supervised training method relying on two-dimensional images and video frames has inherent limitations, and it is difficult to effectively ensure the authenticity of the generated video in the three-dimensional physical world. The generated video may seem reasonable at the two-dimensional level, but when viewed from a three-dimensional perspective, the object shape may be distorted unreasonably or the motion trajectory may violate the physical law, and the quality of the generated video is poor. SUMMARY

[0004] Therefore, the present disclosure proposes a video generation model training technical solution.

[0005] According to an aspect of the present disclosure, a video generation model training method is provided, the video generation model being used to generate a three-dimensional video, comprising:

[0006] controlling the video generation model to output a video by using the camera pose of a sample video, and supervising and fine-tuning the video generation model through the loss between the output video and the sample video;

[0007] generating a group containing multiple generated videos based on the same set of input conditions by using the video generation model that has been supervised and fine-tuned;

[0008] converting each of the generated videos into a three-dimensional Gaussian point cloud, and rendering each of the three-dimensional Gaussian point clouds into a rendered video;

[0009] determining a reward score of each generated video based on the consistency between each rendered video and the corresponding generated video, and optimizing the video generation model based on the reward scores of each generated video in the same group.

[0010] In a possible implementation manner, the controlling the video generation model to output a video by using the camera pose of a sample video comprises:

[0011] determining the camera pose of the sample video;

[0012] Encode the camera pose as a control signal, and inject the control signal into a denoising process of the video generation model to obtain an output video.

[0013] In a possible implementation, the determining of the reward score of each generated video based on the consistency between each rendered video and the corresponding generated video comprises:

[0014] calculating a peak signal-to-noise ratio (PSNR) and / or a structural similarity (SSIM) between the generated video and the corresponding rendered video to evaluate the geometric consistency and object structure continuity of the generated video and the rendered video;

[0015] determining the reward score based on the PSNR and / or the SSIM.

[0016] In a possible implementation, the determining of the reward score of each generated video based on the consistency between each rendered video and the corresponding generated video comprises:

[0017] obtaining depth information corresponding to each frame of image in the rendered video;

[0018] based on the depth information of the i th frame in the rendered video, and the camera pose of the i th frame and the camera pose of the i+1 th frame, projecting each pixel point of the i th frame in the generated video into the image coordinate system of the i+1 th frame through three-dimensional projection transformation to obtain a projection image;

[0019] determining an inter-frame similarity between the projection image and the i+1 th frame in the generated video, and determining the reward score based on the similarity.

[0020] In a possible implementation, the determining of the reward score of each generated video comprises:

[0021] weighting the PSNR, the SSIM, and / or the inter-frame similarity in the reward function by a weight to determine the reward score.

[0022] In a possible implementation, the method further comprises:

[0023] dynamically determining the weight based on an application scenario of the video generation model.

[0024] In a possible implementation, the dynamically determining of the weight based on the application scenario of the video generation model comprises:

[0025] determining the application scenario of the video generation model based on a text prompt word input by a user, and dynamically determining the weight based on the application scenario.

[0026] In a possible implementation, the method further comprises:

[0027] inputting the sample video and the camera pose thereof into a reconstruction model LRM to obtain a three-dimensional Gaussian point cloud corresponding to the sample video;

[0028] rendering the three-dimensional Gaussian point cloud to obtain a reconstructed rendered video;

[0029] calculating a loss between the reconstructed rendered video and the sample video as a ground truth, and updating parameters of the reconstruction model.

[0030] According to another aspect of the present disclosure, a video generation model training apparatus is provided, which is configured to generate a three-dimensional video, comprising:

[0031] a fine-tuning module configured to control a video generation model to output a video by using a camera pose of a sample video, and to supervise and fine-tune the video generation model by a loss between the output video and the sample video;

[0032] a group generation module configured to generate a group comprising a plurality of generated videos based on a same set of input conditions by using the video generation model after being supervised and fine-tuned;

[0033] a rendered video generation module configured to convert each of the generated videos into a three-dimensional Gaussian point cloud, and to render each of the three-dimensional Gaussian point clouds into a rendered video;

[0034] a model optimization module configured to determine a reward score of each of the generated videos based on a consistency between each of the rendered videos and the corresponding generated video, and to optimize the video generation model based on the reward scores of each of the generated videos in the same group.

[0035] In a possible implementation, the fine-tuning module is configured to:

[0036] determine a camera pose of a sample video;

[0037] encode the camera pose into a control signal, and inject the control signal into a denoising process of the video generation model to obtain an output video.

[0038] In a possible implementation, the model optimization module is configured to:

[0039] calculate a peak signal-to-noise ratio PSNR and / or a structural similarity SSIM between the generated video and the corresponding rendered video to evaluate geometric consistency and object structure continuity of the generated video and the rendered video;

[0040] determine a reward score based on the PSNR and / or the SSIM.

[0041] In a possible implementation, the model optimization module is configured to:

[0042] obtain depth information corresponding to each frame of image included in the rendered video;

[0043] project each pixel point of the i-th frame of the generated video into the image coordinate system of the (i+1)-th frame by three-dimensional projection transformation based on the depth information of the i-th frame of the rendered video, and the camera pose of the i-th frame and the camera pose of the (i+1)-th frame, to obtain a projection image;

[0044] determine the inter-frame similarity between the projection image and the (i+1)-th frame of the generated video, and determine a reward score based on the similarity.

[0045] In a possible implementation, the model optimization module is configured to:

[0046] determine the reward score by weighting the PSNR, the SSIM and / or the inter-frame similarity in the reward function by the weight.

[0047] In a possible implementation, the apparatus further includes:

[0048] a weight determination module configured to dynamically determine the weight based on an application scenario of the video generation model.

[0049] In a possible implementation, the weight determination module is configured to determine the application scenario of the video generation model based on a text prompt word input by a user, and dynamically determine the weight based on the application scenario.

[0050] In a possible implementation, the apparatus further includes a reconstruction model training module configured to:

[0051] input a sample video and a camera pose thereof into a reconstruction model LRM to obtain a three-dimensional Gaussian point cloud corresponding to the sample video;

[0052] render the three-dimensional Gaussian point cloud to obtain a reconstructed rendered video;

[0053] calculate a loss between the reconstructed rendered video and the sample video as a ground truth, and update a parameter of the reconstruction model.

[0054] According to another aspect of the present disclosure, a video generation model training apparatus is provided, including a memory, a processor and a computer program stored in the memory, the processor executes the computer program to implement the steps of the above method.

[0055] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the above method.

[0056] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, or a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above method.

[0057] In the embodiments of the present disclosure, the video generation model is controlled by using the camera pose of the sample video to output a video, and the video generation model is supervised and fine-tuned by the loss between the output video and the sample video, so that the video generation model has a preliminary camera control capability and can start from a starting point with a basic control capability for subsequent diversified exploration; then the video generation model supervised and fine-tuned is used to generate a group containing multiple generated videos based on the same set of input conditions, and the video generation model is optimized based on the reward scores of the generated videos in the same group, so that the model can explore multiple possible output results in parallel, and let the model freely explore video generation and learn how to generate more realistic videos, and the generated videos can be generalized to unseen scenes.

[0058] The reward score is obtained by calculating the difference between the generated video and the rendered video, and the model is optimized accordingly, so that the reward score can accurately and objectively reflect the internal quality of the generated video in the 3D geometry level. A high score means that the generated video itself is very close to its ideal 3D state, and the model will be encouraged to optimize in this direction; a low score indicates that the 3D structure has defects, and the model will be guided to avoid such output. In this way, the video generation model can effectively drive the construction of strong 3D consistency constraints in its internal structure, so that it can generate high-quality 3D scene videos that remain stable, realistic and continuous under dynamic view switching without relying on massive labeled data.

[0059] Other features and aspects of the present disclosure will become apparent from the following detailed description of example embodiments, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate example embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0061] Figure 1 A flowchart of a video generation model training method according to an embodiment of the present disclosure is shown.

[0062] Figure 2A block diagram of a video generation model training apparatus according to an exemplary embodiment is shown.

[0063] Figure 3 A block diagram of a video generation model training apparatus according to an exemplary embodiment is shown. DETAILED DESCRIPTION

[0064] Various exemplary embodiments, features, and aspects of the present disclosure will be explained in greater detail below with reference to the accompanying drawings. Like reference numerals may be used to refer to like elements throughout. While various aspects of embodiments are illustrated, the embodiments are not necessarily mutually exclusive, excluding other embodiments, unless specifically stated herein.

[0065] As used herein, the terms "comprise", "comprising", "have", "having", "include", "including", "contain", "containing", or variants thereof, are open-ended, and include one or more stated features, integers, elements, steps, components or functions but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof.

[0066] When an element is referred to as being "connected", "coupled", "responsive", or variants thereof, to another element, it can be directly connected, coupled, or responsive to the other element, or intervening elements can be present.

[0067] Although the terms first, second, third, etc. can be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of the present inventive concept.

[0068] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0069] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions, procedures, components, and circuits will not be described in detail, as it is appreciated that the present disclosure will become more apparent from the exemplary embodiments described below. It will be appreciated that the present disclosure is not limited in scope to the particular embodiments described herein, which are intended to be illustrative only and not exclusive.

[0070] Three-dimensional video (3D video) is a type of video content that can provide a three-dimensional visual effect. Compared with traditional two-dimensional video (2D video), three-dimensional video can provide viewers with a richer depth perception and immersive experience. Illustratively, three-dimensional video can be binocular stereoscopic video, which simulates the visual effect of human eyes by capturing and displaying images of two perspectives. The images of each perspective correspond to the left eye and the right eye, respectively, and the two images are presented to the left and right eyes through special display devices (such as 3D glasses or naked-eye 3D displays), thereby producing a sense of depth.

[0071] Figure 1 A flowchart of a method for training a video generation model according to an embodiment of the present disclosure is shown. As shown in the flowchart, the video generation model is used to generate three-dimensional video, and the method comprises: Figure 1

[0072] In step S11, the camera pose of the sample video is used to control the video generation model to output a video, and the video generation model is supervised and fine-tuned by the loss between the output video and the sample video.

[0073] The sample video is a video used for training the video generation model, which can be obtained by shooting with a real shooting device. The camera pose of the sample video is a parameter used to record the position and orientation of the shooting device in three-dimensional space. These parameters usually include a rotation matrix and a translation vector, which are used to accurately describe the viewing angle and position information of the camera at each image capture. For example, when shooting the same static scene from multiple perspectives, each frame will correspond to a set of pose data to represent the shooting viewpoint of the frame.

[0074] The video generation model is used to generate three-dimensional video, which can be an artificial intelligence-based generative model used to synthesize video sequences according to input conditions. The video generation model can be a diffusion model that converts random noise into coherent image frames that meet the condition constraints through a step-by-step denoising process, and then forms a video.

[0075] When the video generation model is supervised and fine-tuned, the pre-trained model can be further adjusted using existing labeled data to adapt to specific tasks. In this process, the difference between the model output and the real sample can be compared to construct a loss function, and the model parameters can be updated guided by the loss function to improve the performance of the model on the target task.

[0076] For example, the difference between the output video and the real video in the pixel space or the feature space can be calculated to form a loss signal to drive the model to better learn the mapping relationship between the control signal and the output content.

[0077] ​In the step, the camera pose information corresponding to the sample video can be used as a control signal to guide the video generation model to generate video output. By encoding the camera pose as a condition vector recognizable by the model and inputting it into the video generation model, the video generation model is controlled to output the video.

[0078] Subsequently, a loss function is constructed by determining the difference between the generated video and the original sample video. This loss function can reflect the degree of deviation between the generated content and the expected target, and then the model parameters are updated based on the gradient descent principle, so that the model gradually learns to generate video content of the corresponding view angle according to the input pose. For example, the pose information can be injected into the video generation model through a network structure such as ControlNet to guide the denoising process of the video generation model, and the model is optimized through mean square error or perceptual loss to make its output as consistent as possible with the real video.

[0079] This step uses a small amount of sample data to let the video generation model learn and understand the mapping relationship between the control signal of camera pose and the output video content, which lays a foundation for the subsequent model generalization on unseen videos and effectively reduces the dependence on large-scale high-quality labeled data.

[0080] In step S12, a group of generated videos is generated based on the same set of input conditions using the supervised fine-tuned video generation model.

[0081] The supervised fine-tuned video generation model has the ability to generate corresponding view angle videos according to camera pose. Based on this, by fixing the input conditions and calling the model for forward inference multiple times, a group consisting of multiple generated videos can be obtained.

[0082] The same set of input conditions here can be text descriptions and camera control signals that remain unchanged during the model generation process. These conditions together constitute specific constraints for the generation task, ensuring that the video generation model synthesizes content under the same set of semantics and view angle planning. Exemplarily, a text prompt word describing a scene together with a preset camera pose parameter can constitute a set of generation conditions.

[0083] During the generation process, by changing the sampling noise or random seed, the model explores different generation paths while following the same control signal, thereby introducing reasonable diversity in the output. A set of generated results is obtained based on the same input conditions through multiple independent samplings.

[0084] During the generation process, by introducing different random initial states or noise sequences, the video generation model can generate multiple video outputs that differ slightly in content details, texture performance, or local motion. These outputs together constitute a candidate set for comparison and evaluation.

[0085] For example, a video generation model can generate multiple video versions with slight differences in the lighting and shadows on the building surface or the amplitude of leaf swaying, based on the same text description and camera trajectory, due to differences in initial noise.

[0086] In step S13, each of the generated videos is converted into a three-dimensional Gaussian point cloud, and each of the three-dimensional Gaussian point clouds is rendered into a rendered video.

[0087] 3D Gaussian point clouds are a data format used to represent the structure of 3D scenes. They discretely model continuous 3D space using a large number of Gaussian distributions with specific attributes such as position, color, transparency, scale, and rotation. This representation method can efficiently store and render the geometric and appearance information of complex scenes. For example, an indoor scene can be represented as thousands of tiny, ellipsoidal colored points, each representing a tiny surface area in the scene.

[0088] In this step, the video generated by the video generation model can be converted into a 3D Gaussian point cloud. This conversion process can be based on a pre-trained large-scale reconstruction model that can infer the consistent 3D structure behind multiple video images and output it as a 3D Gaussian point cloud. Specific training methods can be found in the possible implementations provided in this disclosure, and will not be elaborated here.

[0089] Subsequently, using the obtained 3D Gaussian point cloud as input, a renderer is employed to regenerate a new rendered video based on the camera pose sequence corresponding to the original generated video. This rendered video is derived from the inferred 3D geometry.

[0090] In step S14, based on the consistency between each rendered video and the corresponding generated video, the reward score of each generated video is determined, and the video generation model is optimized based on the reward scores of each generated video in the same group.

[0091] 3D consistency refers to the characteristic of a generated video maintaining stable and consistent geometric structure and object form across different viewpoints. This consistency is reflected in the fact that when the viewpoint changes, objects in the scene do not exhibit unreasonable deformation, flickering, or floating phenomena, and their spatial relationships and surface textures remain coherent and realistic. For example, in a video with good 3D consistency, as the camera rotates around an object, the object's outline, size, and surface pattern will show smooth changes that conform to the laws of perspective.

[0092] If the 3D consistency of the generated video output by the video generation model is high, meaning that the multi-view information of the generated video is self-consistent, when converting it into a three-dimensional Gaussian point cloud representation, the resulting point cloud model will have a clear structure and few noise points, and the video rendered from this high-quality point cloud will also be very clear. Conversely, if the 3D consistency of the generated video is low, there are geometric conflicts between different views, and a large number of floating noise points (floaters) will be generated in the converted point cloud model, resulting in a significant drop in the quality of the final rendered video.

[0093] Therefore, when evaluating the three-dimensional consistency quality of each generated video, it can be achieved by comparing the consistency of the original generated video and its corresponding rendered video. Specifically, the differences between the two videos in terms of pixel intensity, structural information, or deep reading information can be calculated, and these differences can be quantified into a comprehensive reward score. If the consistency between the generated video and the rendered video is low, it indicates that the quality of the generated video is not high, and if the consistency between the generated video and the rendered video is high, it indicates that the quality of the generated video is high.

[0094] The reward score is a quantitative evaluation index used to measure the performance of the generated video in a specific quality dimension. The higher the score, the better the three-dimensional consistency of the generated video. The score is calculated by a pre-defined reward function, and the reward score is used to guide the update direction of the model parameters.

[0095] When optimizing the video generation model based on the reward score, a group relative strategy optimization method can be used to compare the reward scores obtained by different videos in the same group and calculate the relative advantage degree of each generated result. This advantage degree reflects the pros and cons of a specific generation strategy relative to the average level of the group.

[0096] During optimization, the parameters of the video generation model are adjusted so that the model tends to use generation strategies that produce high reward scores. For example, if a video obtains a higher reward score due to better three-dimensional consistency, the optimization algorithm will guide the model parameters to adjust in a direction that produces similar output characteristics, thereby gradually improving the overall generation quality of the model.

[0097] The optimized video generation model can be used to generate videos. When generating a video, a text prompt word and a camera pose parameter can be input, and the video generation model can generate a video that meets the camera pose parameter based on the text prompt word.

[0098] In the embodiments of the present disclosure, the camera pose of the sample video is used to control the video generation model to output a video, and the video generation model is supervised and fine-tuned through the loss between the output video and the sample video, so that the video generation model obtains a preliminary camera control capability; then, based on the same set of input conditions, the video generation model after supervised fine-tuning is used to generate a group containing multiple generated videos, each of the generated videos is converted into a three-dimensional Gaussian point cloud, and each of the three-dimensional Gaussian point clouds is rendered into a rendered video, the reward score of each generated video is determined based on the consistency between each of the rendered videos and the corresponding generated video, and the video generation model is optimized based on the reward scores of each generated video in the same group. By using the camera pose of the sample video to control the video generation model to output a video, and supervising and fine-tuning the video generation model through the loss between the output video and the sample video, the video generation model obtains a preliminary camera control capability, which enables the video generation model to start from a starting point with basic control capability for subsequent diversified exploration. Then, based on the same set of input conditions, the video generation model after supervised fine-tuning is used to generate a group containing multiple generated videos, and the video generation model is optimized based on the reward scores of each generated video in the same group, so that the model can explore multiple possible output results in parallel, and let the model freely explore video generation and learn how to generate more realistic videos, and the generated videos can be generalized to unseen scenes.

[0099] If the 3D consistency of the generated video output by the video generation model is high, which means that the multi-view information of the generated video is self-consistent, when the generated video is converted into a three-dimensional Gaussian point cloud representation, the obtained point cloud model will have a clear structure and few floater points, and the video rendered from the high-quality point cloud will also be very clear. On the contrary, if the 3D consistency of the generated video is low, there is a geometric conflict between different views, and a large number of floater points will be generated in the obtained point cloud model, resulting in a serious decline in the video quality of the final rendered video.

[0100] Therefore, the reward score is obtained by calculating the difference between the generated video and the rendered video, and the model is optimized accordingly, so that the reward score can accurately and objectively reflect the internal quality of the generated video in the 3D geometric level. A high score means that the generated video itself is close to its ideal 3D state, and the model will be encouraged to optimize in this direction; a low score indicates that the 3D structure has defects, and the model will be guided to avoid such output. In this way, the video generation model can be effectively driven to build strong 3D consistency constraints in its internal structure, so that it can generate high-quality 3D scene videos that remain stable, realistic and continuous under dynamic view switching without relying on massive labeled data.

[0101] In a possible implementation, the camera pose of the sample video is determined, the camera pose is encoded into a control signal, and the control signal is injected into a denoising process of the video generation model to obtain an output video.

[0102] In this implementation, when determining the camera pose of the sample video, camera parameter information corresponding to each frame can be extracted from original video data. These parameters are usually obtained directly through computer vision algorithms or device sensor data, and are used to accurately describe the perspective, position, and direction relationship at the time of shooting. For example, a Structure from Motion method can be used to calculate the rotation matrix and translation vector of the camera from multiple frames of images in a video sequence.

[0103] The control signal refers to camera pose data that has been processed through encoding and can be recognized and processed by a neural network. The signal converts spatial geometric information into a vector representation in the latent space of the model, thereby serving as a conditional input for the generation process.

[0104] Then, the camera pose parameters of each frame can be parsed from the sample video, and these parameters are converted into continuous vector representations through an encoding network (for example, a ControlNet network) to form a conditional control signal. Subsequently, when the video generation model performs a denoising operation, the control signal is fused with the noise features of the current time step to influence the prediction behavior of the model through a conditional modulation mechanism.

[0105] For example, in a latent diffusion model, the pose encoding vector and the time step encoding can be jointly input to an attention layer, so that the denoising process considers both the time dimension and the spatial constraint, and finally outputs a video sequence that is consistent with the specified camera trajectory.

[0106] In the embodiments of the present disclosure, the camera pose of the sample video is determined, the camera pose is encoded into a control signal, and the control signal is injected into a denoising process of the video generation model to obtain an output video. In this way, the video generation model can learn the correlation between the camera pose and the video content, and achieve the purpose of generating content through geometric parameters. This control method based on conditional injection enables the generated video to have accurate perspective correspondence, laying a foundation for subsequent three-dimensional consistency optimization. At the same time, this implementation maintains the original generation flexibility of the diffusion model, and can realize continuous generation of different perspectives by adjusting the control signal.

[0107] It should be noted that the process can be implemented by a small number of video samples. In the related art, when the video generation model is supervised fine-tuned, a large number of sample videos need to be collected in the real world, which requires a lot of manpower and material resources, that is, the video is collected by a real camera. If the video generation model to be trained has good generalization ability, tens of thousands or even millions of videos are needed.

[0108] In the embodiments of the present disclosure, when supervised fine-tuning is performed, only a small number of sample videos are needed to enable the model to learn to control video generation with camera poses, only a few hundred or a few thousand are needed. Subsequently, the video generation model can be allowed to freely explore video generation through reinforcement learning, which can enable the model to learn more information and generate videos that can be generalized to unseen scenes.

[0109] In a possible implementation, the reward score of each generated video is determined based on the consistency between each rendered video and the corresponding generated video, including: calculating the peak signal-to-noise ratio (PSNR) and / or structural similarity (SSIM) between the generated video and the corresponding rendered video to evaluate the geometric consistency and object structure continuity of the generated video and the rendered video; and determining the reward score based on the PSNR and / or SSIM.

[0110] When evaluating the consistency between the rendered video and the generated video, the evaluation can be based on the peak signal-to-noise ratio (PSNR) and / or structural similarity (SSIM).

[0111] The peak signal-to-noise ratio (PSNR) is an evaluation index for measuring the quality of video reconstruction. It evaluates the fidelity by calculating the mean square error between the original signal and the reconstructed signal. The index is in decibels, and the higher the value, the smaller the distortion, which can reflect the degree of accurate matching of the image in the pixel value level. For example, in video quality evaluation, PSNR can be used to quantify the difference in geometric information between the generated video and the reference video.

[0112] Structural similarity (SSIM) is a quality evaluation index based on the characteristics of the human visual system, which can comprehensively compare the similarity of two images from three dimensions of brightness, contrast and structure. Unlike PSNR, SSIM can capture the preservation of image structure information. Therefore, SSIM can detect the integrity of object edge preservation and the restoration degree of texture details in the generated video.

[0113] Then, when evaluating the consistency between the rendered video and the generated video, the generated video can be frame-aligned with the corresponding rendered video to ensure the consistency of timestamps and spatial positions. Then, the PSNR and / or SSIM indicators can be calculated for each frame image. When calculating the PSNR, the geometric consistency can be evaluated by calculating the mean square error between the pixel values of the two frames of images; when calculating the SSIM, the continuity of the object structure can be evaluated by comparing the structural features of the local image blocks.

[0114] The indicators calculated frame by frame can be aggregated into an overall score of the entire video sequence through temporal pooling. For example, the average or minimum value of all frame indicator values can be taken as the final consistency measure.

[0115] When determining the reward score based on these indicators, the PSNR and / or SSIM values and other related indicator values can be mapped into a unified reward value in a weighted combination manner. Since the PSNR and SSIM reflect the video quality from different angles, the combination of the two can more comprehensively evaluate the three-dimensional consistency. For example, a positive reward can be given when the PSNR and SSIM both reach a high value, and the reward score can be reduced accordingly when any indicator significantly decreases.

[0116] In this implementation, the geometric consistency and object structure continuity of the generated video and the rendered video are evaluated by calculating the peak signal-to-noise ratio (PSNR) and / or structural similarity (SSIM) between the generated video and the corresponding rendered video, and then determining the reward score based on the PSNR and / or SSIM. In this way, a continuous and fine-grained reward signal can be provided for reinforcement learning to guide the model to optimize the three-dimensional consistency. The PSNR indicator ensures that the generated video is close to the ideal three-dimensional reconstruction result at the geometric pixel level, and the SSIM indicator ensures that the structural integrity of the visual content is maintained. The quality of the generated video can be more comprehensively evaluated, so that the model can optimize the geometric accuracy and visual naturalness, and ultimately produce video content with high three-dimensional consistency.

[0117] In a possible implementation, the determining of the reward score of each generated video based on the consistency between each rendered video and the corresponding generated video includes: obtaining depth information corresponding to each frame image included in the rendered video; based on the depth information of the i th frame of the rendered video, and the camera pose of the i th frame and the camera pose of the i+1 th frame, projecting each pixel point of the i th frame of the generated video into the image coordinate system of the i+1 th frame through three-dimensional projection transformation to obtain a projected image; determining the inter-frame similarity between the projected image and the i+1 th frame of the generated video, and determining the reward score based on the similarity.

[0118] In evaluating the consistency between the rendered video and the generated video, the inter-frame depth consistency loss can also be evaluated.

[0119] The depth information is the distance data between each pixel point in the image and the optical center of the camera, which is usually represented in the form of a gray image or a distance matrix. This information can reflect the three-dimensional spatial structure and relative position relationship of objects in the scene, providing spatial geometric constraints for geometric consistency evaluation. For example, approximate depth information can be inferred from a single frame of image by using monocular depth estimation network, or more accurate depth information can be obtained by using multi-view stereo matching algorithm.

[0120] A rendered video sequence rendered by a 3D Gaussian point cloud can be obtained, as well as the depth information corresponding to each frame of image in the rendered video. At the same time, a generated video sequence to be evaluated is obtained. For two consecutive frames of images, given the depth information of the i-th frame, the camera pose of the i-th frame and the camera pose of the i+1-th frame, each pixel point in the i-th frame of the generated video can be projected into the image coordinate system of the next frame by three-dimensional projection transformation. In this projection process, the pixel of the current frame is back-projected to the three-dimensional space through the depth information, and then the pixel in the three-dimensional space is rotated and translated based on the camera pose of the i-th frame and the camera pose of the i+1-th frame, and then the pixel after the rotation and translation operation is projected onto the image plane of the next frame.

[0121] After the projection is completed, the projected image is compared with the i+1-th frame of image in the generated video to determine the similarity between the two. The specific similarity can be various, for example, the structural similarity index (SSIM) or normalized cross correlation (NCC) between the two images is calculated. Through this comparison, it can be evaluated whether the three-dimensional structure of the objects in the scene remains stable and consistent during the change of the camera view angle. For example, when the camera has a translational motion, the static objects in the scene should present the corresponding parallax change according to their depth information, and if the actual change in the generated video is inconsistent with the expected change calculated based on the depth and pose, it indicates that there is a problem of three-dimensional inconsistency in the generated video.

[0122] The smaller the similarity, the worse the three-dimensional consistency between the generated video and the consecutive frames; the larger the similarity, the better the three-dimensional consistency is maintained. Then, the reward score can be determined based on the similarity to guide the optimization process of the video generation model.

[0123] In the embodiments of the present disclosure, the depth information corresponding to each frame of image contained in the rendered video is obtained; based on the depth information of the i th frame of the rendered video, and the camera pose of the i th frame and the camera pose of the i+1 th frame, each pixel point of the i th frame of the generated video is projected into the image coordinate system of the i+1 th frame through three-dimensional projection transformation to obtain a projection image; the similarity between the projection image and the i+1 th frame of the generated video is determined, and the reward score is determined based on the similarity. Thus, the shape distortion, position jump or unreasonable motion trajectory of the object in the process of view angle change can be effectively identified. By incorporating the similarity into the reward mechanism, the video generation model can be guided to learn to maintain the stability of the three-dimensional structure of the object in the scene, thereby improving the spatial coherence and visual realism of the generated video.

[0124] In a possible implementation, the determining of the reward score of each generated video comprises: weighting the PSNR, the SSIM and / or the inter-frame similarity in the reward function by a weight to determine the reward score.

[0125] In this implementation, the determination of the reward score adopts a multi-index weighted fusion strategy. By combining multiple complementary evaluation dimensions, a comprehensive evaluation system for the quality of the generated video is formed. Each index reflects the quality characteristics of the video from a different angle, and a more comprehensive and balanced quality evaluation can be achieved through reasonable weight distribution.

[0126] The peak signal-to-noise ratio (PSNR) mainly reflects the closeness of the generated video and the rendered video in the pixel value level, and can capture obvious color and brightness distortion. The structural similarity (SSIM) pays more attention to the human visual perception characteristics, and evaluates the consistency of the two videos in structural information preservation. The inter-frame similarity focuses on the consistency of the depth information between consecutive frames, and detects the temporal stability of the three-dimensional geometric structure. These indexes each have different dimensions and numerical ranges, and therefore usually need to be normalized before weighting to make them within a unified numerical interval.

[0127] It should be noted that, in addition to the indexes mentioned in the present disclosure, other indexes can also be included in the reward function, which are not limited by the present disclosure.

[0128] In the weighting process, the distribution of the weight can be adjusted according to the needs of the specific application scenario, and specific reference can be made to the possible implementation provided by the present disclosure. The determination of the weight can be realized by experience setting or performance tuning based on a validation set. The final reward score can be obtained by weighted summation of the indexes, and the formula can be expressed as:

[0129] Reward score = w1 x PSNR + w2 x SSIM + w3 x inter-frame similarity + w4 x other indexes

[0130] wherein w1, w2, w3, w4 are weight coefficients corresponding to each index.

[0131] In the embodiments of the present disclosure, through the multi-index weighted fusion method, a more comprehensive and stable reward signal can be generated to provide clear and consistent guidance for model optimization. The complementary characteristics between different indexes help to reduce the evaluation bias that may be caused by a single index, so that the reward score can more accurately reflect the comprehensive quality of the generated video.

[0132] In a possible implementation, the method further includes: dynamically determining the weights based on an application scenario of the video generation model.

[0133] The application scenario is a specific application field and use environment served by the video generation model, and these scenarios may have different emphasis requirements for the quality characteristics of the generated content. Different application scenarios often focus on different quality dimensions, for example, virtual reality applications may pay more attention to visual immersion and three-dimensional geometric accuracy, while social media content generation may pay more attention to texture reality and real-time rendering efficiency. These differentiated requirements directly affect the importance allocation of each quality index in the final evaluation.

[0134] The process of dynamically determining the weights can be implemented based on a configurable weight management mechanism, which first identifies the current application scenario type (which can be set by the user independently), and then assigns appropriate weight coefficients to the PSNR, SSIM, and inter-frame similarity indexes according to the preset scenario-weight mapping relationship.

[0135] The weight configuration can be pre-set based on domain knowledge, or automatically learned from the scene features through machine learning methods. For example, in the building visualization scenario, the geometric consistency related indexes may be given higher weights, while in the film special effect scenario, more attention may be paid to the texture reality and visual aesthetic related indexes.

[0136] In actual operation, the application scenario can be automatically identified according to the text prompt word input by the user. After the identification is completed, the corresponding weight configuration file is called, and these weights are applied to the calculation process of the reward function. This dynamic weight adjustment can be performed in real time during model inference, so that the same model can be trained with different weights to adapt to the quality requirements of different scenarios.

[0137] The working principle of the dynamic weight mechanism is based on the differentiated requirements of different application scenarios for video quality. By analyzing the quality preferences of a specific scenario, the system can automatically adjust the evaluation focus, so that the reward function better fits the actual application requirements. The weight adjustment process can be regarded as an online parameter optimization, which maintains the uniformity of the evaluation framework while providing flexibility to adapt to different scenarios.

[0138] In a possible implementation, the application scenario of the video generation model is determined based on the text prompt input by the user, and the weight is dynamically determined based on the application scenario.

[0139] In this implementation, the application scenario type of the video generation task is automatically identified by analyzing the text prompt provided by the user. The semantic information contained in the text prompt can reflect the intended use and quality requirements of the generated content. For example, keywords such as "building walkthrough", "product demonstration", or "art creation" appearing in the prompt can indicate different application scenario classifications. These scenario classifications correspond to different quality evaluation focuses, thereby affecting the weight distribution of each indicator in the reward function.

[0140] Based on the identified application scenario, the corresponding weight can be determined according to the mapping relationship from scenario type to weight value. For example, when the text prompt contains keywords such as "virtual reality" and "VR", the system may judge it as an immersive application scenario, at which time the weights of inter-frame depth consistency and geometric accuracy are increased; while the prompt contains expressions such as "artistic effect" and "stylization", the weight of the geometric indicator can be appropriately reduced, and the weight proportion of texture quality and visual effect is increased.

[0141] In the determination process of the application scenario, rule-based matching or semantic similarity-based calculation can be used. Rule-based matching determines the corresponding application scenario category through keyword recognition, while the semantic similarity-based method can determine the corresponding scenario category by embedding the prompt into a vector space and then calculating the similarity score with each application scenario category.

[0142] In the embodiments of the present disclosure, by dynamically adjusting the weight based on the text prompt, the video generation system can intelligently adapt to the quality requirements of different application scenarios. By understanding the user's language description intent, the weight distribution of the evaluation indicators can be automatically optimized, so that the generated video content can better meet the professional needs of specific scenarios while maintaining basic quality.

[0143] In a possible implementation, the method further includes: inputting a sample video and its camera pose into a reconstruction model LRM to obtain a three-dimensional Gaussian point cloud corresponding to the sample video; rendering the three-dimensional Gaussian point cloud to obtain a reconstructed rendered video; calculating a loss between the reconstructed rendered video and the sample video as a true value, and updating parameters of the reconstruction model.

[0144] The reconstruction model can realize conversion of the generated video into a three-dimensional Gaussian point cloud and rendering of the three-dimensional Gaussian point cloud into a rendered video. The reconstruction model can be obtained by training. The reconstruction model is trained to have the conversion capability.

[0145] Specifically, in a data set used for training the reconstruction model, a plurality of sample videos and camera pose parameters corresponding to each video frame are included. The sample videos and the corresponding camera poses are input into the reconstruction model. The model learns the geometric correlation and visual consistency between the video frames, and outputs a corresponding three-dimensional Gaussian point cloud representation to continuously describe the geometric structure and appearance attributes of the scene in the form of a discrete Gaussian distribution set.

[0146] After obtaining the three-dimensional Gaussian point cloud, it is re-rendered into a video by a differentiable renderer. The rendering process is based on the camera pose sequence of the original sample video, ensuring that the rendered video completely corresponds to the original sample video in terms of perspective and timing. Then, a difference loss between the rendered video and the original sample video is calculated.

[0147] Based on the calculated loss, the parameters of the reconstruction model are updated by a back propagation algorithm to optimize its ability to reconstruct a three-dimensional scene from a multi-view video.

[0148] In the embodiments of the present disclosure, the reconstruction model is trained to have the ability to convert any video into an accurate three-dimensional Gaussian point cloud. This provides an important technical foundation for the evaluation of subsequent video generation models. The generated video output by the video generation model can be converted into a three-dimensional Gaussian point cloud and re-rendered by the reconstruction model to obtain a rendered video, which is used to calculate a reward score. This makes the evaluation process of the generated video not only based on two-dimensional visual similarity, but also based on three-dimensional geometric consistency, thereby more accurately guiding the video generation model to optimize its three-dimensional performance capability.

[0149] A specific implementation of the present disclosure is described below, which mainly includes the following stages:

[0150] Dataset construction stage. Through open source datasets or specially collected static scene videos, the camera internal and external parameters corresponding to each frame of image are calculated by using tools such as Colmap, to provide necessary multi-view data support for subsequent training. These data sets containing accurate pose annotations will serve the training of large reconstruction models and video generation models.

[0151] A large-scale reconstruction model is then trained. Video sequences and their camera poses in the scene dataset are input into the LRM, which learns to convert video frames into a three-dimensional Gaussian point cloud representation. Through a differentiable renderer, the point cloud is re-rendered into a video, and the model parameters are updated to enable the LRM to obtain the ability to convert video into high-quality three-dimensional reconstruction, with the perceptual loss between the rendered video and the original sample video as the optimization objective.

[0152] Next, the video generation model is supervised fine-tuned. Using the videos and camera poses in the scene dataset, the pose information is encoded into control signals through the ControlNet architecture and injected into the denoising process of the diffusion model. By calculating the difference loss between the generated video and the real video, the model initially obtains the ability to generate videos of corresponding perspectives according to camera poses.

[0153] Finally, the video generation model is fine-tuned through reinforcement learning optimization using a group relative strategy. The video generation model generates a group containing multiple generated videos based on the same set of input conditions through multiple samplings, converts each generated video into a three-dimensional Gaussian point cloud and renders it into a rendered video. Based on the consistency difference between the generated video and the rendered video, the reward score is calculated, and the model parameters are updated through the GRPO algorithm to prioritize the optimization of the 3D consistency and geometric continuity of the generated video. The entire process adapts to the needs of different application scenarios through a dynamic weight mechanism, ultimately achieving high-quality and high-consistency 3D static scene video generation.

[0154] Figure 2 A block diagram of a video generation model training apparatus of an embodiment of the present disclosure is shown. As shown in Figure 2 The video generation model is used to generate three-dimensional videos, and the apparatus 20 includes:

[0155] The fine-tuning module 21 is configured to control the video generation model to output videos using the camera poses of the sample videos, and to supervise the fine-tuning of the video generation model through the loss between the output videos and the sample videos.

[0156] The group generation module 22 is configured to generate a group containing multiple generated videos based on the same set of input conditions using the video generation model that has been supervised and fine-tuned.

[0157] The rendered video generation module 23 is configured to convert each of the generated videos into a three-dimensional Gaussian point cloud, and to render each of the three-dimensional Gaussian point clouds into a rendered video.

[0158] The model optimization module 24 is configured to determine the reward score of each generated video based on the consistency between each rendered video and the corresponding generated video, and to optimize the video generation model based on the reward scores of each generated video in the same group.

[0159] In a possible implementation, the fine-tuning module is configured to:

[0160] determine a camera pose of the sample video;

[0161] encode the camera pose into a control signal, and inject the control signal into a denoising process of the video generation model to obtain an output video.

[0162] In a possible implementation, the model optimization module is configured to:

[0163] calculate a peak signal-to-noise ratio (PSNR) and / or a structural similarity (SSIM) between the generated video and the corresponding rendered video, to evaluate the geometric consistency and object structure continuity of the generated video and the rendered video;

[0164] determine a reward score based on the PSNR and / or the SSIM.

[0165] In a possible implementation, the model optimization module is configured to:

[0166] obtain depth information corresponding to each frame of the rendered video;

[0167] based on the depth information of the i th frame of the rendered video, and the camera pose of the i th frame and the camera pose of the i+1 th frame, project each pixel point of the i th frame of the generated video into the image coordinate system of the i+1 th frame through three-dimensional projection transformation, to obtain a projection image;

[0168] determine an inter-frame similarity between the projection image and the i+1 th frame of the generated video, and determine a reward score based on the similarity.

[0169] In a possible implementation, the model optimization module is configured to:

[0170] weight the PSNR, the SSIM, and / or the inter-frame similarity in the reward function by a weight, to determine the reward score.

[0171] In a possible implementation, the apparatus further includes:

[0172] a weight determination module configured to dynamically determine the weight based on an application scenario of the video generation model.

[0173] In a possible implementation, the weight determination module is configured to determine the application scenario of the video generation model based on a text prompt word input by a user, and dynamically determine the weight based on the application scenario.

[0174] In a possible implementation, the apparatus further includes a reconstruction model training module configured to:

[0175] inputting a sample video and a camera pose thereof into a reconstruction model LRM to obtain a three-dimensional Gaussian point cloud corresponding to the sample video;

[0176] rendering the three-dimensional Gaussian point cloud to obtain a reconstructed rendered video;

[0177] calculating a loss between the reconstructed rendered video and the sample video as a true value, and updating parameters of the reconstruction model.

[0178] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can be referred to the description of the above method embodiments. For brevity, it will not be described here again.

[0179] The embodiments of the present disclosure also provide a video generation model training apparatus, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above method.

[0180] The embodiments of the present disclosure also provide a non-volatile computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the above method.

[0181] The embodiments of the present disclosure also provide a computer program product, including a computer program or a non-volatile computer readable storage medium carrying the computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0182] Figure 3 is a block diagram of an apparatus for video generation model training according to an exemplary embodiment. For example, the apparatus 1900 can be provided as a server or a terminal device. Referring to Figure 3 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0183] The apparatus 1900 can also include a power supply component 1926 configured to perform power management of the apparatus 1900, a wired or wireless network interface 1950 configured to connect the apparatus 1900 to a network, and an input output interface 1958 (I / O interface). The apparatus 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TMMacOS X TM Unix TM Linux TM FreeBSD TM or the like.

[0184] In an example embodiment, there is also provided a non-transitory computer- readable storage medium, e.g., memory 1932 including computer program instructions, which can be executed by the processing component(s) 1922 of the apparatus 1900 to implement the above-described methods.

[0185] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0186] Computer programs (or computer readable program instructions), which describe the methods described herein, can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device from a network, e.g., the Internet, a local area network, a wide area network, and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0187] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0188] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0189] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0190] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0191] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0192] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive. Many modifications and variations of the described embodiments are possible and are within the scope of the disclosure. The selection of terms is intended to best describe the principles of the embodiments, practical application, or technical improvements in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a video generation model, characterized in that, The video generation model is used to generate 3D video, including: The video generation model is controlled by the camera pose of the sample video to output the video, and the loss between the output video and the sample video is used to supervise and fine-tune the video generation model. Using the supervised fine-tuned video generation model, a group containing multiple generated videos is generated based on the same set of input conditions; Each of the generated videos is converted into a 3D Gaussian point cloud, and each of the 3D Gaussian point clouds is rendered into a rendered video. Based on the consistency between each rendered video and its corresponding generated video, a reward score is determined for each generated video, and the video generation model is optimized based on the reward scores of each generated video in the same group.

2. The method according to claim 1, characterized in that, The video generation model that uses the camera pose of sample videos to control the output of video includes: Determine the camera pose of the sample video; The camera pose is encoded into a control signal, and the control signal is injected into the denoising process of the video generation model to obtain the output video.

3. The method according to claim 1, characterized in that, The determination of the reward score for each generated video based on the consistency between each rendered video and its corresponding generated video includes: Calculate the peak signal-to-noise ratio (PSNR) and / or structural similarity (SSIM) between the generated video and the corresponding rendered video to evaluate the geometric consistency and object structural continuity between the generated video and the rendered video; The reward score is determined based on the PSNR and / or SSIM.

4. The method according to claim 1, characterized in that, The determination of the reward score for each generated video based on the consistency between each rendered video and its corresponding generated video includes: Obtain the depth information corresponding to each frame of the rendered video; Based on the depth information of the i-th frame in the rendered video, as well as the camera pose of the i-th frame and the camera pose of the (i+1)-th frame, each pixel of the i-th frame in the generated video is projected into the image coordinate system of the (i+1)-th frame through a 3D projection transformation to obtain the projected image. Determine the inter-frame similarity between the projected image and the (i+1)th frame in the generated video, and determine the reward score based on the similarity.

5. The method according to claim 1, characterized in that, The determination of the reward score for each generated video includes: The reward score is determined by weighting the PSNR, SSIM, and / or inter-frame similarity in the reward function.

6. The method according to claim 5, characterized in that, The method further includes: The weights are dynamically determined based on the application scenario of the video generation model.

7. The method according to claim 6, characterized in that, The application scenario based on the video generation model dynamically determines the weights, including: Based on the text prompts input by the user, the application scenario of the video generation model is determined, and the weights are dynamically determined based on the application scenario.

8. The method according to claim 1, characterized in that, The method further includes: The sample video and its camera pose are input into the reconstruction model LRM to obtain the three-dimensional Gaussian point cloud corresponding to the sample video. The three-dimensional Gaussian point cloud is rendered to obtain a reconstructed rendered video; Calculate the loss between the reconstructed rendered video and the sample video as the ground truth, and update the parameters of the reconstruction model.

9. A video generation model training device, characterized in that, The video generation model is used to generate 3D video, including: The fine-tuning module is used to control the output video of the video generation model by using the camera pose of the sample video, and to perform supervised fine-tuning of the video generation model by comparing the loss between the output video and the sample video. The group generation module is used to generate a group containing multiple generated videos based on the same set of input conditions using the video generation model after supervised fine-tuning. The video rendering module is used to convert each of the generated videos into three-dimensional Gaussian point clouds, and to render each of the three-dimensional Gaussian point clouds into a video rendering module. The model optimization module is used to determine the reward score of each generated video based on the consistency between each rendered video and the corresponding generated video, and to optimize the video generation model based on the reward scores of each generated video in the same group.

10. A video generation model training device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

11. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images