Video generation model training method and device, video generation method and device, electronic equipment, storage medium and computer program product

Abstract

The invention provides a video generation model training method and device, a video generation method and device, electronic equipment, a storage medium and a computer program product, and relates to the technical field of artificial intelligence. Wherein the sample data comprises a sample object graph, sample description information and a sample video; inputting the sample video into the first video generation model for supervised fine tuning to obtain a second video generation model; inputting the sample object graph and the sample description information into a second video generation model to obtain a plurality of output videos, and determining the loss of the second video generation model based on one or more of an object consistency feature, a text stability feature and a creative rationality feature in each output video, and updating parameters of the second video generation model based on the loss to obtain a third video generation model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a video generation model training method, a video generation method, a device, an electronic device, a storage medium and a computer program product. BACKGROUND

[0002] A video generation model is an important research direction in the field of artificial intelligence, aiming to automatically generate video content that meets specific needs through algorithms. With the development of deep learning and generation models, diffusion models, Transformers and other technologies have made significant progress in video generation, enabling the generation of videos with high visual quality. However, in specific application scenarios such as e-commerce, video generation not only requires realistic pictures, but also needs to meet complex constraints such as clear subject, stable text, and reasonable shot logic.

[0003] In related technologies, a model is usually trained with a small amount of video data with product attributes to generate videos, so that it can master basic product display or text rendering capabilities. However, these methods still have deficiencies in handling object consistency, text stability, and creative rationality, especially in the multi-round generation process, which can easily cause subject confusion, text disappearance, or creative deviation, resulting in low video quality. SUMMARY

[0004] The present application provides a video generation model training method, a video generation method, a device, an electronic device, a storage medium and a computer program product.

[0005] The technical solution of the present application is implemented as follows: The present application provides a video generation model training method, comprising: Obtaining a plurality of sample data; wherein the sample data comprises a sample object graph, sample description information and a sample video; Inputting the sample video into a first video generation model for supervised fine-tuning to obtain a second video generation model; Inputting the sample object graph and the sample description information into the second video generation model to obtain a plurality of output videos, and determining a loss of the second video generation model based on one or more of object consistency features, text stability features and creative rationality features in each output video, and updating parameters of the second video generation model based on the loss to obtain a third video generation model.

[0006] In the scheme, the sample object graph and the sample description information are input into the second video generation model to obtain a plurality of output videos, and one or more of object consistency features, text stability features and creative rationality features in each of the output videos are used to determine a loss of the second video generation model, including: The sample object graph, the sample description information, a latent variable and current policy information for a current state are input into the second video generation model, and a plurality of output videos are generated based on a Markov decision process, and next policy information for a next state is generated during the generation process; wherein the latent variable is determined based on random back sampling; Based on the object consistency features of the sample objects included in the output videos and the text stability features of the text information, and the creative rationality features of the output video description content, a reward parameter is determined; Based on the reward parameter, the number of the plurality of output videos, and the ratio of each of the next policy information to the corresponding current policy information, the loss of the second video generation model is determined.

[0007] In the scheme, the reward parameter is determined based on the object consistency features of the sample objects included in the output videos and the text stability features of the text information, and the creative rationality of the output video description content, including: Based on the similarity of the sample objects in each video frame of the output video to the sample objects of other video frames, the object consistency features are determined; wherein the other video frames are any frame other than the video frame in the output video; Based on the difference between the text information in the video frame of the output video and the text information in adjacent video frames and reference video frames, the text stability features are determined; The output video is processed based on a preset model to determine the creative rationality features; wherein the preset model is used to determine the narrative fluency, creative evolution intensity and semantic consistency of the output video description content; The reward parameter is determined based on the object consistency features, the text stability features and the creative rationality features.

[0008] In the scheme, the object consistency features are determined based on the similarity of the sample objects in each video frame of the output video to the sample objects of other video frames, including: Object features of the sample objects of each video frame are extracted, and a first confidence corresponding to the sample objects is determined; determine an object similarity degree corresponding to each of the video frames based on the object feature of each of the video frames, similarity between the object feature of each of the video frames and the object feature of other video frames, and the first confidence degree; determine the object consistency feature based on a mean value of the object similarity degrees of the video frames and a proportion of the object similarity degrees greater than a similarity threshold in the object similarity degrees.

[0009] In the above scheme, the text stability feature is determined based on a difference between the text information in the video frame of the output video and the text information in adjacent video frames and a reference video frame. extract a text feature of the text information of each of the video frames and determine a second confidence degree corresponding to the text information; determine a cross-frame stability degree based on a first difference value between the text feature of each of the video frames and the text feature of adjacent video frames and the corresponding second confidence degree; determine a reference frame text fidelity degree based on a second difference value between the text feature of each of the video frames and the text feature of a reference video frame and the corresponding second confidence degree; determine the text stability feature based on the cross-frame stability degree and the reference frame text fidelity degree.

[0010] In the above scheme, the creative rationality feature is determined based on processing the output video by a preset model. divide the output video into multiple sub-videos based on video content scenes in the output video; wherein video frames in a same sub-video correspond to the same video content scene; input the multiple sub-videos into the preset model to determine narrative fluency information, creative evolution intensity information, and semantic consistency information; determine the creative rationality feature based on the narrative fluency information, the creative evolution intensity information, and the semantic consistency information.

[0011] Embodiments of the present application also provide a video generation method, comprising: obtain a current object graph and current description information; input the current object graph and the current description information into a third video generation model to obtain a corresponding target video; The third video generation model is obtained by inputting sample object graphs and sample description information into the second video generation model to obtain multiple output videos, and by determining the loss of the second video generation model based on one or more of the object consistency features, text stability features, and creative rationality features in each output video, and updating the parameters of the second video generation model based on the loss; the second video generation model is obtained by supervising the fine-tuning of the first video generation model by inputting sample videos.

[0012] This application also provides a video generation model training device, including: The first data acquisition unit is used to acquire multiple sample data; wherein, the sample data includes: sample object image, sample description information and sample video; The first training unit is used to input the sample video into the first video generation model for supervised fine-tuning to obtain the second video generation model. The second training unit is used to input the sample object graph and the sample description information into the second video generation model to obtain multiple output videos, and to determine the loss of the second video generation model based on one or more of the object consistency feature, text stability feature and creative rationality feature in each output video, and to update the parameters of the second video generation model based on the loss to obtain a third video generation model.

[0013] This application also provides a video generation apparatus, including: The second data acquisition unit is used to acquire the current object graph and current description information; The generation unit is used to input the current object image and the current description information into the third video generation model to obtain the corresponding target video; The third video generation model is obtained by inputting sample object graphs and sample description information into the second video generation model to obtain multiple output videos, and by determining the loss of the second video generation model based on one or more of the object consistency features, text stability features, and creative rationality features in each output video, and updating the parameters of the second video generation model based on the loss; the second video generation model is obtained by supervising the fine-tuning of the first video generation model by inputting sample videos.

[0014] This application also provides a first electronic device, including a first memory and a first processor. The first memory stores a computer program that can run on the first processor. When the first processor executes the computer program, it implements the steps in the video generation model training method.

[0015] This application also provides a second electronic device, including a second memory and a second processor. The second memory stores a computer program that can run on the second processor. When the second processor executes the computer program, it implements the steps in the video generation method.

[0016] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a first processor, implements the steps in the video generation model training method.

[0017] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a second processor, implements the steps in the video generation method.

[0018] This application also provides a computer program product, including a computer program that, when executed by a first processor, implements the steps in the video generation model training method.

[0019] This application also provides a computer program product, including a computer program that, when executed by a second processor, implements the steps in the video generation method.

[0020] In this embodiment, multiple sample data are acquired, including sample object graphs, sample description information, and sample videos. The sample videos are input into a first video generation model for supervised fine-tuning to obtain a second video generation model. The sample object graphs and sample description information are then input into the second video generation model to obtain multiple output videos. Based on the object consistency features, text stability features, and creative rationality features in each output video, the loss of the second video generation model is determined. The parameters of the second video generation model are updated based on the loss to obtain a third video generation model. Thus, by introducing a reinforcement learning optimization mechanism after supervised fine-tuning, and combining three types of reward signals—object consistency, text stability, and creative rationality—the video generation model is optimized online. This significantly improves the cross-frame consistency of product appearance, the clarity and stability of Chinese text, and the commercial logic of creative expression in e-commerce short videos, effectively solving the problems of unstable, inaccurate, and unreasonable video content in existing technologies, and improving the quality of generated videos. Attached Figure Description

[0021] Figure 1 A flowchart illustrating the video generation model training method provided in this application embodiment. Figure 1 ; Figure 2 A flowchart illustrating the video generation model training method provided in this application embodiment. Figure 2 ; Figure 3A flowchart illustrating the video generation model training method provided in this application embodiment. Figure 3 ; Figure 4 This is a schematic diagram illustrating the effect of the video generation model training method provided in the embodiments of this application; Figure 5 Flowchart of the video generation method provided in the embodiments of this application Figure 1 ; Figure 6 This is a schematic diagram of the structure of the video generation model training device provided in the embodiments of this application; Figure 7 This is a schematic diagram of a hardware entity of a first electronic device provided in an embodiment of this application.

[0022] Figure 8 This is a schematic diagram of the structure of the video generation apparatus provided in the embodiments of this application; Figure 9 This is a schematic diagram of a hardware entity of a second electronic device provided in an embodiment of this application.

[0023] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The video generation model training methods provided in the embodiments of this application can be executed by electronic devices, which may be devices with computing capabilities such as computers, servers, or cloud platforms. The video generation model training methods in the embodiments of this application can be executed independently by any of the aforementioned devices, or they can be executed collaboratively by multiple devices.

[0026] This application provides a method for training a video generation model. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating the video generation model training method provided in this application embodiment. Figure 1 , will combine Figure 1 The steps shown are explained below: S101. Acquire multiple sample data; wherein, the sample data includes: sample object image, sample description information, and sample video.

[0027] In this embodiment, each sample data point is used to perform supervised fine-tuning (SFT) on the first video generation model to obtain a second video generation model with basic e-commerce short video generation capabilities. Specifically, the sample object image refers to a product image, usually a high-resolution picture of the product, containing its main appearance features. The sample description information refers to Chinese text content related to the product, such as product name, functional selling points, brand introduction, performance parameters, etc. The sample video is a real e-commerce short video created based on the sample object image and sample description information, serving as a reference for the target output. The sample object image, sample description information, and sample video together constitute the first stage input for training, enabling the first video generation model to learn how to generate video content that meets e-commerce requirements based on the sample object image and sample description information.

[0028] In this embodiment, the sample data can be extracted from the historical video library of the e-commerce platform or constructed through manual annotation. Due to the high degree of industry diversity in e-commerce short videos, the sample data should cover a variety of product categories and copywriting styles to improve the model's generalization ability. Therefore, when collecting sample data, the diversity of the sample data should be ensured to avoid the model learning only limited templated content.

[0029] S102. Input the sample video into the first video generation model for supervised fine-tuning to obtain the second video generation model.

[0030] In this embodiment of the application, the SFT of the first video generation model is used to adapt the general video generation model to the e-commerce short video scenario. The model optimized by the supervised fine-tuning method has basic product display capabilities, including matching between sample object graphs and sample description information, learning common shot logic, etc.

[0031] The first video generation model is a general video generation model that has been pre-trained on large-scale video data, such as Wan, Kling, or Seedance. By taking sample videos as input, the first video generation model generates videos similar to the sample videos, compares them with real videos, calculates a loss function to adjust model parameters, and finally obtains the second video generation model. The sample videos include sample objects and sample description information from the same sample data.

[0032] In this embodiment, during the training of the first video generation model, regarding the data construction in stream matching, for a given real video data x0 and Gaussian noise x1, FM defines the intermediate state at any time t∈[0,1]: x t =(1-t)x0+tx1. Model learning velocity field v θ For (xt,t), minimize the following loss: .

[0033] S103. Input the sample object graph and the sample description information into the second video generation model to obtain multiple output videos, and determine the loss of the second video generation model based on one or more of the object consistency feature, text stability feature and creative rationality feature in each output video, and update the parameters of the second video generation model based on the loss to obtain a third video generation model.

[0034] In this embodiment, after receiving the sample object image and sample description information, the second video generation model generates multiple output videos. These multiple output videos reflect the second video generation model's current understanding and generation capabilities regarding the input conditions. Each output video is evaluated, and reward parameters are calculated based on the evaluation results. The reward parameters can be determined based on one or more of the three key features of the output videos generated by the second video generation model: object consistency, text stability, and creative rationality. Object consistency reflects the degree of consistency of the product across different frames, including consistency in shape, material, color, and details. Text stability reflects the clarity, completeness, and stability of the Chinese text over time. Creative rationality measures the rationality of the creative evolution exhibited during the video generation process, including narrative fluency, the intensity of creative evolution, and semantic consistency.

[0035] In this embodiment, for the same input condition c (sample object graph and sample description information), the model generates G output videos. in, This represents the i-th output video, and the reward parameters are calculated for each video. By comparing these reward parameters, an advantage function can be calculated, and the model parameters of the second video generation model can be updated accordingly. This process employs the Group Relative Policy Optimization (GRPO) algorithm, which gradually makes the model tend to generate high-reward video content. As training progresses, the model parameters are continuously optimized until predetermined training conditions (such as maximum number of iterations, convergence threshold, etc.) are reached, at which point the process stops, ultimately yielding the third video generation model.

[0036] The reward parameter is a numerical metric used to quantify video quality. It is typically calculated based on the key features of the output video and serves to guide model optimization. For example, if an output video demonstrates good product consistency, its reward parameter value will be increased accordingly.

[0037] Object consistency refers to the degree to which a product maintains consistency across different frames, including consistency in shape, material, color, and detail. For example, a product should not exhibit sudden distortion or color changes in the video. The Product Consistency Bonus (ICScore) is a reward used to measure object consistency. A higher ICScore indicates better consistency across the output video. For example, a higher ICScore indicates better consistency if the product is clearly visible in all frames without significant distortion.

[0038] Text stability refers to the degree to which Chinese text remains clear, complete, and stable over time. For example, when designing subtitles, video producers should ensure that the subtitles do not flicker frequently and avoid typos. The Chinese text stability bonus (VTScore) is a reward used to measure the stability of Chinese text; a higher score indicates a more stable presentation of the text in the video. For example, a higher bonus score is awarded when the output video subtitles are consistently clear, readable, and free of jitter.

[0039] The "Creative Rationality" feature refers to the reasonableness of the creative evolution displayed during the video's generation process, including narrative fluency, the intensity of creative evolution, and semantic consistency. For example, the output video should not contain abrupt editing or illogical plot jumps. The "Creative Rationality Bonus (CCScore)" is a reward used to measure the creative rationality feature; a higher score indicates that the output video's creative expression is more natural and reasonable. For example, a higher CCScore is awarded when the plot development is smooth and the semantics are coherent.

[0040] The third video generation model refers to the model optimized using reinforcement learning based on GRPO, which is the final model further optimized from the second video generation model. For example, the third video generation model may outperform the second video generation model in several aspects, such as product consistency, text stability, and creative rationality. The model optimized using reinforcement learning based on GRPO refers to the model obtained by gradually optimizing the model parameters by comparing the rewards of multiple sets of videos using the Group Relative Policy Optimization (GRPO) algorithm.

[0041] In this embodiment, multiple sample data are acquired, including sample object graphs, sample description information, and sample videos. The sample object graphs, sample description information, and sample videos are input into a first video generation model for supervised fine-tuning to obtain a second video generation model. The sample object graphs and sample description information are then input into the second video generation model to obtain multiple output videos. The loss of the second video generation model is determined based on the reward parameters of each output video. The parameters of the second video generation model are updated based on the loss until a predetermined training condition is met, resulting in a third video generation model. The reward parameters are determined based on one or more of the following features of the output videos: object consistency, text stability, and creative rationality. By introducing a reinforcement learning optimization mechanism after supervised fine-tuning, and combining the three types of reward signals—object consistency, text stability, and creative rationality—the video generation model is optimized online. This significantly improves the cross-frame consistency of product appearance, the clarity and stability of Chinese text, and the commercial logic of creative expression in e-commerce short videos, effectively solving the problems of unstable, inaccurate, and unreasonable video content in existing technologies and improving the quality of generated videos.

[0042] Please see Figure 2 This is a flowchart illustrating the video generation model training method provided in this application embodiment. Figure 2 , Figure 1 S103 in the above can also be implemented through S201 to S203, combining Figure 2 The steps shown are explained below: S201. Input the sample object graph, the sample description information, the latent variables, and the current policy information for the current state into the second video generation model, and generate multiple output videos based on the Markov decision process, as well as the next policy information for the next state during the generation process; wherein, the latent variables are determined based on random backsampling.

[0043] In this embodiment, latent variables refer to random variables used to enhance the diversity and exploratory capabilities of the model during video generation. Latent variables are typically generated from a preset probability distribution, which is used to perturb the model's generation path, making each generation slightly different. The introduction of latent variables can prevent the model from getting trapped in local optima, improving the diversity and creativity of the generated videos. In this embodiment, latent variables are determined using a random backsampling method, that is, by backsampling noise or intermediate states to generate uncertain initial conditions, thereby influencing the direction of the generation results.

[0044] Current policy information refers to the action strategy that the model adopts based on the existing state at the current time step. This information determines the direction and intensity of the generated action. For example, in video generation, current policy information might include the direction of camera movement and the trend of object position changes. Current policy information provides the model with contextual guidance for the generation process, enabling the model to make reasonable predictions for the next step based on historical information.

[0045] In this embodiment, video generation is modeled as a Markov decision process. In this process, each generation operation corresponds to a state transition, and each action corresponds to the agent's generation decision. This modeling approach allows the agent to progressively optimize its generation strategy within a reinforcement learning framework, thereby maximizing the final reward function. The model first receives input conditions (sample object graph and sample description information) and, combined with the current policy information and randomly generated latent variables, generates a series of candidate output videos. Simultaneously, the model dynamically updates the next policy information for the next state during the generation process, providing feedback and adjustment for subsequent generation.

[0046] In this embodiment, there is a synergistic relationship between latent variables and current policy information. Latent variables introduce uncertainty, while current policy information provides directional guidance. The latent variables and current policy information work together to influence the model's generative path, ensuring that the model can explore diverse outcomes without deviating from the theme.

[0047] S202. Based on the object consistency characteristics of the sample objects included in the output video, the text stability characteristics of the text information, and the creative rationality characteristics of the description content of the output video, determine the reward parameters.

[0048] In this embodiment, object consistency features can be determined based on the consistency of sample objects included in the output video, and text stability features can be determined based on the stability of text information in the output video. Creative rationality features are then determined based on the narrative fluency, creative evolution intensity, and semantic consistency of the output video's descriptive content. Finally, reward parameters are determined based on the weighted sum of object consistency features, text stability features, and creative rationality features. In this context, sample object consistency refers to whether the sample objects in the sample object graph maintain their shape, material, color, and detail throughout the entire sequence in the generated video. Sample object consistency helps ensure that the generated video meets the stringent requirements for product display in e-commerce scenarios.

[0049] The stability of text information refers to whether the Chinese text presented in the video remains clear, stable, and accurate across frames. Since e-commerce videos typically contain a large amount of key information such as brand names and selling points, this information must always be visible and easily readable throughout the video. If the text exhibits jitter, missing characters, or garbled text, it will severely impact users' understanding and trust in the product.

[0050] Narrative fluency refers to whether the video content has a natural and coherent narrative logic. Good narrative fluency helps viewers understand the development of the video content and enhances the viewing experience. For example, a product demonstration video should unfold gradually from appearance display to function demonstration and then to usage scenarios, rather than jumping between video content.

[0051] Creative evolution intensity refers to whether a video possesses sufficient creative expression while showcasing a product, including variations in visual design, camera language, and narrative structure. Strong creative evolution intensity can attract user attention and enhance a video's reach and commercial value. However, creative expression must not deviate from the product's core theme; otherwise, the content will stray from its central objective.

[0052] Semantic consistency refers to whether the video content revolves around the same theme as a whole, and whether the semantic logic between different parts is consistent. For example, if the first half of the video shows the appearance of a product, and the second half suddenly shifts to a completely unrelated scene, it will break semantic consistency and confuse the audience.

[0053] In this embodiment, each output video is evaluated according to the five dimensions mentioned above, and a reward parameter is calculated based on the combined results. The reward parameter will serve as an important feedback signal for model training, measuring the degree to which video quality matches business needs.

[0054] S203. Based on the reward parameters, the number of multiple output videos, and the ratio of each next policy information to the corresponding current policy information, determine the loss of the second video generation model.

[0055] In this embodiment, the ratio of the next policy information to the current policy information is used to measure the degree of improvement of the model's policy during the generation process. If the next policy is significantly better than the current policy, it indicates that the model is developing in the right direction. Conversely, if the ratio is close to or even lower than 1, it may indicate that the model has not yet found an effective generation policy. A comprehensive loss function is calculated by combining the reward parameter, the number of multiple output videos, and the ratio of each next policy information to its corresponding current policy information. This comprehensive loss function considers not only the quality of individual output videos but also the overall evolution trend of the generation policy, thereby guiding the model to continuously optimize its generation capabilities during the training process.

[0056] In this embodiment, the sample object graph and sample description information serve as the basic inputs throughout the process, triggering the generation mechanism of the generative model based on this input data. Latent variables and current policy information together determine the diversity and directionality of the generation path. The generation results are evaluated using multi-dimensional reward parameters, and a loss function is constructed in conjunction with the policy evolution trend, thereby driving the generative model to continuously iterate and optimize. Ultimately, the system achieves the goal of generating high-quality e-commerce short videos.

[0057] In this embodiment, an advantage parameter can be calculated based on the reward parameter corresponding to each output video. Then, based on the advantage parameter and the ratio of the next policy information to the corresponding current policy information in each state during the output video generation process, a product is determined. Next, another product of the pruning information (the ratio of the next policy information to the corresponding current policy information) and the advantage parameter is determined, and the sum of these two products is calculated for each state. The mean of this sum is calculated to obtain a parameter corresponding to each output video. The mean of this parameter is then calculated, and finally, the expected value is calculated to determine the loss corresponding to the second video generation model. This loss is then calculated by combining the loss with the corresponding pruning information and the advantage parameter of the output video. The advantage parameter can be calculated using formula (1). The loss can be calculated using formula (2).

[0058] Formula (1) in, This represents the reward parameter for the i-th output video. This represents the average of multiple reward parameters. This represents the standard deviation of multiple reward parameters.

[0059] Formula (2) Where G represents the number of output videos, and T represents the T states in the output video generation process. This represents the ratio of a policy information to the corresponding current policy information. This represents the advantage parameter of the i-th output video. This indicates the clipping of the contrast value. Represents a constant. It indicates a desire for the expected value.

[0060] In this embodiment, by introducing latent variables, constructing a Markov decision process, setting multi-dimensional reward parameters, and designing a policy ratio loss function, the diversity and exploratory capabilities of the model can be enhanced. Furthermore, by using the ratio of the next policy information to the current policy information to construct the loss function, the direction of model behavior improvement can be more accurately evaluated, improving model training efficiency and final generation quality. This effectively enhances the video generation model's performance in terms of object consistency, Chinese text stability, and creative rationality. With the support of the above methods, the video generation model can generate high-quality video content.

[0061] Please see Figure 3 This is a flowchart illustrating the video generation model training method provided in this application embodiment. Figure 3 , Figure 2 S202 in the process can also be implemented through S301 to S304, combining Figure 3 The steps shown are explained below: S301. Based on the similarity between the sample object in each video frame of the output video and the sample object in other video frames, determine the object consistency feature; wherein, the other video frames are any other frame in the output video besides the video frame.

[0062] In this embodiment, the object consistency feature can be calculated as follows: First, the object region in each frame is identified using an object detection model. Then, the visual features of the object region identified by the object detection model are used. Subsequently, the similarity between the visual features of the current frame and other frames processed by the object detection model is calculated, for example, using cosine similarity or Euclidean distance.

[0063] The above method effectively assesses the temporal continuity and stability of objects in the output video. Adopting this method improves the quality of the generated video, making it more aligned with the business needs of e-commerce users. This improvement enhances the user experience while watching the video and further increases purchase conversion rates.

[0064] S302. Based on the text information in the video frame of the output video and the difference between the text information in the adjacent video frame and the reference video frame, determine the text stability feature.

[0065] In this embodiment, the Chinese text and its corresponding confidence level can be extracted from each frame. Then, by comparing the text content, font size, and position coordinates of the Chinese text in adjacent frames with those in a reference frame, the variation range of these information is evaluated, thereby determining the text stability characteristics. If the Chinese text remains highly consistent across multiple frames, it indicates good stability of the Chinese text information; otherwise, there may be a problem.

[0066] S303. Process the output video based on a preset model to determine the creative rationality features; wherein, the preset model is used to determine the narrative fluency, creative evolution intensity, and semantic consistency of the content described in the output video.

[0067] In this embodiment, the output video can be processed based on a preset model to determine the scores corresponding to the narrative fluency, creative evolution intensity, and semantic consistency of the output video, and then the rationality of the creative idea can be determined by combining these three parameters.

[0068] The preset model may include a multimodal large language model (MLLM). In other embodiments, other large language models with the same functionality may also be used.

[0069] In this embodiment, the output video is divided into several segments or scenes and input into a preset model for reasoning. The preset model outputs three sub-scores: Narrative Coherence, Creative Evolution Strength, and Semantic Consistency. Narrative Coherence, Creative Evolution Strength, and Semantic Consistency reflect the output video's performance in terms of structural arrangement, creative development, and content logic, respectively. Finally, the scores corresponding to Narrative Coherence, Creative Evolution Strength, and Semantic Consistency can be combined according to their weights into an overall creative rationality score, which is the creative rationality feature.

[0070] S304. Determine the reward parameters based on the object consistency feature, the text stability feature, and the creative rationality feature.

[0071] In this embodiment, the reward parameters can be determined by weighted summation based on the object consistency feature, the text stability feature, and the creative rationality feature.

[0072] In this embodiment, the reward parameter is a core variable in the reinforcement learning optimization process. The reward parameter determines how the model adjusts its strategy during training to generate higher-quality videos. The inventors use object consistency features, text stability features, and creative rationality features as the main criteria to construct a comprehensive reward function, enabling the model to achieve a balance between product consistency, copywriting stability, and creative expression.

[0073] In this embodiment of the application, the reward parameter can be calculated using formula (3).

[0074] Formula (3) in, Indicates the reward parameter. Represents the consistency characteristic of objects. Indicates text stability characteristics. This indicates the rationality of the creative idea.

[0075] In this embodiment, reward parameters are constructed from three dimensions: object consistency, text stability, and creative rationality. This allows the model to receive multidimensional feedback during training, further enhancing its adaptability in e-commerce video generation tasks, ensuring that product display effects meet commercial requirements, and improving the quality of generated videos.

[0076] In this embodiment of the application, S301 shown can also be implemented by S3011 to S3013, which will be described in conjunction with the steps: S3011. Extract the object features of the sample object in each video frame and determine the first confidence level corresponding to the sample object.

[0077] In this embodiment, sample object regions can be extracted from video frames, then vectorized to determine object features, and each sample object region can be detected to determine a first confidence level.

[0078] In this embodiment, the product region is extracted from the video frame using Grounded SegmentAnything Model 2 (Grounded-SAM2), an adaptive segmentation model based on a visual-language model. Corresponding object features are then extracted from the product region. YOLO-World is then used to detect the product region and determine the first confidence level.

[0079] By introducing object features and a first confidence level, the quality and representativeness of sample objects in each video frame can be judged more accurately, thereby improving the accuracy of subsequent similarity calculations and providing a reliable basis for determining the final object consistency features.

[0080] S3012. Based on the object features of each video frame, the similarity between the object features of other video frames, and the first confidence level, determine the object similarity corresponding to each video frame.

[0081] In this embodiment, object similarity refers to the degree of matching between sample objects in the current video frame and those in other video frames in the feature space. Object similarity is typically calculated using methods such as cosine similarity, Euclidean distance, or Mahalanobis distance, combined with a first confidence level, to reflect the stability of sample objects in the current video frame over time.

[0082] For example, object similarity can be calculated using formula (4).

[0083] Formula (4) in, This represents the object similarity in the i-th video frame. This represents the first confidence level of the i-th video frame. Represents the object characteristics of other video frames. This represents the object features of the i-th video.

[0084] S3013. Based on the average object similarity of multiple video frames and the proportion of object similarities greater than a similarity threshold among the multiple object similarities, determine the object consistency feature.

[0085] In this embodiment, object consistency feature is an overall evaluation of the consistency of sample objects across video frames in the entire output video. The calculation of object consistency feature relies on two core indicators: the first core indicator is the mean of object similarity across all video frames, which reflects the overall level of consistency; the second core indicator is the proportion of video frames with similarity above a similarity threshold, which reflects the distribution of the number of highly similar video frames.

[0086] In this embodiment, the consistency characteristics of the object can be determined based on the product of the percentage raised to the power of X and a first predetermined multiple of the mean, plus the product of the mean and a second predetermined multiple.

[0087] The similarity threshold is a preset value used to distinguish between highly consistent and inconsistent video frames. Only video frames with object similarity higher than the similarity threshold are considered to have sufficient continuity, thus affecting the overall object consistency score. By using the mean object similarity and the proportion of video frames above the similarity threshold, the temporal stability of sample objects in the output video can be comprehensively characterized. The design of the mean object similarity and the proportion of video frames above the similarity threshold allows for the identification of local inconsistencies in sample objects within the output video and the assessment of its overall coherence, effectively supporting the high-quality display requirements of sample objects in the output video.

[0088] In this embodiment of the application, the object consistency feature can be calculated using formula (5).

[0089] Formula (5) in, The object consistency feature is represented by Q, which represents the mean, and P represents the proportion of the object similarity greater than the similarity threshold among the multiple object similarities.

[0090] In this embodiment, by introducing object features and confidence scores to calculate object similarity, and by combining information from multiple video frames to statistically calculate object consistency features, the overall consistency of goods in the video can be reflected more comprehensively and accurately, avoiding the bias caused by a single frame judgment, thereby improving the reliability of the reward signal, which in turn improves the performance of the video generation model, enabling the video generation model to generate high-quality videos.

[0091] In this embodiment of the application, S302 shown can also be implemented by S3021 to S3024, which will be described in conjunction with the steps: S3021. Extract the text features of the text information of each video frame and determine the second confidence level corresponding to the text information.

[0092] In this embodiment, text features refer to the structured representation of each frame of Chinese text extracted through optical character recognition, such as text content, font size, position coordinates, and layout style. Text features are used to characterize the visual presentation of text information in video frames. The second confidence level is a reliability assessment of the text recognition results in video frames by optical character recognition, typically a value between 0 and 1, with higher values ​​indicating more accurate recognition.

[0093] S3022. Based on the first difference value between the text features of each video frame and the text features of adjacent video frames, and the corresponding second confidence level, determine the cross-frame stability.

[0094] In this embodiment, the first difference value is the degree of difference in text features between the current video frame and adjacent video frames, used to measure the changes in text between consecutive frames. For example, if the position of text in a video frame shifts or the font changes, the first difference value will increase. Cross-frame stability is an index derived by comprehensively considering the first difference value and the second confidence level, used to reflect the continuity of text in the video sequence in the temporal dimension.

[0095] In this embodiment, a first sum of multiple first values ​​can be determined, and a first product of the mean of the second confidence levels can be obtained; wherein, the first value is determined based on the difference between a predetermined value and the first difference value; the cross-frame stability is determined based on the ratio of the first product to the second sum of multiple second confidence levels.

[0096] The cross-frame stability can be calculated using formula (6).

[0097] Formula (6) in, Indicates cross-frame stability. Indicates the first difference value. The text features representing the i-th video frame are... The text features representing the (i+1)th video frame are... Let represent the mean of the second confidence scores between the i-th video frame and the (i+1)-th video frame. This represents the sum of the second confidence scores of the i-th video frame and the (i+1)-th video frame.

[0098] S3023. Determine the text fidelity of the reference frame based on the second difference value between the text features of each video frame and the text features of the reference video frame, and the corresponding second confidence level.

[0099] In this embodiment, the reference video frame is a frame selected from the entire video sequence as a baseline, representing the ideal text display state in the video containing the reference video frame. The second difference value is the text feature difference between the current video frame and the reference video frame, used to measure whether the current video frame reproduces the text content in the reference video frame. The reference frame text fidelity is an index calculated based on the second difference value and the second confidence level, used to evaluate the text consistency between each frame in the video and the reference video frame.

[0100] In this embodiment, a second product of a second value and the corresponding second confidence level is determined; wherein the second value is determined based on the difference between a predetermined value and the second difference value; and the text fidelity of the reference frame is determined based on the ratio of a third sum of multiple second products to the number of multiple video frames.

[0101] The fidelity of the reference frame text can be calculated using formula (7).

[0102] Formula (7) in, Indicates the fidelity of the reference frame text. The number of video frames, Indicates the second difference value. Textual features representing reference video frames, The text features representing the i-th video frame are... This represents the second confidence level of the i-th video frame.

[0103] S3024. Based on the cross-frame stability and the text fidelity of the reference frame, determine the text stability feature.

[0104] In this embodiment of the application, text stability features can be determined based on the average of cross-frame stability and reference frame text fidelity.

[0105] Among them, text stability feature is the final evaluation index derived by comprehensively calculating cross-frame stability and reference frame text fidelity. Text stability feature is used to comprehensively reflect the overall stability of Chinese text in the video. Text stability feature not only focuses on the continuity of text in the temporal dimension, but also on the accuracy of text in the spatial dimension. Text stability feature is one of the important bases for measuring the quality of video generation.

[0106] In this embodiment, by analyzing the differences in text features between video frames and between the video frame and the reference frame, and by combining confidence level to evaluate the stability of text changes, the continuity and accuracy of Chinese copywriting in e-commerce videos can be effectively measured, thereby providing a clear direction for model optimization and thus obtaining a high-performance model to output high-quality video data.

[0107] In this embodiment of the application, S303 can also be implemented by S3031 to S3033, which will be described in conjunction with the steps: S3031. Based on the video content scene in the output video, the output video is divided into multiple sub-videos; wherein, the video frames in the same sub-video correspond to the same video content scene.

[0108] In this embodiment, video frames can be divided according to the video content scene in the output video, and video frames corresponding to the same video content scene can be divided into a sub-video to obtain multiple sub-video segments.

[0109] In this embodiment, the output video can also be divided into multiple segments according to the timeline to obtain multiple sub-videos.

[0110] S3032. After dividing the preset model into multiple segments, input the segments into the preset model to determine narrative fluency information, creative evolution intensity information, and semantic consistency information.

[0111] In this embodiment, each sub-video segment is fed into the model as an independent input for analysis, thereby extracting narrative fluency information, creative evolution intensity information, and semantic consistency information from each sub-video.

[0112] Narrative fluency is used to measure whether the plot development in the output video is natural and coherent. For example, in e-commerce short videos, product demonstrations should progress gradually, from appearance introduction to function demonstration to user reviews, forming a logically clear narrative chain. If the output video jumps frequently and lacks obvious thematic changes, it indicates poor narrative fluency.

[0113] Among these, the creative evolution intensity information refers to the degree of innovation and rhythmic change in the creative expression of the output video; that is, whether the creativity gradually strengthens over time, rather than being monotonously repetitive. Creative evolution intensity information is particularly important in e-commerce scenarios because the diversity of creativity can attract user attention and promote purchasing behavior.

[0114] Semantic consistency information indicates the strength of semantic connections between segments in the output video. For example, do consecutive frames revolve around the same selling point, or are there obvious semantic breaks? By evaluating the various dimensions of semantic consistency information, it is possible to determine whether the output video possesses strong business logic and visual appeal.

[0115] S3033. Based on the narrative fluency information, the creative evolution intensity information, and the semantic consistency information, determine the creative rationality feature.

[0116] In this embodiment, the three sub-features are weighted and fused to obtain an overall score (creative rationality feature) that reflects the creative rationality of the output video. These three indicators together construct an evaluation system for creative rationality, enabling the model to not only focus on the quality of individual frames but also understand the narrative logic and creative value of the entire output video.

[0117] For example, the creative rationality feature can be calculated using formula (8).

[0118] Formula (8) in, Indicates the rationality of the creative idea. Indicates information about narrative flow, Indicates the intensity of creative evolution. It represents semantic consistency information.

[0119] In this embodiment, by inputting video segments into a preset model and extracting multiple key dimension information, the creative expressiveness of the video can be comprehensively evaluated, enabling the model to have a stronger ability to understand business logic during training, thereby generating more attractive video content that meets the needs of e-commerce.

[0120] The following example illustrates the process of training a video generation model. Please refer to... Figure 4 : This application proposes an optimization method for generating short videos. It constructs a dedicated reward system and combines it with reinforcement learning to optimize the video generation model (WAN2.2) online. The overall technical framework of this application is shown in Figure 4, comprising two main stages (SFT + RL) and three reward parameters (ICScore, VTScore, and CCScore). The following provides a detailed description of the overall process and each step.

[0121] As shown in Figure 4, the technical solution of this application consists of the following parts: Phase 1: Supervised Fine-tuning (SFT) Using e-commerce video data with sample objects, text descriptions, and target videos, we perform domain adaptation on existing image-video generation models to obtain a model with basic video generation capabilities (the second video generation model). Phase Two: GRPO-based Reinforcement Learning Optimization (RL) The video generation process is represented as a Markov Decision Process (MDP). The second video generation model is optimized online using group relative policy optimization, enabling the model to improve product consistency, Chinese text stability, and creative rationality based on reward signals.

[0122] Reward system (ICScore + VTScore + CCScore) To meet the commercial requirements of short videos, this application designs three types of awards: Product Consistency Award (ICScore): evaluates the stability of product appearance across frames; Chinese Text Stability Award (VTScore): evaluates the accuracy and temporal consistency of Chinese text rendering; and Creative Rationality Award (CCScore): evaluates the visual narrative logic of creative evolution in the video.

[0123] Please see Figure 5 The above is a flowchart illustrating the video generation method provided in the embodiments of this application. Figure 1 , will combine Figure 5 The steps shown are explained below: S401. Obtain the current object graph and current description information.

[0124] In this embodiment, when it is necessary to generate a corresponding short video for the current object, the corresponding current object image and the current description information of the object can be obtained. The current description information may include the current object's product name, functional selling points, brand introduction, performance parameters, etc.

[0125] The video generation methods provided in the embodiments of this application can be executed by electronic devices, which may be devices with computing capabilities such as computers, servers, or cloud platforms. The video generation methods in the embodiments of this application can be executed independently by any of the aforementioned devices, or they can be executed collaboratively by multiple devices.

[0126] S402. Input the current object image and the current description information into the third video generation model to obtain the corresponding target video.

[0127] In this embodiment, the current object graph and the current description information are input into the third video generation model to obtain the corresponding target video.

[0128] The third video generation model is obtained by inputting sample object graphs and sample description information into a second video generation model to obtain multiple output videos, and by determining the loss of the second video generation model based on one or more of the object consistency features, text stability features, and creativity and rationality features in each output video, and updating the parameters of the second video generation model based on the loss; the second video generation model is obtained by supervised fine-tuning by inputting sample object graphs, sample description information, and sample videos into a first video generation model.

[0129] In this embodiment, a two-stage training mechanism combining supervised fine-tuning and reward-parameter-based reinforcement learning effectively improves the adaptability and generation quality of the video generation model in e-commerce scenarios, enabling the generation of higher-quality e-commerce short videos that better meet commercial needs. Furthermore, by combining three types of reward signals—object consistency, text stability, and creative rationality—online optimization of the video generation model significantly improves the cross-frame consistency of product appearance, the clarity and stability of Chinese text, and the commercial logic of creative expression in e-commerce short videos. This effectively solves the problems of unstable, inaccurate, and unreasonable video content in existing technologies, thereby improving the quality of generated videos.

[0130] Please see Figure 6 This is a schematic diagram of the structure of the video generation model training device provided in the embodiments of this application.

[0131] This application provides a video generation model training device 600, including: a first data acquisition unit 601, a first training unit 602, and a second training unit 603.

[0132] The first data acquisition unit 601 is used to acquire multiple sample data; wherein, the sample data includes: sample object image, sample description information and sample video; The first training unit 602 is used to input the sample video into the first video generation model for supervised fine-tuning to obtain the second video generation model; The second training unit 603 is used to input the sample object graph and the sample description information into the second video generation model to obtain multiple output videos, and to determine the loss of the second video generation model based on one or more of the object consistency feature, text stability feature, and creative rationality feature in each output video, and to update the parameters of the second video generation model based on the loss to obtain a third video generation model. In this embodiment of the application, the second training unit 603 in the video generation model training device 600 is used to input the sample object graph, the sample description information, the latent variables and the current policy information for the current state into the second video generation model, and generate multiple output videos based on the Markov decision process, as well as the next policy information for the next state during the generation process; wherein, the latent variables are determined based on random backsampling; The reward parameters are determined based on the object consistency features of the sample objects included in the output video, the text stability features of the text information, and the creative rationality features of the content described in the output video. The loss of the second video generation model is determined based on the reward parameters, the number of multiple output videos, and the ratio of each next policy information to the corresponding current policy information.

[0133] In this embodiment of the application, the second training unit 603 in the video generation model training device 600 is used to determine the object consistency feature based on the similarity between the sample object in each video frame of the output video and the sample object in other video frames; wherein, the other video frames are any other frame in the output video besides the video frame. Based on the text information in the video frames of the output video, and the difference between the text information in the adjacent video frames and the reference video frames, the text stability feature is determined; The output video is processed based on a preset model to determine the rationality features of the creative concept; wherein, the preset model is used to determine the narrative fluency, creative evolution intensity, and semantic consistency of the content described in the output video. The reward parameters are determined based on the object consistency feature, the text stability feature, and the creative rationality feature.

[0134] In this embodiment of the application, the second training unit 603 in the video generation model training device 600 is used to extract the object features of the sample object in each video frame and determine the first confidence level corresponding to the sample object. Based on the object features of each video frame, the similarity between the object features of other video frames, and the first confidence level, the object similarity corresponding to each video frame is determined. The object consistency feature is determined based on the average object similarity of multiple video frames and the proportion of object similarities greater than a similarity threshold among the multiple object similarities.

[0135] In this embodiment of the application, the second training unit 603 in the video generation model training device 600 is used to extract the text features of the text information of each video frame and determine the second confidence level corresponding to the text information. Based on the first difference value between the text features of each video frame and the text features of adjacent video frames, and the corresponding second confidence level, cross-frame stability is determined. The text fidelity of the reference frame is determined based on the second difference value between the text features of each video frame and the text features of the reference video frame, and the corresponding second confidence level. The text stability features are determined based on the cross-frame stability and the text fidelity of the reference frame.

[0136] In this embodiment of the application, the second training unit 603 in the video generation model training device 600 is used to divide the output video into multiple sub-videos based on the video content scene in the output video; wherein, the video content scene corresponding to the video frame in the same sub-video is the same. Multiple sub-video segments are input into the preset model to determine narrative fluency information, creative evolution intensity information, and semantic consistency information; Based on the narrative fluency information, the creative evolution intensity information, and the semantic consistency information, the creative rationality features are determined.

[0137] It should be noted that, in the embodiments of this application, if the above-described video generation model training method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a video generation model training device (which may be a personal computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0138] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a first processor, implements the steps in the method of the video generation model training device.

[0139] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0140] It should be noted that, Figure 7 A hardware entity schematic diagram of the first electronic device provided in the embodiments of this application, such as... Figure 7 As shown, this application embodiment provides a first electronic device 700, including a first memory 702 and a first processor 701. The first memory 702 stores a computer program that can run on the first processor 701. When the first processor 701 executes the program, it implements the steps in the above-described method, wherein; The first processor 701 typically controls the overall operation of the first electronic device 700.

[0141] The first memory 702 is configured to store instructions and applications executable by the first processor 701, and can also cache data to be processed or already processed by the first processor 701 and the various modules in the first electronic device 700 (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0142] Correspondingly, this application embodiment also provides a computer program product, including a computer program that can be executed by a first processor 701 of a first electronic device 700 to complete the steps in the method of the video generation model training device 600.

[0143] Please see Figure 8 This is a schematic diagram of the structure of the video generation device provided in the embodiments of this application.

[0144] This application also provides a video generation apparatus 800, including: a second data acquisition unit 801 and a generation unit 802.

[0145] The second data acquisition unit 801 is used to acquire the current object graph and the current description information; The generation unit 802 is used to input the current object image and the current description information into the third video generation model to obtain the corresponding target video; The third video generation model is obtained by inputting sample object graphs and sample description information into the second video generation model to obtain multiple output videos, and by determining the loss of the second video generation model based on one or more of the object consistency features, text stability features, and creative rationality features in each output video, and updating the parameters of the second video generation model based on the loss; the second video generation model is obtained by supervising the fine-tuning of the first video generation model by inputting sample videos.

[0146] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a second processor, implements the steps in the method on one side of the video generation apparatus.

[0147] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0148] It should be noted that, Figure 9 This is a schematic diagram of a hardware entity of the second electronic device provided in an embodiment of this application, such as... Figure 9 As shown, this application embodiment provides a second electronic device 900, including a second memory 902 and a second processor 901. The second memory 902 stores a computer program that can run on the second processor 901. When the second processor 901 executes the program, it implements the steps in the above-described method, wherein; The second processor 901 typically controls the overall operation of the second electronic device 900.

[0149] The second memory 902 is configured to store instructions and applications executable by the second processor 901, and can also cache data to be processed or already processed by the second processor 901 and the various modules in the second electronic device 900 (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0150] Correspondingly, this application embodiment also provides a computer program product, including a computer program that can be executed by a second processor 901 of a second electronic device 900 to complete the steps in the method of the video generation device 800.

[0151] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for training a video generation model, characterized in that, include: Acquire multiple sample data; wherein, the sample data includes: sample object image, sample description information, and sample video; The sample video is input into the first video generation model for supervised fine-tuning to obtain the second video generation model; The sample object graph and the sample description information are input into the second video generation model to obtain multiple output videos. Based on one or more of the object consistency features, text stability features, and creative rationality features in each output video, the loss of the second video generation model is determined. The parameters of the second video generation model are updated based on the loss to obtain a third video generation model.

2. The video generation model training method according to claim 1, characterized in that, The process involves inputting the sample object graph and the sample description information into the second video generation model to obtain multiple output videos, and determining the loss of the second video generation model based on one or more of the object consistency feature, text stability feature, and creative rationality feature in each output video, including: The sample object graph, the sample description information, the latent variables, and the current policy information for the current state are input into the second video generation model, and multiple output videos are generated based on the Markov decision process, as well as the next policy information for the next state during the generation process; wherein, the latent variables are determined based on random backsampling; The reward parameters are determined based on the object consistency features of the sample objects included in the output video, the text stability features of the text information, and the creative rationality features of the content described in the output video. The loss of the second video generation model is determined based on the reward parameters, the number of multiple output videos, and the ratio of each next policy information to the corresponding current policy information.

3. The video generation model training method according to claim 2, characterized in that, The method for determining reward parameters based on the object consistency features of the sample objects included in the output video, the text stability features of the text information, and the creative rationality features of the description content of the output video includes: The object consistency feature is determined based on the similarity between the sample object in each video frame of the output video and the sample object in other video frames; wherein, the other video frames are any other frame in the output video besides the aforementioned video frame. Based on the text information in the video frames of the output video, and the difference between the text information in the adjacent video frames and the reference video frames, the text stability feature is determined; The output video is processed based on a preset model to determine the rationality features of the creative concept; wherein, the preset model is used to determine the narrative fluency, creative evolution intensity, and semantic consistency of the content described in the output video. The reward parameters are determined based on the object consistency feature, the text stability feature, and the creative rationality feature.

4. The video generation model training method according to claim 3, characterized in that, The determination of the object consistency feature based on the similarity between the sample object in each video frame of the output video and the sample object in other video frames includes: Extract the object features of the sample object in each video frame and determine the first confidence level corresponding to the sample object; Based on the object features of each video frame, the similarity between the object features of other video frames, and the first confidence level, the object similarity corresponding to each video frame is determined. The object consistency feature is determined based on the average object similarity of multiple video frames and the proportion of object similarities greater than a similarity threshold among the multiple object similarities.

5. The video generation model training method according to claim 3, characterized in that, The determination of the text stability feature based on the difference between the text information in the video frame of the output video and the text information in adjacent video frames and reference video frames includes: Extract the text features of the text information of each video frame and determine the second confidence level corresponding to the text information; Based on the first difference value between the text features of each video frame and the text features of adjacent video frames, and the corresponding second confidence level, cross-frame stability is determined. The text fidelity of the reference frame is determined based on the second difference value between the text features of each video frame and the text features of the reference video frame, and the corresponding second confidence level. The text stability features are determined based on the cross-frame stability and the text fidelity of the reference frame.

6. The video generation model training method according to claim 3, characterized in that, The process of processing the output video based on a preset model to determine the creative rationality features includes: Based on the video content scene in the output video, the output video is divided into multiple sub-videos; wherein, the video frames in the same sub-video correspond to the same video content scene; Multiple sub-video segments are input into the preset model to determine narrative fluency information, creative evolution intensity information, and semantic consistency information; Based on the narrative fluency information, the creative evolution intensity information, and the semantic consistency information, the creative rationality features are determined.

7. A video generation method, characterized in that, include: Get the current object graph and current description information; The current object image and the current description information are input into the third video generation model to obtain the corresponding target video; The third video generation model is obtained by inputting sample object graphs and sample description information into the second video generation model to obtain multiple output videos, and by determining the loss of the second video generation model based on one or more of the object consistency features, text stability features, and creative rationality features in each output video, and updating the parameters of the second video generation model based on the loss. The second video generation model is obtained by supervising the fine-tuning of the first video generation model by inputting sample videos.

8. A video generation model training device, characterized in that, include: The first data acquisition unit is used to acquire multiple sample data; wherein, the sample data includes: sample object image, sample description information and sample video; The first training unit is used to input the sample video into the first video generation model for supervised fine-tuning to obtain the second video generation model. The second training unit is used to input the sample object graph and the sample description information into the second video generation model to obtain multiple output videos, and to determine the loss of the second video generation model based on one or more of the object consistency feature, text stability feature and creative rationality feature in each output video, and to update the parameters of the second video generation model based on the loss to obtain a third video generation model.

9. A video generation apparatus, characterized in that, include: The second data acquisition unit is used to acquire the current object graph and current description information; The generation unit is used to input the current object image and the current description information into the third video generation model to obtain the corresponding target video; The third video generation model is obtained by inputting sample object graphs and sample description information into the second video generation model to obtain multiple output videos, and by determining the loss of the second video generation model based on one or more of the object consistency features, text stability features, and creative rationality features in each output video, and updating the parameters of the second video generation model based on the loss. The second video generation model is obtained by supervising the fine-tuning of the first video generation model by inputting sample videos.

10. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the computer program to implement the steps of the method of any one of claims 1 to 6, or to implement the steps of the method of claim 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6, or the steps of the method according to claim 7.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6, or the steps of the method according to claim 7.

Images