Model training method, video generation method, electronic device, and storage medium

Abstract

The application provides a model training method, a video generation method, an electronic device and a storage medium, and relates to the technical field of large models and data processing. The method comprises the following steps: obtaining a training prompt text; performing video generation processing on the training prompt text by using an initial video generation model to obtain a plurality of candidate videos; performing video preference evaluation on the plurality of candidate videos from a plurality of dimensions to obtain an evaluation result, wherein the plurality of dimensions at least include a dynamic visual dimension, and the evaluation result is used at least for evaluating the motion amplitude of a plurality of motion objects associated with the dynamic visual dimension; and training the initial video generation model based on the evaluation result to generate a target video generation model, wherein the target video generation model is used for performing video generation processing on an input text to obtain a target video. The application solves the technical problem of low motion richness and poor authenticity of the output video of the video generation method provided in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model technology and data processing, in particular to a model training method, a video generation method, an electronic device and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, especially the application of deep learning in the field of vision, the generation technology of text to video has become an important tool for media content creation. According to the text description, the video segment is automatically generated, which greatly enriches the digital content ecology. However, in the pursuit of high quality and vitality of the generated video, the traditional video generation model faces a core challenge, that is, how to ensure that the generated results are consistent with human subjective preferences. In the related technology, through the Direct Preference Optimization (DPO) framework, the preference signal provided by the reward model is used to guide the optimization of the video generation model. However, in the related technology, the evaluation of video quality is oversimplified when optimizing the video generation model, and the inherent conflict between the motion amplitude and the visual clarity is not effectively handled, which directly leads to the video generation model tending to generate low dynamic content to avoid risks and obtain a higher comprehensive score, thereby causing the motion richness of the output video of the video generation model to be low and the reality to be poor, which further affects the user's visual interaction experience.

[0003] At present, there is no effective solution to the above problems. SUMMARY

[0004] The embodiments of the present application provide a model training method, a video generation method, an electronic device and a storage medium to at least solve the technical problem of low motion richness and poor reality of the output video of the video generation method provided in the related technology.

[0005] According to an aspect of an embodiment of the present application, a model training method is provided, comprising: obtaining a training prompt text; using an initial video generation model to perform video generation processing on the training prompt text to obtain a plurality of candidate videos; performing video preference evaluation on the plurality of candidate videos from a plurality of dimensions to obtain an evaluation result, wherein the plurality of dimensions at least include a dynamic visual dimension, and the evaluation result is at least used to evaluate the motion amplitude of a plurality of motion objects associated with the dynamic visual dimension; training the initial video generation model based on the evaluation result to generate a target video generation model, wherein the target video generation model is used to perform video generation processing on an input text to obtain a target video.

[0006] According to another aspect of the embodiments of the present application, a video generation method is further provided, including: obtaining input text; performing video generation processing on the input text by using a target video generation model to obtain a target video; wherein the target video generation model is generated according to the model training method of any one of the embodiments of the present application.

[0007] According to another aspect of the embodiments of the present application, a video generation method is further provided, including: obtaining a film and television content creation text; performing video generation processing on the film and television content creation text by using a target video generation model to obtain a film and television content creation video; wherein the target video generation model is generated according to the model training method of any one of the embodiments of the present application.

[0008] According to another aspect of the embodiments of the present application, a video generation method is further provided, including: obtaining a video generation request through a first application programming interface, wherein the request data carried in the video generation request includes input text; returning a video generation response through a second application programming interface, wherein the response data carried in the video generation response includes a target video, the target video being obtained by performing video generation processing on the input text by using a target video generation model, and the target video generation model being generated according to the model training method of any one of the embodiments of the present application.

[0009] According to another aspect of the embodiments of the present application, a video generation method is further provided, including: obtaining a current input video generation dialogue request, wherein the request data carried in the video generation dialogue request includes input text; in response to the video generation dialogue request, returning a video generation dialogue reply, wherein the information carried in the video generation dialogue reply includes a target video, the target video being obtained by performing video generation processing on the input text by using a target video generation model, and the target video generation model being generated according to the model training method of any one of the embodiments of the present application; and playing the target video in a graphical user interface.

[0010] According to another aspect of the embodiments of the present application, a video generation method is further provided, including: in response to an input instruction acting on an operation interface, displaying input text on the operation interface; in response to a processing instruction acting on the operation interface, playing a target video on the operation interface; wherein the target video is obtained by performing video generation processing on the input text by using a target video generation model, and the target video generation model is generated according to the model training method of any one of the embodiments of the present application.

[0011] According to another aspect of the embodiments of the present application, a video generation system is provided, comprising: a client configured to send input text; a server connected to the client, configured to perform video generation processing on the input text by using a target video generation model, to obtain a target video, wherein the target video generation model is generated according to the model training method in any of the embodiments of the present application; and the client is further configured to output the target video.

[0012] According to another aspect of the embodiments of the present application, a computing device is provided, comprising: a memory storing an executable program; and a processor configured to execute the program, wherein the program performs the method in any of the embodiments of the present application when executed.

[0013] According to another aspect of the embodiments of the present application, an electronic device is provided, comprising: a memory storing an executable program; and a processor connected to the memory through a bus, configured to execute the program, wherein the program performs the method in any of the embodiments of the present application when executed.

[0014] According to another aspect of the embodiments of the present application, a computer readable storage medium is provided, comprising a stored executable program, wherein the executable program controls the device where the computer readable storage medium is located to perform the method in any of the embodiments of the present application when executed.

[0015] According to another aspect of the embodiments of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method in any of the embodiments of the present application.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, comprising a non-volatile computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the method in any of the embodiments of the present application.

[0017] According to another aspect of the embodiments of the present application, a computer program is provided, which, when executed by a processor, implements the method in any of the embodiments of the present application.

[0018] In the embodiment of the present application, by obtaining the training prompt text, and then using the initial video generation model to generate video for the training prompt text, a plurality of candidate videos are obtained, then the plurality of candidate videos are evaluated from multiple dimensions, and the evaluation results are obtained, and finally the initial video generation model is trained based on the evaluation results to generate a target video generation model, which is used to generate a target video for the input text. Thus, by introducing a multi-dimensional video preference evaluation mechanism, especially the evaluation of dynamic visual dimension, the deficiency of insufficient motion performance caused by single-dimensional evaluation in the related art is overcome. By evaluating the plurality of candidate videos generated by the training prompt text from multiple dimensions and in all directions, the motion amplitude in the candidate video can be accurately identified and quantified, and the optimization strategy is adjusted to ensure that the initial video generation model can improve the motion richness and realism of the generated video during the training process. In addition, the model training process based on the evaluation results promotes the capture and reproduction of dynamic visual elements by the initial video generation model, significantly improves the liveliness and realism of the target video, balances and optimizes the motion performance and visual quality, generates video content that is more in line with human aesthetic preferences, and thus solves the technical problems of low motion richness and poor realism of the output video of the video generation method provided in the related art.

[0019] The above general description and the following detailed description are intended to illustrate and explain the present application, and do not constitute a limitation on the present application. BRIEF DESCRIPTION OF DRAWINGS

[0020] The drawings described herein are intended to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are intended to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is an application scenario diagram of a model training method according to an embodiment of the present application;

[0022] Figure 2 is a flowchart of a model training method according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a model training method according to an embodiment of the present application;

[0024] Figure 4 is a flowchart of a video generation method according to an embodiment of the present application;

[0025] Figure 5 is a flowchart of another video generation method according to an embodiment of the present application;

[0026] Figure 6is a flowchart of still another video generation method according to an embodiment of the application;

[0027] Figure 7 is a flowchart of still another video generation method according to an embodiment of the application;

[0028] Figure 8 is a flowchart of still another video generation method according to an embodiment of the application;

[0029] Figure 9 is a structural block diagram of a model training apparatus according to an embodiment of the application;

[0030] Figure 10 is a structural block diagram of a video generation apparatus according to an embodiment of the application;

[0031] Figure 11 is a structural block diagram of still another video generation apparatus according to an embodiment of the application;

[0032] Figure 12 is a structural block diagram of still another video generation apparatus according to an embodiment of the application;

[0033] Figure 13 is a structural block diagram of still another video generation apparatus according to an embodiment of the application;

[0034] Figure 14 is a structural block diagram of still another video generation apparatus according to an embodiment of the application;

[0035] Figure 15 is a structural block diagram of a computing device according to an embodiment of the application;

[0036] Figure 16 is a structural block diagram of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0037] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the embodiments described below are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in other orders. Other orders here refer to orders other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0039] The technical solutions provided by the present application are mainly implemented by using large model technology. Here, the large model refers to a deep learning model with a large number of model parameters, which can typically include hundreds of millions, billions, tens of billions, hundreds of billions, or even tens of billions of model parameters. The large model can also be referred to as a foundation model. Through large-scale unlabeled corpus pre-training, a pre-trained model with hundreds of millions of parameters is output. Such a model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language models (LLM) and multi-modal pre-training models.

[0040] It should be noted that in actual application, the pre-trained model can be fine-tuned by a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing, etc. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), image generation, etc. It can also be widely used in text-based sentiment classification, text summarization generation, machine translation, etc. Natural language processing tasks. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiments of the present application, the data processing by the target video generation model in the video generation scenario is taken as an example for explanation and description.

[0041] First, some of the nouns or terms appearing in the description of the embodiments of the present application are applicable to the following explanations:

[0042] Human video preference: Human's subjective evaluation and personal preference judgment on video content based on subjective feelings combined with objective rules, obtaining an evaluation level or score for a video, or preference judgment and ranking results for multiple videos.

[0043] Human video preference alignment: A task that aligns the video generated by the video generation model with human preference in terms of aesthetics, video quality, harmlessness, and whether the video content conforms to physical laws.

[0044] Text-to-video generation: Given a text description, use deep learning models to generate a video segment with the same content as the text description.

[0045] Direct preference optimization algorithm (DPO): A method based on human preference optimization model, which does not rely on explicit additional reward modeling.

[0046] Agent model / policy model: In the process of reinforcement learning training, the agent is the subject model that makes decisions, learns and completes the final task goal by interacting with the environment.

[0047] Reward model: A model or function used in reinforcement learning training process to provide the agent with the reward value of the current state, guiding the optimization direction of the policy model.

[0048] Reward hacking: In the process of using reinforcement learning strategy to optimize the model, the agent uses the defects or ambiguities in the reward model to obtain high rewards without truly learning or completing the expected task.

[0049] With the continuous evolution of text-to-video generation technology, it has made significant progress in semantic accuracy and visual quality, and the generated video segments are increasingly realistic and rich. However, the core challenge faced by this technology is still how to ensure that the video generation results can accurately align with human subjective preferences. Currently, the mainstream video preference alignment strategy tends to be based on preference learning, optimizing the video generation model through the preference signals provided by the reward model. Although preference learning based on preference learning improves the quality of generated videos to some extent, its limitations lie in simplifying human preferences into a single scalar score, thereby ignoring the multidimensional complexity of video content quality, especially in key dimensions such as aesthetics, motion coherence, and consistency.

[0050] Through experimental research, it is found that the motion-related dimensions, including but not limited to object motion and camera motion, have a significant negative correlation with other important dimensions such as visual clarity and composition stability. Therefore, the more dynamic a video is, the lower the score in static visual quality tends to be. This reveals a systematic bias in the reward model during optimization, which prefers videos with smaller motion amplitude and more stable composition. This tendency severely suppresses the motion expressiveness of the generated content, greatly reducing the vitality and vitality of the generated video, thereby limiting its value in virtual reality, film special effects, sports live broadcast and other application scenarios that highly rely on dynamic performance.

[0051] Specifically, although the related art has attempted to optimize the generation model using text-image alignment scores and improve the stability of alignment by weighting the confidence of the preferred sample, the common blind spot of both is the failure to consider the coupling relationship between the dimensions of video quality, especially the internal relationship between motion amplitude and visual clarity, aesthetic quality. This leads to the formation of an implicit preference for low-motion-amplitude videos during the training process of the generation model, i.e., the "reward hacking" phenomenon. The video generation model sacrifices dynamic richness to achieve high scores in other dimensions, resulting in a video that, although still of acceptable quality, lacks dynamic performance and cannot fully meet users' demand for high-dynamic, realistic video content, restricting the diversity and appeal of generated videos in practical applications. Therefore, there is an urgent need for a preference alignment mechanism that can actively identify and correct dimensional bias to promote the balance between motion expressiveness and overall visual quality and improve the practicality and user experience of video generation technology.

[0052] According to an embodiment of the present application, a model training method is provided. The steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0053] Considering that the model parameter amount of a large model is large, and the operation resources of a mobile terminal are limited. The above-mentioned method provided by the embodiments of the present application can be applied to the application scenario as shown in Figure 1 The above-mentioned method provided by the embodiments of the present application can be applied to the application scenario as shown in Figure 1In the illustrated application scenario, the large model is deployed in a server 10. The server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client devices 20 here can include, but are not limited to, smartphones, tablet computers, notebook computers, palm computers, personal computers, smart home devices, vehicle-mounted devices, and the like. The client devices 20 can interact with the user through a graphical user interface to call the large model, and thus implement the method provided in the embodiments of the present application.

[0054] In the embodiments of the present application, the system composed of the client device and the server can perform the following steps: the client device is configured to send an input text; the server connected to the client device is configured to perform video generation processing on the input text by using a target video generation model, to obtain a target video, wherein the target video generation model is generated according to the model training method of any one of the embodiments of the present application; and the client device is further configured to output the target video. In the training process of the target video generation model, a training prompt text is obtained, and then an initial video generation model is used to perform video generation processing on the training prompt text to obtain a plurality of candidate videos, and then the plurality of candidate videos are evaluated from multiple dimensions to obtain an evaluation result, and finally the initial video generation model is trained based on the evaluation result to generate the target video generation model, which is used to perform video generation processing on the input text to obtain the target video. Thus, by introducing a multi-dimensional video preference evaluation mechanism, especially the evaluation of the dynamic visual dimension, the defect of insufficient motion performance caused by single-dimensional evaluation in the related art is overcome.

[0055] With the rapid development of high-performance computing units, in other application scenarios, the above-mentioned method provided in the embodiments of the present application can also be applied to a model all-in-one machine. In one optional embodiment, a plurality of models are built-in in the model all-in-one machine. A user can select one model for adjustment according to needs to obtain a model of the user. Thus, the high-performance computing unit built-in in the model all-in-one machine can directly call the adjusted model to execute the above-mentioned method provided in the embodiments of the present application. In another optional embodiment, a trained model is built-in in the large model all-in-one machine, so that the high-performance computing unit built-in in the model all-in-one machine can directly call the model to execute the above-mentioned method provided in the embodiments of the present application.

[0056] Further, when the user needs to train his own model, he can also upload his own data set through the client, which is sent to the server, so that the server can adjust the pre-trained model with the data set to obtain the user's own model, and then deploy it to the production environment. In order to facilitate the user's adjustment requirement of the model, the server can provide complete adjustment tools, development frameworks and processes, and can support multiple adjustment strategies, so that the adjusted model can better adapt to different field applications and realize high customization.

[0057] In the above running environment, the present application provides a model training method as shown in Figure 2 Figure 2 is a flowchart of a model training method according to an embodiment of the present application. As shown in Figure 2 , the method can include the following steps:

[0058] Step S21, obtaining a training prompt text;

[0059] Step S22, using an initial video generation model to perform video generation processing on the training prompt text to obtain a plurality of candidate videos;

[0060] Step S23, performing video preference evaluation on the plurality of candidate videos from multiple dimensions to obtain an evaluation result, wherein the multiple dimensions at least include a dynamic visual dimension, and the evaluation result is at least used to evaluate the motion amplitude of a plurality of motion objects associated with the dynamic visual dimension;

[0061] Step S24, training the initial video generation model based on the evaluation result to generate a target video generation model, wherein the target video generation model is used to perform video generation processing on an input text to obtain a target video.

[0062] ​In the text-to-video generation technology, the training prompt text described above can be a text description used to guide the initial video generation model to learn to generate specific content videos. The training prompt text usually contains key information such as scenes, actions, objects, etc. that the target video can contain, aiming to let the initial video generation model understand and generate video clips that match it. For example, the training prompt text can be: "a group of lions hunting on the savannah, showing the cooperation of the lion group and the process of chasing prey". Each training prompt text can describe a scene or event in detail, including but not limited to background, subject, action, and event details, providing a clear goal for the initial video generation model. The initial video generation model will generate video clips that match the prompt according to the training prompt text, so as to continuously adjust during the training process, in order to achieve higher quality and more human preference video generation effect. For example, under the prompt of "lion hunting", the video generation model should generate a video containing the dynamic running of the lion group, the hunting action, and the details of the savannah environment, rather than just static lion images or savannah scenery.

[0063] The way to obtain the training prompt text is diverse, mainly depending on the application scenario and training goal of the text-to-video generation model. The training prompt text can come from user input, data set extraction, or a combination of network extraction and automatic summary generation, aiming to provide the initial video generation model with rich, specific and human preference-aligned text descriptions to generate video content with higher dynamic expression and visual quality.

[0064] The initial video generation model described above is a model used to generate candidate videos at the beginning of training. The initial video generation model can be based on a pre-trained architecture, but has not been fine-tuned for specific preferences or quality standards, so there is room for improvement in the quality and preference alignment of the generated content.

[0065] When inputting the training prompt text into the initial video generation model for video generation processing, the training prompt text can be converted into a form that the initial video generation model can understand, and then the initial video generation model can be used to construct a video frame sequence based on the conversion result, and finally the video frame sequence is synthesized into a coherent video clip, i.e. multiple candidate videos are obtained. The candidate videos reflect the different understandings and generation attempts of the initial video generation model for the same training prompt text, providing diversity for subsequent preference evaluation and model training.

[0066] After obtaining the plurality of candidate videos, video preference evaluation is performed on the plurality of candidate videos from multiple dimensions to obtain evaluation results, and the multiple dimensions at least include a dynamic visual dimension. The above evaluation results can be quantitative scores obtained by scoring the plurality of candidate videos on each dimension by a reward model, and at least used to evaluate the motion amplitude of the plurality of moving objects associated with the dynamic visual dimension. The above dynamic visual dimension can represent a dimension for evaluating the motion amplitude and motion coherence of objects and cameras in the candidate video. The dynamic visual dimension is crucial for measuring the dynamic expressiveness of the video. A high score of the dynamic visual dimension means that the candidate video exhibits rich and natural motion amplitude.

[0067] The above multiple dimensions can further include but are not limited to an aesthetic quality dimension, a motion coherence dimension, a text-image alignment dimension, a definition and detail dimension, a diversity and creativity dimension, a reality and rationality dimension, and a sound and music dimension, etc. The aesthetic quality dimension can evaluate whether the composition, color, light and shadow effect of the candidate video conforms to the aesthetic principle, and whether it can attract the attention of the audience and evoke emotional resonance. The motion coherence dimension can evaluate whether the transition between video frames of the candidate video is natural and smooth, and whether the motion in the candidate video is coherent, avoiding abrupt or discordant visual effects. The text-image alignment dimension language tests whether the video content accurately reflects the text description, including whether the scene, object, action, etc. elements are consistent with the training prompt text, and whether the candidate video conveys the emotion and atmosphere in the text. The definition and detail dimension can check the definition, texture detail, resolution of the video frame, to ensure that the generated content is clear and rich in vision, avoiding problems such as blurring or pixelation. The diversity and creativity dimension can evaluate the diversity of video content, to ensure that the generated candidate video not only matches the text, but also has a unique perspective and creative expression, avoiding repetitive or overly templated content. The reality and rationality dimension can check whether the physical laws and behavior logic in the candidate video are reasonable, and whether the candidate video can reflect the behavior and phenomenon of the real world, avoiding scenes that violate common sense or physical rules. For videos with sound, the sound and music dimension can evaluate whether the audio content matches the video visual content, including the selection of background music, the use of sound effects, and the overall sound and picture synchronization effect.

[0068] By performing preference evaluation on multiple dimensions, the evaluation results obtained can provide a comprehensive perspective, helping the initial video generation model to improve other visual quality aspects such as the definition of the picture, the aesthetic of the composition, the reality and appropriateness of the content, while maintaining dynamic expressiveness, thereby generating video content of higher quality and more in line with human comprehensive preferences.

[0069] Further, based on the evaluation result, the initial video generation model is trained to generate a target video generation model, which aims to generate a target video that closely matches the training prompt text and meets human preference. The target video generation model significantly improves the score in the dynamic visual dimension while maintaining or improving the overall visual quality, and can generate more lively and motion-rich video content.

[0070] Based on the above steps S21 to S24, by obtaining the training prompt text, further using the initial video generation model to generate a plurality of candidate videos, then performing video preference evaluation on the plurality of candidate videos from multiple dimensions to obtain an evaluation result, and finally training the initial video generation model based on the evaluation result to generate a target video generation model, the target video generation model is used for video generation processing on the input text to obtain a target video. Thus, by introducing a multi-dimensional video preference evaluation mechanism, especially the evaluation of the dynamic visual dimension, the deficiency of insufficient motion performance caused by single-dimensional evaluation in the related art is overcome. By performing multi-dimensional and all-around video preference evaluation on the plurality of candidate videos generated by the training prompt text, the motion amplitude in the candidate videos can be accurately identified and quantified, and the optimization strategy is adjusted to ensure that the initial video generation model can improve the motion richness and realism of the generated video during the training process. In addition, the model training process based on the evaluation result promotes the capture and reproduction of dynamic visual elements by the initial video generation model, significantly improves the liveliness and realism of the target video, and realizes the balanced optimization of motion performance and visual quality, thereby generating video content that meets human aesthetic preferences, and further solving the technical problems of low motion richness and poor realism of the output video of the video generation method provided in the related art.

[0071] It is easy to note that the target video generation model trained by the model training method of the embodiments of the present application can be applied to the fields of high-quality video generation, film and television content creation, virtual reality, etc. For example, in the film production, television drama shooting, animation design and other industries, the target video generation model can be used to generate video clips that highly match the script or story concept, accelerate the pre-visual development and special effect preview process, and improve the efficiency of creative exploration and the realism and dynamic of video content. In virtual reality technology, the target video generation model can generate virtual environments with rich dynamic effects, enhance the immersion and interaction experience of users, and provide more realistic and lively visual content for virtual reality games, education, tourism and other applications.

[0072] The model training method in the embodiments of the present application will be further introduced below.

[0073] In an optional embodiment, in step S23, the video preference evaluation of the plurality of candidate videos from multiple dimensions is performed to obtain the evaluation result, including: using the trained reward model to evaluate the video preference of the plurality of candidate videos from multiple dimensions to obtain quantitative scores of the multiple dimensions, and determining the evaluation result based on the quantitative scores of the multiple dimensions.

[0074] The reward model described above can be an artificial intelligence model specially used for evaluating the quality of video clips, and can give quantitative scores according to the performance of the candidate videos in multiple dimensions such as aesthetics, motion coherence, and definition. In the embodiments of the present application, the reward model is trained to understand the complexity of human video preferences and can make detailed evaluations of the quality of video clips in different dimensions. In the embodiments of the present application, the reward model is first trained with a large amount of data to understand the quality characteristics of the multiple candidate videos in different dimensions and human preference patterns, so as to ensure that it can accurately and fairly evaluate the quality of videos.

[0075] In the process of video preference evaluation, the reward model can not only evaluate the overall quality of the candidate videos, but also make detailed quality evaluations in specific aspects, such as the object motion amplitude and camera motion amplitude in the motion dimension, the definition and color saturation in the visual quality dimension, and the degree of consistency of the text description in the content matching degree dimension. The multi-dimensional evaluation ensures comprehensive measurement of the quality and preference of the candidate videos, rather than a single dimension.

[0076] The quantitative scores of the multiple dimensions are specific numerical representations of the evaluation results of the candidate videos in specific dimensions by the reward model. Each candidate will obtain a score in each evaluation dimension, which provides a quantitative basis for subsequent video optimization and model optimization.

[0077] The evaluation result integrates the quantitative scores of the multiple dimensions and can be obtained through certain mathematical operations or statistical analysis, which is used to intuitively reflect the preference level or ranking of the candidate videos as a whole. The evaluation result can guide the adjustment direction of the initial video generation model, and by comparing the evaluation results of different candidates, the preference alignment problem of the initial video generation model can be identified and improved.

[0078] Based on the above optional embodiment, by using the trained reward model to evaluate the video preference of the plurality of candidate videos from multiple dimensions, the quantitative scores of the multiple dimensions are obtained, and the evaluation result is determined based on the quantitative scores of the multiple dimensions, thereby significantly improving the accuracy of the model training process, helping the initial video generation model to achieve higher level of preference alignment in motion expressiveness, visual quality, and content expression, so as to generate more vivid, high-quality, and highly consistent video content with human preferences.

[0079] In an optional embodiment, the initial video generation model is trained based on the evaluation result to generate a target video generation model in step S24, which includes:

[0080] In step S241, the multiple candidate videos are preliminarily judged based on the evaluation result to obtain positive video samples and negative video samples.

[0081] In step S242, motion deviation detection is performed on the positive video samples and the negative video samples to obtain detection results.

[0082] In step S243, the initial video generation model is trained according to the detection results to generate a target video generation model.

[0083] Through evaluation, the evaluation result can be used to preliminarily judge the multiple candidate videos, so as to determine the positive video samples and the negative video samples in the multiple candidate videos. The positive video samples can be video clips that are more optimal or more consistent with the training target, especially videos with high motion amplitude and dynamic performance in the dynamic visual dimension. The positive video samples usually have high scores in multidimensional evaluation, such as dynamic vision, motion coherence, picture clarity, content and text description matching degree, and overall visual effect and aesthetic quality. The positive video samples can be the optimal results generated by the initial video generation model based on the current optimization strategy in a specific training stage, and can be used as a standard to guide the initial video generation model to imitate or surpass the characteristics of these videos in subsequent training. In the embodiments of the present application, the performance of the positive video samples in the motion dimension is particularly important, that is, the amplitude and dynamics of the motion object in the video, and the fluency and naturalness of the camera motion.

[0084] On the contrary, the negative video samples have lower evaluation scores and may have obvious deficiencies in one or more dimensions, such as motion blur, low picture quality, content deviation from the training text description, etc. Although the overall score of the negative video samples is low, they can provide important feedback information to help the initial video generation model identify the deficiencies and optimization space in the training, and are "counterexamples" in the model learning process. The existence of negative video samples is crucial for identifying and correcting biases like "reward hackers", which can prompt the initial video generation model to consider how to avoid similar problems in the training process to generate more balanced and high-quality video content.

[0085] Further, motion bias detection is performed on the positive video sample and the negative video sample to obtain a detection result. The motion bias detection is a process of quantitatively analyzing the difference between the positive video sample and the negative video sample in the dynamic visual dimension. Specifically, it can include calculation of a motion amplitude score and a camera motion amplitude score, aiming to identify the preference bias in the dynamic visual dimension and provide data support for subsequent correction strategies. In the embodiments of the present application, the motion bias detection is the basis for designing the motion correction weight. By calculating the score difference and using a normalization function, a dynamic optimization weight is generated to promote the alignment of the initial video generation model in the dynamic visual dimension.

[0086] According to the detection result of the motion bias detection, the weights of the positive video sample and the negative video sample in the training process can be dynamically adjusted, and then the initial video generation model is trained. Through weighted adjustment, the performance in the dynamic visual dimension can be emphasized and improved, and the initial video generation model is iterated gradually until a target video generation model that can output motion richness and enhanced realism is obtained.

[0087] Based on the above optional embodiments, by preliminarily judging a plurality of candidate videos based on the evaluation result to obtain a positive video sample and a negative video sample, and then performing motion bias detection on the positive video sample and the negative video sample to obtain a detection result, and finally training the initial video generation model according to the detection result to generate a target video generation model, the dynamic performance and visual appeal of video generation are significantly improved, making it closer to the video preferences of users, and further improving the user experience.

[0088] In an optional embodiment, the dynamic visual dimension includes a motion amplitude of a video display object and a motion amplitude of a video capture device. In step S242, the motion bias detection on the positive video sample and the negative video sample to obtain a detection result includes:

[0089] In step S2421, a first quantitative score of the positive video sample in the motion amplitude of the video display object, a second quantitative score of the positive video sample in the motion amplitude of the video capture device, a third quantitative score of the negative video sample in the motion amplitude of the video display object, and a fourth quantitative score of the negative video sample in the motion amplitude of the video capture device are obtained.

[0090] In step S2422, based on the first quantitative score, the second quantitative score, the third quantitative score, and the fourth quantitative score, the motion bias detection is performed on the positive video sample and the negative video sample to obtain a detection result.

[0091] The dynamic visual dimensions include the motion amplitude of the video display object and the motion amplitude of the video capture device (such as a camera). The motion amplitude of the display object relates to the motion intensity and speed of the object or person in the candidate video, and the motion amplitude of the video capture device focuses on the shooting angle, the speed and stability of the lens movement, which together constitute the basis of the dynamic performance of the video and directly affect the liveliness and realism of the candidate video.

[0092] The reward model scores the positive video samples and the negative video samples in the two dimensions of the motion amplitude of the video display object and the motion amplitude of the video capture device, respectively, to generate four sets of quantitative scores. Specifically, the score of the positive video sample in the motion amplitude of the video display object is referred to as the first quantitative score s om w, and the score in the motion amplitude of the video capture device is referred to as the second quantitative score s om l. Similarly, the scores of the negative video sample in the two dimensions are the third quantitative score s cm w and the fourth quantitative score s cm l. The four sets of quantitative scores provide intuitive data about the dynamic performance of the video, laying the foundation for further motion bias detection.

[0093] Further, based on the first quantitative score, the second quantitative score, the third quantitative score, and the fourth quantitative score, the motion bias of the positive video sample and the negative video sample is detected to obtain a detection result. Specifically, it involves comparing the difference in motion amplitude between the positive video sample and the negative video sample, identifying whether there is a systematic preference bias, especially for the improper reward of static or low dynamic video. By calculating the score difference of the positive and negative samples in the motion amplitude of the display object and the motion amplitude of the video capture device, the preference bias in the motion dimension can be quantified to obtain the detection result, which is used for subsequent design of weight optimization and fine-tuning of the video generation model.

[0094] Based on the above optional embodiment, by obtaining the first quantitative score of the positive video sample in the motion amplitude of the video display object, the second quantitative score of the positive video sample in the motion amplitude of the video capture device, the third quantitative score of the negative video sample in the motion amplitude of the video display object, and the fourth quantitative score of the negative video sample in the motion amplitude of the video capture device, and further based on the first quantitative score, the second quantitative score, the third quantitative score, and the fourth quantitative score, the motion bias of the positive video sample and the negative video sample is detected to obtain a detection result, which significantly enhances the accuracy of the dynamic quality control of the video, ensures that the adjustment of the candidate video in the motion amplitude has a quantitative basis, creates favorable conditions for subsequent model training, directly promotes the improvement of the generated video in the motion performance, reduces the motion suppression problem caused by dimension coupling, and further presents more lively, natural and high-quality video content to the user.

[0095] In an optional embodiment, in step S2422, based on the first quantitative score, the second quantitative score, the third quantitative score and the fourth quantitative score, motion bias detection is performed on the positive video sample and the negative video sample to obtain a detection result including:

[0096] A first difference value between the first quantitative score and the third quantitative score is calculated, and a second difference value between the second quantitative score and the fourth quantitative score is calculated;

[0097] The detection result is determined based on the first difference value and the second difference value.

[0098] Specifically, a first difference value s_om_l between the first quantitative score s_om_w and the third quantitative score is calculated, and a second difference value between the second quantitative score s_cm_w and the fourth quantitative score s_cm_l is calculated, and then the detection result is determined by using the first difference value and the second difference value, wherein the detection result includes a first motion correction weight w w of the positive video sample and a second motion correction weight w l of the negative video sample, and the detection result can be calculated by the following formula:

[0099]

[0100]

[0101] wherein σ is a normalization function. When the motion amplitude of the positive video sample video is significantly better than that of the negative video sample, w w increases, which strengthens the reward for high dynamic video; otherwise, w w decreases, which avoids false punishment of the video with rich motion due to other dimension disadvantages.

[0102] Based on the above optional embodiment, by calculating a first difference value between the first quantitative score and the third quantitative score, and calculating a second difference value between the second quantitative score and the fourth quantitative score, and then determining the detection result based on the first difference value and the second difference value, the recognition ability of the video motion feature bias is significantly enhanced, which creates conditions for the implementation of subsequent model melody and motion correction strategies, and ensures that the initial video generation model can more accurately capture the details of the dynamic visual dimension, thereby reducing unintended preference for low motion amplitude video in the generation process and improving the motion expressiveness of the video.

[0103] In an optional embodiment, in step S243, the initial video generation model is trained according to the detection result to generate a target video generation model, including:

[0104] In step S2431, the first motion correction weight of the positive video sample and the second motion correction weight of the negative video sample are determined according to the detection result;

[0105] Step S2432, based on the first motion correction weight and the second motion correction weight, training the initial video generation model to generate a target video generation model.

[0106] The first motion correction weight is a weight value calculated according to the scoring advantages and disadvantages of the positive video samples in the motion dimension in the detection result, which is used to emphasize or correct the optimization weight of the motion performance of the positive video samples in the subsequent training process. The second motion correction weight corresponds to the negative video samples, which is also calculated based on the detection result, and is used to adjust the optimization weight of the negative video samples in the motion dimension, so that the model training can reasonably consider the motion characteristics of different samples and avoid the wrong training direction caused by systematic preference bias.

[0107] The video preference alignment strategy in the related art often fails to fully consider the complex relationship between the video motion amplitude and other visual quality dimensions, resulting in the phenomenon of "reward hacker" in model training, that is, the video generation model tends to generate a video with smaller motion amplitude as a positive sample to evade the deduction in other dimensions. By introducing the concept of motion correction weight, the embodiments of the present application can dynamically adjust the influence degree of positive and negative samples in model training, and explicitly correct the motion suppression bias caused by dimension coupling. The embodiments of the present application not only quantify the preference bias between dimensions, but also propose a specific and feasible weight adjustment strategy, which provides a more delicate and balanced training guidance for the initial video generation model, helps to improve the motion richness and realism of the generated video, and at the same time, takes into account other visual quality dimensions.

[0108] Based on the above optional embodiments, by determining the first motion correction weight of the positive video samples and the second motion correction weight of the negative video samples according to the detection result, and then training the initial video generation model based on the first motion correction weight and the second motion correction weight to generate a target video generation model, the motion performance can be significantly improved without sacrificing the overall visual effect, which is more in line with human preference for lively and realistic video content, and provides a strong guarantee for automatic generation of high-quality videos.

[0109] In an optional embodiment, in step S2432, based on the first motion correction weight and the second motion correction weight, the initial video generation model is trained to generate a target video generation model, which includes:

[0110] Based on the first motion correction weight and the second motion correction weight, the initial direct preference optimization loss function is weighted and adjusted to obtain a target direct preference optimization loss function;

[0111] The target direct preference optimization loss function is used to train the initial video generation model to generate a target video generation model.

[0112] The initial direct preference optimization loss function is used to quantify the distance between the candidate video output by the initial video generation model and human preference without any motion correction. During the model training process, the initial direct preference optimization loss function guides the initial video generation model to adjust the parameters to reduce the difference between its output and human preference.

[0113] The target direct preference optimization loss function is an optimized target obtained by weighting adjustment of the initial direct preference optimization loss function based on the first motion correction weight and the second motion correction weight. Through the weighting adjustment, the target direct preference optimization loss function can more accurately reflect the multi-dimensional preference of humans for videos, especially the emphasis on motion amplitude and dynamic performance, thereby guiding the generation of more motion-rich and high-quality video content in the model training. The target direct preference optimization loss function can be:

[0114]

[0115] wherein, is a hyperparameter in the motion correction direct preference optimization algorithm, used to adjust the loss proportion of the motion dimension, to ensure that the optimization target of motion richness is not overshadowed by the optimization requirements of other dimensions. and respectively represent the positive video samples y w and negative video samples y l given by the reward model, and the preference score of the input condition x.

[0116] After obtaining the target direct preference optimization loss function, the initial video generation model is trained using the optimization target. Specifically, the model parameters can be continuously adjusted by optimization algorithms such as gradient descent until the video segments output by the model can obtain a smaller loss function, i.e., the motion amplitude, dynamic performance, and other visual quality dimensions are consistent with human preference to the greatest extent. The final generated target video generation model has stronger motion expressiveness while maintaining the overall visual quality level, and is more consistent with human true preference.

[0117] Based on the above optional embodiments, by weighting adjustment of the initial direct preference optimization loss function based on the first motion correction weight and the second motion correction weight, the target direct preference optimization loss function is obtained, and then the initial video generation model is trained using the target direct preference optimization loss function to generate the target video generation model, further improving the expressiveness and authenticity of the target video generation model in processing motion-related scenes.

[0118] In an optional embodiment, training the initial video generation model using the target direct preference optimization loss function to generate the target video generation model includes:

[0119] The target loss is calculated by a target direct preference optimization loss function, and the target loss is used to determine a preference probability of the positive video sample relative to the negative video sample.

[0120] The model parameters of the initial video generation model are fine-tuned based on the target loss to generate a target video generation model.

[0121] The target loss is a quantitative index calculated by the target direct preference optimization loss function, and is used to measure the preference degree of the positive video sample relative to the negative video sample. Specifically, the target loss reflects the relative gap between the generated positive video sample and the negative video sample in the perspective of human preference in the model training process, and is the core basis for guiding the adjustment of the initial video generation model parameters.

[0122] The target loss is used to determine the preference probability of the positive video sample relative to the negative video sample, and the preference probability can represent the possibility that the positive video sample is more preferred by humans than the negative video sample. In the embodiments of the present application, the calculation result of the target loss can be converted into the preference probability, which helps to understand the consistency degree of the candidate video generated by the initial video generation model and the real preference of humans.

[0123] The parameters of the initial video generation model are fine-tuned based on the calculated target loss. In the fine-tuning process, the internal parameters of the initial video generation model can be continuously and slightly adjusted according to the calculated target loss, so as to reduce the target loss and improve the alignment degree of the model output and the human preference. The fine-tuning strategy can usually be assisted by optimization algorithms such as gradient descent, so as to ensure that the initial video generation model can gradually learn more accurate video generation rules.

[0124] Based on the above optional embodiments, by calculating the target loss by using the target direct preference optimization loss function, and then fine-tuning the model parameters of the initial video generation model based on the target loss, the finally generated target video generation model has stronger motion expressiveness, while maintaining good static visual quality, and significantly improves the liveliness and realism of the video content.

[0125] Figure 3 is a schematic diagram of a model training method according to an embodiment of the present application, as Figure 3As shown, the training prompt text is obtained, the initial video generation model is used to perform video generation processing on the training prompt text, and a plurality of candidate videos such as candidate video 1 and candidate video 2 are obtained. The trained reward model is used to perform video preference evaluation on the plurality of candidate videos from multiple dimensions, to obtain quantitative scores of multiple dimensions, and to determine an evaluation result based on the quantitative scores of multiple dimensions. Based on the evaluation result, the plurality of candidate videos are preliminarily determined to obtain positive video samples and negative video samples, the positive video samples and the negative video samples are subjected to motion deviation detection to obtain a detection result. Further, the first motion correction weight of the positive video samples and the second motion correction weight of the negative video samples are determined according to the detection result, and finally, the initial video generation model is trained based on the first motion correction weight and the second motion correction weight to generate a target video generation model.

[0126] The embodiments of the present application are directed to the dimension coupling and "reward hacking" problems in video preference alignment in the related art. Through the proposed motion correction direct preference optimization mechanism, systematic overcoming is achieved. In the related art, human preference is simplified as a single scalar reward signal, or each dimension score is simply weighted and fused, ignoring the complex interaction between multiple dimensions of video quality. Through experimental analysis, it is determined that in the evaluation system of the existing reward model, the motion-related dimension and other static visual quality dimensions (such as aesthetic quality, clarity, consistency, etc.) have significant negative correlation. That is, the more rich and dynamic the video, the more it is deducted in other dimensions due to factors such as picture blur and unstable composition, resulting in a low overall preference score. The above systematic bias makes the video generation model tend to select videos with small motion amplitude and regular pictures as positive samples in the DPO alignment training process, forming a "reward hacker", that is, the video generation model "cheats" the reward model by generating conservative and low dynamic content to obtain a higher overall score, but this generation result seriously deviates from the real preference of humans for lively and expressive videos.

[0127] The model training method in the embodiments of the present application does not avoid the conflict between dimensions, but actively identifies and quantifies the negative correlation between dimensions. In the training process, not only the overall preference label of the positive and negative sample video pair can be received, but also the difference in the fine-grained score of each dimension is analyzed in depth. By calculating the score difference of the object motion amplitude and the camera motion amplitude of the positive video sample and the negative video sample, and inputting it into a normalization function, a dynamic optimization weight, i.e., the first motion correction weight and the second motion correction weight, is generated. When a motion-rich video is determined as a positive video sample, its first motion correction weight will be significantly improved; on the contrary, if a motion-poor video becomes a positive video sample only because of high static quality, its first motion correction weight will be properly inhibited. Thus, the optimization process is no longer simply pursuing high comprehensive score, but consciously "rewarding" the video that dares to show high dynamic motion, even if it is slightly insufficient in other aspects.

[0128] Through the above explicit, evidence-based correction strategy, the target video generation model trained in the embodiments of the present application successfully breaks the implicit preference of the traditional alignment method for low dynamic videos, while maintaining a good level in the dimensions of visual quality, text-image alignment, etc. Finally, the target video generated by the target video generation model is more dynamic, more dramatic and realistic, and can better meet the user's expectation of high-quality, high-dynamic visual content.

[0129] Figure 4 is a flowchart of a video generation method according to an embodiment of the present application, as shown in Figure 4 The method can include the following steps:

[0130] Step S41, obtaining the input text;

[0131] Step S42, performing video generation processing on the input text by using a target video generation model to obtain a target video; wherein the target video generation model is generated according to the model training method of any one of the embodiments of the present application.

[0132] Based on the steps S41 to S42, the target video is obtained by acquiring the input text and performing video generation processing on the input text using the target video generation model. In the training process of the target video generation model, the training prompt text is acquired, and then the initial video generation model is used to perform video generation processing on the training prompt text to obtain a plurality of candidate videos. Subsequently, the plurality of candidate videos are evaluated from multiple dimensions to obtain an evaluation result. Finally, the initial video generation model is trained based on the evaluation result to generate the target video generation model. The target video generation model is used to perform video generation processing on the input text to obtain the target video. Thus, by introducing a multi-dimensional video preference evaluation mechanism, especially the evaluation of dynamic visual dimensions, the deficiency of insufficient motion performance caused by single-dimensional evaluation in related technologies is overcome. By performing multi-dimensional and all-around video preference evaluation on the plurality of candidate videos generated by the training prompt text, the motion amplitude in the candidate videos can be accurately identified and quantified, and the optimization strategy is adjusted to ensure that the initial video generation model can improve the motion richness and realism of the generated video in the training process. In addition, the model training process based on the evaluation result promotes the capture and reproduction of dynamic visual elements by the initial video generation model, significantly improves the liveliness and realism of the target video, and balances and optimizes the motion performance and visual quality, thereby generating video content that is more in line with human aesthetic preferences, and further solving the technical problems of low motion richness and poor realism of the output video of the video generation method provided in related technologies.

[0133] Figure 5 is a flowchart of still another video generation method according to an embodiment of the present application, as shown in Figure 5 The method can include the following steps:

[0134] Step S51, acquiring a film and television content creation text;

[0135] Step S52, performing video generation processing on the film and television content creation text using a target video generation model to obtain a film and television content creation video; wherein the target video generation model is generated according to the model training method of any one of the embodiments of the present application.

[0136] Based on the steps S51 to S52, the film and television content creation text is obtained, and then the film and television content creation text is processed by the target video generation model to obtain the film and television content creation video. In the training process of the target video generation model, the training prompt text is obtained, and then the training prompt text is processed by the initial video generation model to obtain a plurality of candidate videos. Then, the plurality of candidate videos are evaluated from multiple dimensions to obtain an evaluation result. Finally, the initial video generation model is trained based on the evaluation result to generate the target video generation model. The target video generation model is used to process the input text to obtain the film and television content creation video. Thus, by introducing the multi-dimensional video preference evaluation mechanism, especially the evaluation of dynamic visual dimension, the defect of insufficient motion performance caused by single-dimensional evaluation in the related art is overcome. By performing multi-dimensional and all-around video preference evaluation on the plurality of candidate videos generated by the training prompt text, the motion amplitude in the candidate videos can be accurately identified and quantified, and the optimization strategy is adjusted to ensure that the initial video generation model can improve the motion richness and realism of the generated video in the training process. In addition, the model training process based on the evaluation result promotes the capture and reproduction of dynamic visual elements by the initial video generation model, significantly improves the liveliness and realism of the target video, balances and optimizes the motion performance and visual quality, generates video content that is more in line with human aesthetic preferences, and thus solves the technical problems of low motion richness and poor realism of the output video of the video generation method provided in the related art.

[0137] Figure 6 is a flowchart of another video generation method according to an embodiment of the present application, as shown in Figure 6 The method can include the following steps:

[0138] In step S61, a video generation request is obtained through a first application programming interface, wherein the request data carried in the video generation request includes an input text.

[0139] In step S62, a video generation response is returned through a second application programming interface, wherein the response data carried in the video generation response includes a target video, the target video is obtained by processing the input text by a target video generation model, and the target video generation model is generated according to the model training method of any one of the embodiments of the present application.

[0140] Based on the steps S61 to S62, the video generation request is obtained through the first application programming interface, wherein the request data carried in the video generation request includes the input text, and then the video generation response is returned through the second application programming interface, and the response data carried in the video generation response includes the target video, which is obtained by performing video generation processing on the input text by using a target video generation model. In the training process of the target video generation model, the training prompt text is obtained, and then the training prompt text is processed by using an initial video generation model to obtain a plurality of candidate videos. Subsequently, the plurality of candidate videos are evaluated in terms of video preference from a plurality of dimensions to obtain an evaluation result. Finally, the initial video generation model is trained based on the evaluation result to generate the target video generation model, which is used to perform video generation processing on the input text to obtain the target video. Thus, by introducing a multi-dimensional video preference evaluation mechanism, especially the evaluation of dynamic visual dimension, the defect of insufficient motion performance caused by single-dimensional evaluation in the related art is overcome. By performing multi-dimensional and all-around video preference evaluation on the plurality of candidate videos generated by the training prompt text, the motion amplitude in the candidate videos can be accurately identified and quantified, and then the optimization strategy is adjusted to ensure that the initial video generation model can improve the motion richness and realism of the generated video in the training process. In addition, the model training process based on the evaluation result promotes the capture and reproduction of dynamic visual elements by the initial video generation model, significantly improves the liveliness and realism of the target video, realizes the balanced optimization of motion performance and visual quality, generates video content that is more in line with human aesthetic preferences, and thus solves the technical problems of low motion richness and poor realism of the output video in the video generation method provided in the related art.

[0141] Figure 7 is a flowchart of still another video generation method according to an embodiment of the present application, as shown in Figure 7 The method can include the following steps:

[0142] Step S71, obtaining a current input video generation dialogue request, wherein the request data carried in the video generation dialogue request includes the input text;

[0143] Step S72, in response to the video generation dialogue request, returning a video generation dialogue reply, wherein the information carried in the video generation dialogue reply includes the target video, which is obtained by performing video generation processing on the input text by using a target video generation model, and the target video generation model is generated according to the model training method of any one of the embodiments of the present application;

[0144] Step S73, playing the target video in the graphical user interface.

[0145] Based on the steps S71 to S73, a video generation dialogue request is generated by obtaining the current input video, and the request data carried in the video generation dialogue request includes the input text. In response to the video generation dialogue request, a video generation dialogue reply is returned, and the information carried in the video generation dialogue reply includes the target video obtained by video generation processing of the input text by using the target video generation model. Finally, the target video is played in the graphical user interface. In the training process of the target video generation model, the training prompt text is obtained, and then the initial video generation model is used to perform video generation processing on the training prompt text to obtain a plurality of candidate videos. Subsequently, the plurality of candidate videos are evaluated in terms of video preference from multiple dimensions to obtain an evaluation result. Finally, the initial video generation model is trained based on the evaluation result to generate the target video generation model, which is used to perform video generation processing on the input text to obtain the target video. Thus, by introducing the multi-dimensional video preference evaluation mechanism, especially the evaluation of the dynamic visual dimension, the defect of insufficient motion performance caused by single-dimensional evaluation in the related art is overcome. By performing multi-dimensional and all-around video preference evaluation on the plurality of candidate videos generated by the training prompt text, the motion amplitude in the candidate videos can be accurately identified and quantified, and the optimization strategy is adjusted to ensure that the initial video generation model can improve the motion richness and realism of the generated video in the training process. In addition, the model training process based on the evaluation result promotes the capture and reproduction of dynamic visual elements by the initial video generation model, significantly improves the liveliness and realism of the target video, and realizes the balanced optimization of motion performance and visual quality, so as to generate video content that is more in line with human aesthetic preferences, thereby solving the technical problems of low motion richness and poor realism of the output video of the video generation method provided in the related art.

[0146] Figure 8 is a flowchart of another video generation method according to an embodiment of the present application, as shown in Figure 8 , the method can include the following steps:

[0147] Step S81, in response to the input instruction acting on the operation interface, displaying the input text on the operation interface;

[0148] Step S82, in response to the processing instruction acting on the operation interface, playing the target video on the operation interface; wherein the target video is obtained by video generation processing of the input text by using the target video generation model, and the target video generation model is generated according to the model training method of any one of the embodiments of the present application.

[0149] Based on the steps S81 to S82, the input text is displayed on the operation interface in response to the input instruction acting on the operation interface, and the target video is played on the operation interface in response to the processing instruction acting on the operation interface, the target video being obtained by performing video generation processing on the input text by using the target video generation model. In the training process of the target video generation model, the training prompt text is obtained, and then the initial video generation model is used to perform video generation processing on the training prompt text to obtain a plurality of candidate videos. Subsequently, the plurality of candidate videos are evaluated in terms of video preference from a plurality of dimensions to obtain an evaluation result. Finally, the initial video generation model is trained based on the evaluation result to generate the target video generation model, which is used to perform video generation processing on the input text to obtain the target video. Thus, by introducing the multi-dimensional video preference evaluation mechanism, especially the evaluation of the dynamic visual dimension, the defect of insufficient motion performance caused by single-dimensional evaluation in the related art is overcome. By performing multi-dimensional and all-around video preference evaluation on the plurality of candidate videos generated by the training prompt text, the motion amplitude in the candidate videos can be accurately identified and quantified, and the optimization strategy is adjusted to ensure that the initial video generation model can improve the motion richness and realism of the generated video in the training process. In addition, the model training process based on the evaluation result promotes the capture and reproduction of dynamic visual elements by the initial video generation model, significantly improves the liveliness and realism of the target video, and balances and optimizes the motion performance and visual quality, so as to generate video content that is more in line with human aesthetic preferences, thereby solving the technical problems of low motion richness and poor realism of the output video of the video generation method provided in the related art.

[0150] It should be noted that the user data involved in the present application is all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data comply with relevant laws, regulations and standards of relevant countries and regions, and appropriate operation portals are provided for users to choose authorization or refusal.

[0151] For the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited to the action sequence described. Because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0152] Those skilled in the art can clearly understand the method according to the above-mentioned embodiments can be realized by means of software and necessary general hardware platform, or can be realized by means of hardware. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the method described in each embodiment of the present application.

[0153] According to the embodiments of the present application, a model training device for implementing the above-mentioned model training method is further provided, Figure 9 is a structural block diagram of a model training device according to an embodiment of the present application, as shown in the figure, the device comprises: Figure 9

[0154] The acquisition module 901 is configured to acquire the training prompt text.

[0155] The generation module 902 is configured to perform video generation processing on the training prompt text by using the initial video generation model, to obtain a plurality of candidate videos.

[0156] The evaluation module 903 is configured to perform video preference evaluation on the plurality of candidate videos from a plurality of dimensions, to obtain an evaluation result, wherein the plurality of dimensions at least include a dynamic visual dimension, and the evaluation result is at least used to evaluate the motion amplitude of a plurality of motion objects associated with the dynamic visual dimension.

[0157] The training module 904 is configured to train the initial video generation model based on the evaluation result, to generate a target video generation model, wherein the target video generation model is used to perform video generation processing on an input text to obtain a target video.

[0158] Optionally, the evaluation module 903 is further configured to perform video preference evaluation on the plurality of candidate videos from a plurality of dimensions by using the trained reward model, to obtain quantitative scores of the plurality of dimensions, and determine the evaluation result based on the quantitative scores of the plurality of dimensions.

[0159] Optionally, the training module 904 is further configured to perform preliminary determination on the plurality of candidate videos based on the evaluation result, to obtain positive video samples and negative video samples; perform motion deviation detection on the positive video samples and the negative video samples, to obtain a detection result; and train the initial video generation model according to the detection result, to generate the target video generation model.

[0160] ​Optionally, the training module 904 is further configured to: obtain a first quantitative score of the positive video sample on the motion amplitude of the video display object, a second quantitative score of the positive video sample on the motion amplitude of the video capture device, a third quantitative score of the negative video sample on the motion amplitude of the video display object, and a fourth quantitative score of the negative video sample on the motion amplitude of the video capture device; and perform motion bias detection on the positive video sample and the negative video sample based on the first quantitative score, the second quantitative score, the third quantitative score, and the fourth quantitative score to obtain a detection result.

[0161] Optionally, the training module 904 is further configured to: calculate a first difference value between the first quantitative score and the third quantitative score, and calculate a second difference value between the second quantitative score and the fourth quantitative score; and determine the detection result based on the first difference value and the second difference value.

[0162] Optionally, the training module 904 is further configured to: determine a first motion correction weight of the positive video sample and a second motion correction weight of the negative video sample according to the detection result; and train the initial video generation model based on the first motion correction weight and the second motion correction weight to generate a target video generation model.

[0163] Optionally, the training module 904 is further configured to: perform weighted adjustment on the initial direct preference optimization loss function based on the first motion correction weight and the second motion correction weight to obtain a target direct preference optimization loss function; and train the initial video generation model based on the target direct preference optimization loss function to generate the target video generation model.

[0164] Optionally, the training module 904 is further configured to: calculate a target loss based on the target direct preference optimization loss function, wherein the target loss is used to determine a preference probability of the positive video sample relative to the negative video sample; and fine-tune model parameters of the initial video generation model based on the target loss to generate the target video generation model.

[0165] The above obtaining module 901, the generating module 902, the evaluating module 903, and the training module 904 correspond to steps S21 to S24 in the above embodiments. The four modules have the same examples and application scenarios as the corresponding steps, but are not limited to the above disclosed content. The above modules or units can be hardware components or software components stored in the memory and processed by one or more processors. The above modules can also be run in the server provided in the above embodiments as part of the device.

[0166] Figure 10 is a structural block diagram of a video generation device according to an embodiment of the present application, as shown in Figure 10 The device comprises:

[0167] The acquisition module 1001 is configured to acquire input text.

[0168] The generation module 1002 is configured to perform video generation processing on the input text by using a target video generation model to obtain a target video, wherein the target video generation model is generated according to the model training method in any one of the embodiments of the present application.

[0169] The acquisition module 1001 and the generation module 1002 correspond to steps S41-S42 in the above embodiments. The two modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiments. The modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The modules can also be run in the server provided in the above embodiments as part of the device.

[0170] Figure 11 is a structural block diagram of still another video generation device according to an embodiment of the present application, as shown in Figure 11 The device includes:

[0171] The acquisition module 1101 is configured to acquire a film and television content creation text.

[0172] The generation module 1102 is configured to perform video generation processing on the film and television content creation text by using a target video generation model to obtain a film and television content creation video, wherein the target video generation model is generated according to the model training method in any one of the embodiments of the present application.

[0173] The acquisition module 1101 and the generation module 1102 correspond to steps S51-S52 in the above embodiments. The two modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiments. The modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The modules can also be run in the server provided in the above embodiments as part of the device.

[0174] Figure 12 is a structural block diagram of still another video generation device according to an embodiment of the present application, as shown in Figure 12 The device includes:

[0175] The acquisition module 1201 is configured to acquire a video generation request through a first application programming interface, wherein the request data carried in the video generation request includes input text.

[0176] The returning module 1202 is configured to return a video generation response through a second application programming interface, wherein response data carried in the video generation response includes a target video, and the target video is obtained by performing video generation processing on the input text by using a target video generation model, and the target video generation model is generated according to the model training method in any one of the embodiments of the present application.

[0177] The obtaining module 1201 and the returning module 1202 correspond to steps S61 to S62 in the above embodiments. The two modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiments. The modules or units described above can be hardware components or software components stored in a memory and processed by one or more processors. The modules described above can also be run in the server provided in the above embodiments as part of the device.

[0178] Figure 13 is a structural block diagram of still another video generation device according to an embodiment of the present application, as shown in Figure 13 The device includes:

[0179] The obtaining module 1301 is configured to obtain a currently input video generation dialogue request, wherein request data carried in the video generation dialogue request includes input text.

[0180] The returning module 1302 is configured to return a video generation dialogue reply in response to the video generation dialogue request, wherein information carried in the video generation dialogue reply includes a target video, and the target video is obtained by performing video generation processing on the input text by using a target video generation model, and the target video generation model is generated according to the model training method in any one of the embodiments of the present application.

[0181] The playing module 1303 is configured to play the target video in a graphical user interface.

[0182] The obtaining module 1301, the returning module 1302, and the playing module 1303 correspond to steps S71 to S73 in the above embodiments. The three modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiments. The modules or units described above can be hardware components or software components stored in a memory and processed by one or more processors. The modules described above can also be run in the server provided in the above embodiments as part of the device.

[0183] Figure 14 is a structural block diagram of still another video generation device according to an embodiment of the present application, as shown in Figure 14 The device includes:

[0184] The display module 1401 is configured to display the input text on the operation interface in response to an input instruction acting on the operation interface.

[0185] The playing module 1402 is configured to play the target video on the operation interface in response to a processing instruction acting on the operation interface, wherein the target video is obtained by performing video generation processing on the input text by using a target video generation model, and the target video generation model is generated according to the model training method in any one of the embodiments of the present application.

[0186] The display module 1402 and the playing module 1402 correspond to steps S81-S82 in the above embodiments. The two modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiments. The above modules or units can be hardware components or software components stored in the memory and processed by one or more processors. The above modules can also be run in the server provided in the above embodiments as part of the device.

[0187] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios, implementation processes as the schemes provided in the above embodiments, and will not be repeated here.

[0188] The embodiments of the present application can provide a video generation system, comprising: a client configured to send an input text; a server connected with the client, configured to perform video generation processing on the input text by using a target video generation model to obtain a target video, wherein the target video generation model is generated according to the model training method in any one of the embodiments of the present application; and the client is further configured to output the target video.

[0189] The embodiments of the present application can provide an electronic device. Figure 15 is a structural block diagram of a computing device according to an embodiment of the present application. As shown in the figure, the computing device can include one or more (one is shown in the figure) processors 152, a memory 154, a storage controller, and a peripheral interface. Figure 15

[0190] The above computing device can be understood as an integrated intelligent terminal, including but not limited to a server, a desktop computer, a PC (Personal Computer, personal computer), a model integrated machine, etc. Moreover, the computing device can be pre-installed with the model described in the above embodiments of the present application.

[0191] ​Specifically, the computing device can preinstall multiple types of models, including but not limited to models in the fields of natural language processing, visual processing, speech processing, code processing, multi-modal task processing, etc., to provide diversified model selection. In different product forms, the computing device can support one or more model usage modes, including but not limited to model training, model calling, model fine-tuning, model deployment, model inference and application, etc. In some product forms, the computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative and other types of models), model version control (supporting the control of different model versions), model evaluation (based on model evaluation tools, evaluating the performance and effect of the model), etc. In other product forms, the computing device can also create applications based on models and provide API calling capabilities. Models can be called into created applications through API interfaces, and application management tools are provided to realize the management and control of the applications.

[0192] Further, the computing device can also include data management (supporting the creation and management of model optimization datasets), training center (providing rich training resources to help users learn and master AI technology), basic management capability (providing enterprise-level basic management capability to ensure the security and efficient operation of the system). Through the above functions, a comprehensive and integrated AI development, training, deployment and application device is provided.

[0193] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the methods in the above embodiments. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the terminal through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0194] The processor can call the executable programs stored in the memory through the transmission device to execute the methods described in any one of the above embodiments.

[0195] Embodiments of the present application can provide an electronic device. Figure 16 is a structural block diagram of an electronic device according to an embodiment of the present application. As shown in the figure, the electronic device can include an input / output device 162, a memory 164 and a processor 166, wherein the processor 166 is connected with the input / output device 162 and the memory 164 through a bus 168.

[0196] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and apparatuses in the embodiments of the present application. The processor executes various functions and data processing by running the software programs and modules stored in the memory, i.e., implements the methods in the above embodiments. The memory can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the terminal through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0197] The processor can invoke the executable programs stored in the memory through the transmission device to execute the methods in any one of the above embodiments.

[0198] Those skilled in the art can understand that the structure shown in the figure is a schematic structure. The computing device can also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, etc. The figure does not limit the structure of the computing device. For example, the computing device can further include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure.

[0199] Those skilled in the art can understand that all or part of the steps in the methods of the above embodiments can be completed by programs instructing the hardware of the terminal device. The programs can be stored in a computer readable storage medium. The storage medium can include a flash disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0200] The embodiments of the present application also provide a computer readable storage medium. Optionally, in the embodiments, the above computer readable storage medium can be used to save program codes executed by the methods provided by the above embodiments.

[0201] Optionally, in the embodiments, the above storage medium can be located in the computing device.

[0202] Optionally, in the embodiments, the computer readable storage medium is configured to store executable programs. When the executable programs are running, the device where the computer readable storage medium is located executes the methods in any one of the above embodiments.

[0203] The embodiment of the present application further provides a computer program product. Optionally, the computer program product can include a computer program. The computer program is executed by a processor to implement the method provided by the above embodiment.

[0204] The embodiment of the present application further provides a computer program product. Optionally, the computer program product can include a non-volatile computer readable storage medium, which can be used to store a computer program. The computer program is executed by a processor to implement the method provided by the above embodiment.

[0205] The embodiment of the present application further provides a computer program. Optionally, the computer program is executed by a processor to implement the method provided by the above embodiment.

[0206] In the above embodiments of the present application, the description of each embodiment has its own focus. The parts not described in detail in a certain embodiment can refer to the relevant description of other embodiments.

[0207] In several embodiments provided in the present application, the disclosed technical contents can be implemented by other ways. Among them, the above described device embodiments are illustrative, for example, the division of the units is a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0208] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment.

[0209] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0210] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or, in other words, the part that contributes to the prior art, or the whole or part of the technical solutions, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, etc.

[0211] The above is the preferred embodiment of the present application. For those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made. These improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A model training method, characterized in that, The method comprises the following steps: obtaining a training prompt text; generating a video for the training prompt text by using an initial video generation model to obtain a plurality of candidate videos; evaluating the plurality of candidate videos from multiple dimensions to obtain evaluation results, wherein the multiple dimensions at least include a dynamic visual dimension, and the evaluation results are used to evaluate at least the motion amplitude of a plurality of moving objects associated with the dynamic visual dimension, and the dynamic visual dimension includes the motion amplitude of a video display object and the motion amplitude of a video capture device; performing preliminary determination on the plurality of candidate videos based on the evaluation results to obtain positive video samples and negative video samples; obtaining a first quantitative score of the positive video samples on the motion amplitude of the video display object, a second quantitative score of the positive video samples on the motion amplitude of the video capture device, a third quantitative score of the negative video samples on the motion amplitude of the video display object, and a fourth quantitative score of the negative video samples on the motion amplitude of the video capture device; performing motion deviation detection on the positive video samples and the negative video samples based on the first quantitative score, the second quantitative score, the third quantitative score, and the fourth quantitative score to obtain detection results; training the initial video generation model according to the detection results to generate a target video generation model, wherein the target video generation model is used to generate a video for an input text to obtain a target video.

2. The model training method of claim 1, wherein, The method for evaluating the plurality of candidate videos from the multiple dimensions to obtain the evaluation results comprises: using a trained reward model to evaluate the plurality of candidate videos from the multiple dimensions to obtain quantitative scores of the multiple dimensions, and determining the evaluation results based on the quantitative scores of the multiple dimensions.

3. The model training method of claim 1, wherein, The method for performing motion deviation detection on the positive video samples and the negative video samples based on the first quantitative score, the second quantitative score, the third quantitative score, and the fourth quantitative score to obtain the detection results comprises: calculating a first difference value between the first quantitative score and the third quantitative score, and calculating a second difference value between the second quantitative score and the fourth quantitative score; determining the detection results based on the first difference value and the second difference value.

4. The model training method of claim 1, wherein, The method for training the initial video generation model according to the detection results to generate the target video generation model comprises: determining a first motion correction weight of the positive video samples and a second motion correction weight of the negative video samples according to the detection results; training the initial video generation model based on the first motion correction weight and the second motion correction weight to generate the target video generation model.

5. The model training method of claim 4, wherein, The method for training the initial video generation model based on the first motion correction weight and the second motion correction weight to generate the target video generation model comprises: weighting and adjusting an initial direct preference optimization loss function based on the first motion correction weight and the second motion correction weight to obtain a target direct preference optimization loss function. The initial video generation model is trained by using the target direct preference optimization loss function to generate the target video generation model.

6. The model training method of claim 5, wherein, The initial video generation model is trained by using the target direct preference optimization loss function to generate the target video generation model, including: A target loss is calculated by using the target direct preference optimization loss function, where the target loss is used to determine a preference probability of the positive video sample relative to the negative video sample. The model parameters of the initial video generation model are fine-tuned based on the target loss to generate the target video generation model.

7. A method of video generation, the method comprising: It includes: An input text is obtained. A target video is obtained by performing video generation processing on the input text by using a target video generation model. The target video generation model is generated according to the model training method in any one of claims 1 to 6.

8. A method of video generation, the method comprising: It includes: A film and television content creation text is obtained. A film and television content creation video is obtained by performing video generation processing on the film and television content creation text by using a target video generation model. The target video generation model is generated according to the model training method in any one of claims 1 to 6.

9. A method of video generation, the method comprising: It includes: A video generation request is obtained through a first application programming interface, where the request data carried in the video generation request includes an input text. A video generation response is returned through a second application programming interface, where the response data carried in the video generation response includes a target video, the target video is obtained by performing video generation processing on the input text by using a target video generation model, and the target video generation model is generated according to the model training method in any one of claims 1 to 6.

10. A method of video generation, the method comprising: It includes: A current input video generation conversation request is obtained, where the request data carried in the video generation conversation request includes an input text. In response to the video generation conversation request, a video generation conversation reply is returned, where the information carried in the video generation conversation reply includes a target video, the target video is obtained by performing video generation processing on the input text by using a target video generation model, and the target video generation model is generated according to the model training method in any one of claims 1 to 6. The target video is played in a graphical user interface.

11. A method of video generation, the method comprising: It includes: In response to an input instruction acting on an operation interface, an input text is displayed on the operation interface. In response to a processing instruction acting on the operation interface, a target video is played on the operation interface. The target video is obtained by performing video generation processing on the input text by using a target video generation model, and the target video generation model is generated according to the model training method in any one of claims 1 to 6.

12. A video generation system characterized by comprising: It includes: A client is configured to send an input text. A server is connected to the client and is configured to perform video generation processing on the input text by using a target video generation model to obtain a target video, where the target video generation model is generated according to the model training method in any one of claims 1 to 6. The client is further configured to output the target video.

13. An electronic device, comprising: It includes: a memory storing an executable program; a processor configured to execute the program, wherein the program, when executed, implements the model training method of any one of claims 1-6 or the video generation method of any one of claims 7-11.

14. A computer-readable storage medium, characterized in that, a computer-readable storage medium storing an executable program, wherein the executable program, when executed, controls a device in which the computer-readable storage medium is located to implement the model training method of any one of claims 1-6 or the video generation method of any one of claims 7-11.

15. A computer program product, characterised in that, a computer program that, when executed by a processor, implements the model training method of any one of claims 1-6 or the video generation method of any one of claims 7-11.

Images