Animation video generation method and device

Abstract

The embodiment of the invention provides an animation video generation method. The method comprises the steps of obtaining a target guide image and a target text prompt; determining a space-time guide condition based on the target guide image; and generating a series of frames associated with the target guide image in time and space and matched with the target text prompt according to the space-time guide condition and the guide of the target text prompt so as to form the animation video clip. According to the technical scheme provided by the embodiment of the invention, the animation video clip which is formed by a series of frames which are transformed in time and space relative to the target guide image and are matched with the target text can be generated under the guidance of the space-time guide condition and the text prompt, the method is suitable for various video generation tasks, and optimization is particularly carried out for animation video generation.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to an animation video generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Producing videos through generative AI is an efficient and low-cost technology that greatly reduces the difficulty of video generation. However, current video generation models still have problems such as uncontrollable generated video content and limited generated video types, which affects user experience.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0004] The embodiments of the present application provide an animation video generation method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0005] One aspect of an embodiment of the present application provides a method for generating an animation video, the method comprising:

[0006] Obtain target guidance images and target text prompts;

[0007] determining spatiotemporal guidance conditions based on the target guidance image;

[0008] According to the spatiotemporal guidance conditions and the guidance of the target text prompt, a series of frames associated with the target guidance image in time and space and matched with the target text prompt are generated to constitute the animation video clip.

[0009] Optionally, the method is implemented by a pre-trained video generation model, the video generation model comprising: a spatiotemporal mask module, an encoding module and a transformer module, wherein:

[0010] Determining a spatiotemporal guidance condition based on the target guidance image by the spatiotemporal mask module, wherein the spatiotemporal guidance condition includes a guidance sequence and a mask sequence, wherein the guidance sequence is used to specify a position of the target guidance image in the animation video segment, and the mask sequence is used to define whether the series of frames are displayed in the animation video segment;

[0011] Encoding the guide sequence and the mask sequence through an encoding module to obtain a guide feature sequence and a mask encoding sequence;

[0012] The animated video clip is generated by the transformer module based on the guiding feature sequence, the mask encoding sequence, noise and the target text prompt.

[0013] Optionally, the video generation model is obtained by the following operations:

[0014] Acquire a training data set, the training data set comprising a plurality of target animation video clips, each target animation video clip having a corresponding text prompt, the target animation video clip being used to determine a guidance condition;

[0015] Based on the multiple target animation video clips and the corresponding text prompts and guiding conditions, the pre-trained base model is supervised and fine-tuned to obtain a target video generation model, and the target video generation model is used to generate animation videos.

[0016] Optionally, obtain a training data set, including:

[0017] Acquire an original video set, wherein the original video set includes multiple animation videos;

[0018] Dividing the multiple animation videos into scenes to obtain multiple video clips;

[0019] Based on a preset filtering rule, determining the multiple target animation video segments from the multiple video segments;

[0020] Inputting the plurality of target animation video clips into a pre-trained video annotation model to obtain a plurality of text prompts through the video annotation model, each text prompt corresponding to a target animation video clip;

[0021] The training dataset is constructed based on the multiple target animation video clips and the multiple text prompts.

[0022] Optionally, the base model includes: a spatiotemporal mask module, an encoding module and a transformer module, and the target animation video clip includes a frame sequence consisting of a plurality of frames;

[0023] Correspondingly, based on the multiple target animation video clips and the corresponding text prompts and guiding conditions, the pre-trained base model is supervised and fine-tuned to obtain a target video generation model, including:

[0024] Inputting the frame sequence into the spatiotemporal mask module, so as to determine a guide frame from the plurality of frames through the spatiotemporal mask module, and determining a guide sequence and a mask sequence based on the guide frame; wherein the guide frame is used to guide the transformer module to generate a series of frames associated with the guide frame;

[0025] Inputting the guide sequence and the mask sequence into the encoding module to obtain a guide feature sequence and a mask encoding sequence through the encoding module;

[0026] Based on the guiding feature sequence, the mask encoding sequence, noise and corresponding text prompts, the transformer module is supervised fine-tuned to obtain the target video generation model.

[0027] Optionally, determining a guide sequence and a mask sequence based on the guide frame includes:

[0028] placing the guide frame at a target position of the guide sequence, other positions of the guide sequence corresponding to a series of frames associated with the guide frame generated by the converter module;

[0029] A first mask sequence is determined based on the guide sequence, the first mask sequence comprising a demasking mask, the demasking mask corresponding to the guide frame.

[0030] Optionally, determining a guide sequence and a mask sequence based on the guide frame includes:

[0031] placing the guide frame at a target position of the guide sequence, other positions of the guide sequence corresponding to a series of frames associated with the guide frame generated by the converter module;

[0032] Acquire a target mask based on the frame sequence, the target mask comprising a masking area and a demasking area, the target mask being used to control a motion area of ​​the series of frames, wherein the masking area corresponds to a non-motion area, and the demasking area corresponds to a motion area;

[0033] A second mask sequence is determined according to the guide sequence and the target mask, the second mask sequence and the guide sequence have the same length, and the second mask sequence includes a plurality of the target masks.

[0034] Optionally, acquiring a target mask based on the frame sequence includes:

[0035] Performing foreground detection on a first frame of the frame sequence to determine a foreground area of ​​the first frame;

[0036] Based on the foreground area of ​​the first frame, generating a foreground mask corresponding to each of the plurality of frames, wherein the foreground mask is used to represent a motion area and a non-motion area of ​​the corresponding frame;

[0037] The target mask is generated according to the foreground masks corresponding to each of the multiple frames.

[0038] Optionally, based on the multiple target animation video clips and corresponding text prompts and guiding conditions, a pre-trained base model is fine-tuned to obtain a target video generation model, including:

[0039] Determine the resolution and frame rate corresponding to each of the plurality of target animation video clips;

[0040] Based on the resolutions and frame rates corresponding to the multiple target animation video clips, the training data set is divided into multiple subsets, wherein at least one of the resolution and frame rate corresponding to each subset is different from that of other subsets;

[0041] Determine the training priority of each subset according to the resolution and frame rate of each subset;

[0042] According to the training priorities corresponding to the subsets, the base model is supervised and fine-tuned using different subsets in sequence to obtain the target video generation model;

[0043] The lower the resolution and / or frame rate of a subset, the higher the corresponding training priority.

[0044] Optionally, the animation video generating method further includes:

[0045] Acquire a benchmark data set, the benchmark data set comprising a plurality of benchmark animation video clips, each benchmark animation video clip having a corresponding text prompt;

[0046] Based on the target video generation model, the reference animation video clip and the corresponding text prompt, obtaining a target video;

[0047] Determining the visual quality and matching degree of the target video;

[0048] A benchmark test result is determined according to the visual quality and the matching degree, and the benchmark test result is used to evaluate the performance of the target visual model.

[0049] Optionally, the target video includes a plurality of frames, the visual quality includes visual smoothness, and the visual smoothness is used to represent the coherence between the plurality of frames;

[0050] Correspondingly, determining the visual quality of the target video includes:

[0051] Acquire visual features of the multiple frames;

[0052] Based on the visual features of the multiple frames, sequentially obtaining similarities between two adjacent frames in the multiple frames;

[0053] Based on the similarity between two adjacent frames in the multiple frames, a first average similarity is determined, where the first average similarity is used to represent the visual smoothness.

[0054] Optionally, the visual quality includes motion attributes; correspondingly, determining the visual quality of the target video includes:

[0055] inputting the corresponding text prompt into a pre-trained motion scoring model to obtain a first motion score through the motion scoring model;

[0056] Inputting the target video into the motion scoring model to obtain a second motion score;

[0057] A similarity between the first motion score and the second motion score is determined, the similarity being used to represent the motion attribute.

[0058] Optionally, the target video comprises a plurality of frames, and the visual quality comprises visual appeal;

[0059] Correspondingly, determining the visual quality of the target video includes:

[0060] determining a plurality of key frames from the plurality of frames;

[0061] Aesthetic scores of the plurality of key frames are obtained to determine an average aesthetic score, wherein the average aesthetic score is used to represent the visual appeal.

[0062] Optionally, the target video includes a plurality of frames, and the matching degree includes a matching degree between the corresponding text prompt and the target video;

[0063] Correspondingly, obtaining the matching degree of the target video includes:

[0064] Inputting the corresponding text prompt and the plurality of frames into a multimodal pre-trained model to obtain a plurality of matching scores, each matching score representing a degree of matching between a corresponding frame and the text prompt;

[0065] An average matching score is determined based on the multiple matching scores, where the average matching score is used to represent the matching degree between the corresponding text prompt and the target video.

[0066] Optionally, the target video includes a plurality of frames, the plurality of frames include a guide frame, the guide frame is used to guide the target video generation model to generate the target video, and the matching degree includes a matching degree between the guide frame and the target video;

[0067] Correspondingly, determining the matching degree of the target video includes:

[0068] Acquire a target text from the corresponding text prompt, wherein the target text is used to specify an image style of the guide frame;

[0069] Inputting the guide frame and the target text into a multimodal pre-trained model to obtain a first matching score;

[0070] Inputting the multiple frames and the target text into the multimodal pre-trained model to obtain multiple second matching scores, each second matching score corresponding to one frame;

[0071] Based on the similarities between the first matching score and the plurality of second matching scores, a second average similarity is determined, where the second average similarity is used to represent the similarity between the guide frame and the target video.

[0072] Optionally, the target video includes a plurality of frames, the target video includes a target character, the reference animation video clip includes a reference character, and the matching degree includes a matching degree between the target character and the reference character;

[0073] Correspondingly, determining the matching degree of the target video includes:

[0074] Acquire the features of the benchmark character based on the benchmark animation video clip;

[0075] Acquire a plurality of sample frames from the plurality of frames;

[0076] Inputting the plurality of sample frames into a pre-trained role detection model to obtain features of the plurality of target roles;

[0077] Based on the similarity between the feature of the reference character and the features of the plurality of target characters, a role similarity is determined, wherein the role similarity is used to indicate a degree of matching between the target character and the reference character.

[0078] Another aspect of the embodiments of the present application provides an animation video generating device, the device comprising:

[0079] An acquisition module, used to acquire a training data set, wherein the training data set includes a plurality of target animation video clips, each target animation video clip has a corresponding text prompt, and the target animation video clip is used to determine the guidance condition;

[0080] The supervised fine-tuning module is used to perform supervised fine-tuning on the pre-trained base model based on the multiple target animation video clips and the corresponding text prompts and guiding conditions to obtain a target video generation model, and the target video generation model is used to generate animation videos.

[0081] Another aspect of an embodiment of the present application provides a computer device, including:

[0082] at least one processor; and

[0083] a memory communicatively coupled to the at least one processor;

[0084] Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0085] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0086] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0087] The above technical solution adopted in the embodiment of the present application may have the following advantages:

[0088] Acquire a target guidance image and a target text prompt. Determine a spatiotemporal guidance condition based on the target guidance image. Based on the guidance of the spatiotemporal guidance condition and the target text prompt, generate a series of frames that are associated with the target guidance image in time and space and match the target text prompt to form the animation video clip. It can be seen that the technical solution of the embodiment of the present application can generate an animation video clip composed of a series of frames that are transformed in time and space relative to the target guidance image and match the target text under the guidance of the spatiotemporal guidance condition and the text prompt, which is suitable for a variety of video generation tasks and is particularly optimized for animation video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0090] Figure 1 The flowchart of the method for generating an animation video according to the first embodiment of the present application is schematically shown;

[0091] Figure 2 The overall flow chart of the method for generating an animated video according to the first embodiment of the present application is schematically shown;

[0092] Figure 3 The training flow chart of the video generation model according to the first embodiment of the present application is schematically shown;

[0093] Figure 4The benchmark test results of the video generation model according to the first embodiment of the present application are schematically shown;

[0094] Figure 5 Schematically showing the motion accuracy test results of the video generation model according to the first embodiment of the present application;

[0095] Figure 6 The effect diagram of the video generation model according to the first embodiment of the present application is schematically shown;

[0096] Figure 7 A block diagram schematically shows an animation video generating device according to the second embodiment of the present application; and

[0097] Figure 8 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0099] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0100] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0101] First, the following terms are explained:

[0102] Supervised Fine-Tuning (SFT): Use labeled data to perform targeted fine-tuning on an already trained model to optimize the model's performance in a specific domain or task.

[0103] UGC (User-Generated Content): User-generated content.

[0104] MMD: Video used to create virtual characters.

[0105] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below:

[0106] Producing videos through generative artificial intelligence is an efficient and low-cost technology that greatly reduces the difficulty of video generation. However, the applicant understands that the relevant video generation model still has problems such as uncontrollable generated video content and limited generated video types, which affects the user experience.

[0107] To this end, the embodiment of the present application provides a technical solution for constructing a video generation model. In this technical solution: (1) a method for constructing an animation dataset (training dataset) is provided, which provides more than 10 million high-quality text-video pairs, which can improve the generalization ability and generation accuracy of the video generation model and meet the diverse video generation needs; (2) a video generation model that supports multiple tasks is constructed, such as image-to-video generation, key frame interpolation, and local image-guided animation; (3) a spatiotemporal mask module is used to enhance the conditional generation ability of the video generation model to achieve motion control for specific areas, such as precise control of character movements; (4) a high-quality benchmark dataset is constructed, and a metric specifically for evaluating animation video generation is developed; (5) full-parameter supervised fine-tuning is performed on the basis of the pre-trained model to adapt to the animation video generation task; (6) image generation is incorporated into the multi-task training framework to improve the generalization ability of the video generation model in diverse artistic styles. See below for details.

[0108] The technical solutions of the present application are described below through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.

[0109] Embodiment 1

[0110] Figure 1 The flowchart of the method for generating an animation video according to the first embodiment of the present application is schematically shown.

[0111] like Figure 1 As shown, the animation video generation method may include steps S1 to S3, wherein:

[0112] Step S1, obtaining a target guidance image and a target text prompt.

[0113] Step S2, determining spatiotemporal guidance conditions based on the target guidance image.

[0114] Step S3, based on the spatiotemporal guidance conditions and the guidance of the target text prompt, generate a series of frames that are associated with the target guidance image in time and space and match the target text prompt to constitute the animation video clip.

[0115] The animation video generation method provided in this embodiment obtains a target guide image and a target text prompt. Based on the target guide image, a spatiotemporal guidance condition is determined. According to the guidance of the spatiotemporal guidance condition and the target text prompt, a series of frames that are associated with the target guide image in time and space and match the target text prompt are generated to constitute the animation video segment. It can be seen that the technical solution of the embodiment of the present application can generate an animation video segment composed of a series of frames that are transformed in time and space relative to the target guide image and match the target text under the guidance of the spatiotemporal guidance condition and the text prompt. It is suitable for a variety of video generation tasks and is particularly optimized for animation video generation.

[0116] The following combination Figure 1 , each step in steps S1 to S3 and other optional steps are described in detail.

[0117] Step S1, obtaining a target guidance image and a target text prompt.

[0118] The target guiding image can be any type of image, such as a real photo, an image of a virtual character, a screenshot of an animation or comic, or a video frame in a video. The number of target guiding images can be one or more. The target text prompt is a text description of the generated animation video clip, which can include scenes, events, characters, actions, artistic styles, emotions, etc. For example, the target guiding image can be a puppy. The target text prompt can be "the puppy is spinning in circles".

[0119] Step S2, determining spatiotemporal guidance conditions based on the target guidance image.

[0120] The spatiotemporal guidance conditions may be determined based on the target guidance image, and the spatiotemporal guidance conditions may include: the position of the target guidance image in the animation video segment (a series of frames) to be generated, or the motion area and non-motion area of ​​each frame.

[0121] Step S3, based on the spatiotemporal guidance conditions and the guidance of the target text prompt, generate a series of frames that are associated with the target guidance image in time and space and match the target text prompt to constitute the animation video clip.

[0122] Exemplarily, under the guidance of the spatiotemporal guidance conditions and the target text prompt, an animation video clip may be generated. The animation video clip may include a series of frames that are temporally and spatially associated with the target guidance image and matched with the target text prompt.

[0123] In this embodiment, under the guidance of spatiotemporal guidance conditions and text prompts, an animation video clip consisting of a series of frames that are transformed in time and space relative to the target guidance image and match the target text can be generated. This clip is suitable for a variety of video generation tasks and is especially optimized for animation video generation.

[0124] In an optional embodiment, the animation video clip can be generated by a video generation model. The video generation model includes: a spatiotemporal mask module, an encoding module and a transformer module. Specifically: the spatiotemporal mask module can determine the spatiotemporal guidance conditions based on the target guidance image. The spatiotemporal guidance conditions include a guidance sequence and a mask sequence. The guidance sequence can be used to specify the position of the target guidance image in the animation video clip, and the mask sequence can be used to define whether a series of frames are displayed in the animation video clip. The guidance sequence and the mask sequence are encoded by the encoding module to obtain a guidance feature sequence and a mask coding sequence. Based on the guidance feature sequence, the mask coding sequence, noise and the target text prompt, the animation video clip can be generated by the transformer module.

[0125] In this embodiment, the spatiotemporal guidance conditions can be determined through the video generation model, and the generated animation video can be controlled based on the spatiotemporal guidance conditions to ensure that the results are controllable and improve the quality of video generation.

[0126] In an optional embodiment, the video generation model is obtained by the following operations:

[0127] Step S100 , obtain a training data set, the training data set includes a plurality of target animation video clips, each target animation video clip has a corresponding text prompt, and the target animation video clip is used to determine the guidance condition.

[0128] The training data set may include multiple sets of training data, each set of training data includes a target animation video clip and its corresponding text prompt, i.e., a text-video pair. The target animation video clip may be collected and screened from a large number of animation videos according to preset rules (e.g., whether the resolution, number of frames, number of views, and interactive data meet the requirements). The corresponding text prompt may include key information such as the scene, event, character, artistic style, and emotion of the target animation video clip. An example scheme is provided below.

[0129] In an optional embodiment, step S100 may include:

[0130] Step S200: obtaining an original video set, where the original video set includes a plurality of animation videos.

[0131] Step S202: segment the multiple animation videos into scenes to obtain multiple animation video clips.

[0132] Step S204: determining the plurality of target animation video segments from the plurality of animation video segments based on a preset filtering rule.

[0133] Step S206: input the multiple target animation video clips into a pre-trained video annotation model to obtain multiple text prompts through the video annotation model, each text prompt corresponding to a target animation video clip.

[0134] Step S208: construct the training data set based on the multiple target animation video clips and the multiple text prompts.

[0135] For example, Figure 2 As shown, the original video set can be obtained based on large-scale, diversified, high-quality popular animated film and television works (such as animated series, animated movies, UGC animations, etc.) and virtual animations (such as MMD, virtual idol videos, etc.). The original video set can include multiple animated videos (such as more than 1 million). Scene detection and segmentation (Shot Detection & Split) of multiple animated videos can obtain multiple animated video clips. According to preset filtering rules (such as whether the number of frames, resolution, duration, content, etc. meet the requirements), multiple target animated video clips (such as more than 10 million) can be obtained from multiple animated video clips. Multiple target animated video clips are input into a pre-trained video annotation model (such as Qwen-VL2, which is a multimodal visual language model that supports image description, video description, etc.), and the corresponding text prompts of multiple target animated video clips can be obtained. Based on multiple target animated video clips and their corresponding text prompts, a large-scale and high-quality training data set (text-video pairs) can be constructed. The data sources used to construct the training data set in the embodiment of the present application are legal and public. The data have been anonymized and desensitized to ensure that no personal privacy or sensitive information is involved, and the acquisition and use of the data are strictly compliant.

[0136] In this embodiment, scene detection, video segmentation, preset filtering rules and video annotation are combined to generate multiple text-video pairs to construct a high-quality training data set, thereby improving the training efficiency of the model and the animation video generation capability.

[0137] In an optional embodiment, step S204 may include:

[0138] Step S300, obtaining the text coverage score, optical flow score, aesthetic score and total number of frames corresponding to each of the plurality of animation video clips. The text coverage score indicates the proportion of the text area in the animation video clip relative to the screen, the optical flow score indicates the motion attribute of the animation video clip, and the aesthetic score indicates the aesthetic quality of the animation video clip.

[0139] Step S302: Determine, as the target animation video segment, an animation video segment whose text coverage score is lower than a corresponding preset threshold, whose optical flow score and whose aesthetic score are both higher than respective preset thresholds, and whose total frame number is within a preset range.

[0140] Exemplarily, the animation video clip can be analyzed based on computer vision technology, and the corresponding text coverage score, optical flow score, aesthetic score and total number of frames can be calculated. Among them, the text coverage score can represent the proportion of the text area in the animation video clip relative to the screen. The lower the text coverage score, the higher the proportion of the text in the screen, and the lower the quality of the animation video clip, such as the end screen with a large number of subtitles. The optical flow score can represent the motion attribute of the animation video clip. The lower the optical flow score, the less motion the animation video clip has, and the lower the quality of the animation video clip, such as: with static images or fast flashback scenes (with low saturation, blur effects). The aesthetic score can represent the aesthetic quality of the animation video clip. The lower the aesthetic score, the worse the aesthetic quality of the animation video clip. As an example, the animation video clip with a text coverage score lower than the corresponding preset threshold, the optical flow score and the aesthetic score are both higher than the respective preset thresholds, and the total number of frames is within a preset range can be determined as the target animation video clip. Among them, based on the total number of frames, the target animation video clip with a time length between a preset range (such as 2s to 20s) can be obtained. Of course, the filtering rules may also be adjusted dynamically, for example, considering one of the indicators or a combination of multiple indicators to filter out a target animation video segment from multiple animation video segments.

[0141] In this embodiment, filtering rules are constructed based on four different dimensions to achieve accurate evaluation and effective screening of animation video clips, thereby obtaining high-quality target animation video clips that meet the standards.

[0142] The target animation video clip can be used to determine the guiding conditions. The guiding conditions may include guiding frames (one or more key frames), motion areas (areas with obvious changes in the video frame), non-motion areas (areas without obvious changes in the video frame), artistic styles, etc. The guiding conditions are used in the model training process as a guide to help the model learn how to generate expected outputs. The model training process will be exemplarily introduced in conjunction with multiple embodiments.

[0143] Step S102 Based on the multiple target animation video clips and the corresponding text prompts and guiding conditions, the pre-trained base model is supervised and fine-tuned to obtain a target video generation model, and the target video generation model is used to generate animation videos.

[0144] The base model can be a video generation model (e.g., a third-party general-purpose large model) pre-trained based on a large-scale dataset (e.g., more than 35 million video clips). The video clips can be natural videos (real world footage) or animated videos. Based on the base model, fully parameter supervised fine-tuning (SFT) can be performed through the training dataset to specifically adapt to the animation video generation task. An exemplary solution is provided below.

[0145] In an optional embodiment, the base model can be a text-to-video diffusion model based on DiT (Denoising Image Transformer), such as Figure 3 As shown, the base model may include a spatiotemporal mask module, an encoding module, and a transformer module. Based on the above three modules, the base model may implement the video generation task aligned with the text prompt. Exemplarily, step S102 may include:

[0146] Step S400, inputting the frame sequence into the spatiotemporal mask module to determine a guide frame from the multiple frames through the spatiotemporal mask module, and determining a guide sequence and a mask sequence based on the guide frame; wherein the guide frame is used to guide the converter module to generate a series of frames associated with the guide frame.

[0147] Step S402: input the guide sequence and the mask sequence into the encoding module to obtain a guide feature sequence and a mask encoding sequence through the encoding module.

[0148] Step S404, based on the guide feature sequence, the mask coding sequence, the noise and the corresponding text prompt, the transformer module is supervised and fine-tuned to obtain the target video generation model.

[0149] The target animation video clip includes a frame sequence consisting of multiple frames. The following takes one set of training data as an example to exemplify the model training process. Specifically, the frame sequence can be input into the spatiotemporal mask module, and the spatiotemporal mask module can determine the guide frame from multiple frames through uniform sampling or random sampling. The guide frame may include one or more frames in the frame sequence, for example: the first frame, the last frame, and other frames. The guide frame can be used to guide the transformer module to generate a video. Based on the guide frame, a guide sequence and a mask sequence can be determined, wherein the guide sequence can be used to specify the position of the guide frame, and the mask sequence can be used to control the guiding effect on the generated video. As Figure 3 As shown, the guide sequence includes at least one frame, and the mask sequence can be a binary sequence or an equal-value sequence.

[0150] The guide sequence and the mask sequence are input into the encoding module, which may include a 3D Casual VAE and a reprojection network. The 3D Casual VAE performs VAE encoding on the guide sequence to obtain a guide feature sequence G = {G1, G2, ..., Gpi, ..., Gn}, where Gpi corresponds to the encoded guide frame, and other positions Gj = 0, j ≠ pi. The mask sequence can be expressed as M = {M1, M2, ..., Mpi, ..., Mn}. The reprojection network can encode the mask sequence to obtain a mask encoding sequence Reproj (M).

[0151] The guided feature sequence G, the mask encoding sequence Reproj(M), the noise (noise t ) and the corresponding text prompt (T5) along the channel dimension, we can get:

[0152] X = Concat(Noise t , Reproj(M), G, T5);

[0153] Taking X as the input of the transformer module, supervised fine-tuning of the transformer module can be performed to finally obtain the target video generation model.

[0154] In this embodiment, a guide frame is selected from multiple frames and added to the model training process as a guide condition, so that the model can learn to generate an animated video that meets the text prompt under the guidance of the guide condition and realize key frame interpolation, that is, generate a video under the conditions of one or more arbitrary frames.

[0155] In an optional embodiment, step S400 may include:

[0156] Step S500: placing the guide frame at a target position of the guide sequence, and other positions of the guide sequence correspond to a series of frames generated by the converter module and associated with the guide frame.

[0157] Step S502: determine a first mask sequence based on the pilot sequence, wherein the first mask sequence includes a demasking mask corresponding to the pilot frame, the first mask sequence has the same length as the pilot sequence, and the first mask sequence also includes multiple demasking masks corresponding to the pilot frame.

[0158] Exemplarily, after determining the guide frame, the guide frame can be placed at the target position of the guide sequence, and the other positions of the guide sequence correspond to a series of frames associated with the guide frame generated by the converter module. That is, the other positions of the guide sequence can be empty values ​​or masked frames, and these positions are waiting for insertion. After the converter module generates a series of frames, these frames will be arranged in order according to the instructions of the guide sequence, thereby forming an animated video, such as Figure 3 As shown. Based on the guide sequence, the corresponding first mask sequence can be determined. The first mask sequence and the guide sequence have the same length, and the first mask sequence includes a demasking mask and multiple shielding masks, wherein the demasking mask (value is 1) corresponds to the guide frame position, and the shielding mask (value is 0) corresponds to the multiple shielding frame positions. The demasking mask and the shielding mask are the same size as the guide frame. The first mask sequence can be expressed as: M = {M1, M2, ..., Mpi, ..., Mn}, where Mpi = 1, Mj = 0, j≠pi. The first mask sequence can be used to represent the guide frame.

[0159] In this embodiment, based on the guide sequence and the first mask sequence, the guide frame can be effectively located, thereby improving the video generation capability of the model under the guide condition.

[0160] In an optional embodiment, step S400 may include:

[0161] Step S600: placing the guide frame at a target position of the guide sequence, and other positions of the guide sequence correspond to a series of frames generated by the converter module and associated with the guide frame.

[0162] Step S602, obtaining a target mask based on the frame sequence, wherein the target mask includes a masked area and a demasked area, and the target mask is used to control the motion area of ​​the series of frames, wherein the masked area corresponds to a non-motion area, and the demasked area corresponds to a motion area.

[0163] Step S604: determining a second mask sequence according to the guide sequence and the target mask, wherein the second mask sequence has the same length as the guide sequence, and the second mask sequence includes a plurality of the target masks.

[0164] Exemplarily, after determining the guide frame, the guide frame can be placed at the target position of the guide sequence to obtain a guide sequence. Based on the frame sequence, a target mask MF can be obtained, and the target mask is the same size as the guide frame. The target mask can be used to control the motion area of ​​the generated video. The target mask may include a shielded area (value is 0) and a non-shielded area (value is 1), wherein the shielded area corresponds to the non-motion area and the unmasked area corresponds to the motion area. Accordingly, the second mask sequence can be expressed as {MF, MF, ..., MF}, and the second mask sequence includes multiple target masks MF, and has the same length as the guide sequence.

[0165] In this embodiment, the motion area and non-motion area in the video generation can be effectively controlled by the target mask, thereby enhancing the dynamic consistency and motion control accuracy of the generated video and realizing local image-guided animation.

[0166] In an optional embodiment, step S602 may include:

[0167] Step S700: performing foreground detection on the first frame of the frame sequence to determine the foreground area of ​​the first frame.

[0168] Step S702: generating foreground masks corresponding to each of the plurality of frames based on the foreground area of ​​the first frame, wherein the foreground masks are used to represent the motion area and the non-motion area of ​​the corresponding frame.

[0169] Step S704: generating the target mask according to the foreground masks corresponding to the multiple frames.

[0170] Exemplarily, a foreground detector may be used to perform foreground detection on the first frame of a frame sequence to determine a foreground area in the first frame. The foreground area is the area where the character or main object in the frame image is located. The foreground area is tracked in subsequent frames, and a corresponding foreground mask is generated for each frame. The foreground mask can be used to represent the motion area and non-motion area of ​​the corresponding frame. The foreground masks of multiple frames are combined to create a unified mask, namely the target mask MF. The target mask MF represents the union of all foreground areas in the frame sequence.

[0171] In this embodiment, the moving area and the non-moving area in the video frame are effectively extracted through foreground detection and tracking, and a high-precision target mask is generated, thereby improving the dynamic accuracy and regional consistency in the generated video.

[0172] In some embodiments, the non-motion area of ​​the guide frame can be used as a video latent feature, and the video latent feature representation z0 is obtained by VAE encoding. The video latent feature z0 can be used as an input of the transformer module to ensure that the static area (non-motion area) in the generated video follows the guidance of the guide frame.

[0173] In an optional embodiment, step S102 may include:

[0174] Step S800: determining the resolution and frame rate corresponding to each of the plurality of target animation video clips.

[0175] Step S802: dividing the training data set into a plurality of subsets based on the resolutions and frame rates corresponding to the plurality of target animation video clips, wherein at least one of the resolution and frame rate corresponding to each subset is different from that of the other subsets.

[0176] Step S804: Determine the training priority corresponding to each subset according to the resolution and frame rate corresponding to each subset.

[0177] Step S806, according to the training priorities corresponding to the various subsets, use different subsets in turn to supervise and fine-tune the base model to obtain the target video generation model; wherein, the lower the resolution and / or frame rate of the subset, the higher the corresponding training priority.

[0178] Exemplarily, the training data set can be divided into multiple subsets according to the resolution (such as: 480p, 720p, 1080p, etc.) and frame rate (8fps, 16fps, 24fps, etc.) of the target animation video clip, and at least one of the resolution and frame rate corresponding to each subset is different from other subsets, such as: subset A (480p, 8fps), subset B (480p, 16fps), subset C (720p, 16fps), etc. Based on the resolution and frame rate corresponding to each subset, the training priority of each subset is determined. For example, the lower the resolution and / or frame rate of the subset, the higher the corresponding training priority, such as the training priority: subset A>subset B>subset C. Based on the training priority of each subset, the base model is supervised and fine-tuned using different subsets in turn, and the target video generation model can be obtained. For example: the model is first trained on subset A (480p, 8fps) for several cycles (such as 3) so that it can capture the basic spatiotemporal dynamics at a lower frame rate. Then, it is trained on subset B (480p, 16fps) for an additional number of epochs (e.g., 1.9) to enable it to refine temporal consistency and adapt to higher frame rates. Subsequently, the model can be fine-tuned on subset C (720p, 16fps) for a number of epochs (e.g., 2.3) to generate high-resolution, temporally coherent video outputs using the previously learned features.

[0179] In this embodiment, a training strategy from weak to strong is adopted, which can gradually improve the learning ability of the model at different resolutions and frame rates, thereby improving the model performance.

[0180] In some embodiments, the training data set can also be adjusted or more strictly screened based on the performance of the target video generation model. For example, the proportion of dialogue videos or videos with large motion in the training data set can be increased to obtain an ultra-high-quality data set (such as 1 million) for fine-tuning in the final stage, which can significantly improve the quality of high-resolution videos.

[0181] In some embodiments, the model can also be fine-tuned using animated video clips without subtitles and watermarks as a dataset. For example, the dataset can consist of 790,000 animated video clips, which are proportionally cropped to remove subtitles or have no subtitles at all. The dataset is used to perform full parameter supervised fine-tuning on the model. After 5.5k iterations, the target video generation model can effectively eliminate the generation of subtitles and watermarks, and the overall performance is improved.

[0182] In some embodiments, the target animation video clips in the training data set are of different lengths. Specifically, 2s to 8s target animation video clips can be used for model training to maximize data utilization. Through the variable length video hybrid training strategy, the model can generate videos of flexible lengths.

[0183] In some embodiments, image generation can be incorporated into a multi-task training framework. That is, on the basis of the training data set, multiple pictures with various artistic styles are added for model training to improve the generalization ability of the model in diverse artistic styles, which can effectively reduce the quality gap of video generation caused by the difference in the style of the guide frame.

[0184] The above-mentioned multiple embodiments exemplarily introduce the training process of the model. The following will exemplarily introduce the evaluation process of the model through multiple embodiments to better construct a video generation model.

[0185] In an optional embodiment, the animation video generation method may further include:

[0186] Step S900: Acquire a benchmark data set, where the benchmark data set includes a plurality of benchmark animation video clips, each benchmark animation video clip having a corresponding text prompt.

[0187] Step S902, acquiring a target video based on the target video generation model, the reference animation video clip and the corresponding text prompt.

[0188] Step S904: determine the visual quality and matching degree of the target video.

[0189] Step S906: determining a benchmark test result according to the visual quality and the matching degree, wherein the benchmark test result is used to evaluate the performance of the target visual model.

[0190] Exemplarily, the benchmark dataset may include multiple benchmark animation video clips (e.g., 943, including 857 2D benchmark animation video clips and 91 3D benchmark animation video clips). These benchmark animation video clips have different action labels, such as: talking, walking & running, eating, etc. There may be more than 100 action labels, and each action label may correspond to 10 to 30 benchmark animation video clips. The text prompts corresponding to the benchmark animation video clips can also be obtained through the video annotation model.

[0191] The target video can be generated by inputting the text prompt into the trained target video generation model. The visual quality (e.g., visual smoothness, motion properties, visual attractiveness) and matching (text-video, image-video, character consistency) of the target video can be determined to determine the benchmark results, which can be used to evaluate the performance of the target visual model.

[0192] In this embodiment, a comprehensive benchmark dataset is constructed and a new metric is developed to accurately evaluate the performance of the model.

[0193] The following will introduce the metric standard by way of examples.

[0194] In an optional embodiment, the visual quality may include visual smoothness, and the visual smoothness is used to represent the coherence between the multiple frames. Correspondingly, step S904 may include:

[0195] Step S1000, obtaining visual features of the multiple frames.

[0196] Step S1002: based on the visual features of the multiple frames, sequentially obtain the similarity between two adjacent frames in the multiple frames.

[0197] Step S1004: determining a first average similarity based on the similarity between two adjacent frames in the plurality of frames, where the first average similarity is used to represent the visual smoothness.

[0198] For example, the visual features of multiple frames can be obtained through the CLIP model, and the similarity (such as cosine similarity) between two adjacent frames can be calculated in sequence. Based on the similarity between two adjacent frames in the multiple frames, a first average similarity can be obtained, and the first average similarity can reflect visual smoothness.

[0199] The calculation method can be expressed as follows:

[0200]

[0201] Among them, I i represents frame, N represents the total number of frames, CLIP represents feature extractor, and Cos represents cosine similarity function.

[0202] In an optional embodiment, the visual quality may include motion attributes, and the motion attributes may be the motion amplitude of the main elements (characters, objects, etc.) in the target video. Correspondingly, step S904 may include:

[0203] Step S1100: input the corresponding text prompt into a pre-trained motion scoring model to obtain a first motion score through the motion scoring model.

[0204] Step S1102: input the target video into the motion scoring model to obtain a second motion score.

[0205] Step S1104: determining a similarity between the first motion score and the second motion score, wherein the similarity is used to represent the motion attribute.

[0206] For example, 10 million animation video clips and corresponding motion descriptions can be collected, and the motion descriptions can be divided into 6 levels (from static to significant motion) to fine-tune the basic model (such as CLIP), and finally obtain the motion scoring model. The text prompt and the target video are scored by the motion scoring model to obtain a first motion score and a second motion score. The similarity between the first motion score and the second motion score is calculated, and the similarity can represent the motion attribute.

[0207] The calculation formula can be expressed as follows:

[0208] S motion = Cos(MCLIP(V), MCLIP(T m ));

[0209] Among them, MCLIP represents the motion scoring model, V represents the target video, and Tm represents the motion-related text segment in the text prompt.

[0210] In an optional embodiment, the visual quality may include visual appeal. Correspondingly, step S904 may include:

[0211] Step S1200: determining a plurality of key frames from the plurality of frames.

[0212] Step S1202 , obtaining aesthetic scores of the multiple key frames to determine an average aesthetic score, wherein the average aesthetic score is used to represent the visual appeal.

[0213] For example, key frames from multiple frames can be collected by a key frame extraction method, and their aesthetic scores can be calculated. An average aesthetic score is further obtained to indicate visual appeal.

[0214] The calculation formula can be expressed as follows:

[0215]

[0216] Among them, KeyFrm represents the key frame extraction method, Aes represents the aesthetic evaluation method, and K represents the number of key frames.

[0217] In an optional embodiment, the matching degree may include the matching degree between the corresponding text prompt and the target video. Correspondingly, step S904 may include:

[0218] Step S1300: input the corresponding text prompt and the multiple frames into a multimodal pre-trained model to obtain multiple matching scores, each matching score representing a matching degree between a corresponding frame and the text prompt.

[0219] Step S1302: determining an average matching score based on the multiple matching scores, wherein the average matching score is used to indicate a matching degree between the corresponding text prompt and the target video.

[0220] Exemplarily, a matching score between a text prompt and each frame can be determined by a multimodal pre-trained model to characterize the matching degree. An average matching score can be determined based on multiple matching scores to represent the matching degree between text and video.

[0221] The calculation formula can be expressed as follows:

[0222]

[0223] Where N represents the total number of frames and T represents the text prompt.

[0224] In an optional embodiment, the matching degree may include the matching degree between the guide frame and the target video. Correspondingly, the determination step S904 includes:

[0225] Step S1400: acquiring a target text from the corresponding text prompt, wherein the target text is used to specify an image style of the guide frame.

[0226] Step S1402: input the guide frame and the target text into a multimodal pre-training model to obtain a first matching score.

[0227] Step S1404: input the multiple frames and the target text into the multimodal pre-training model to obtain multiple second matching scores, each second matching score corresponding to one frame.

[0228] Step S1406: determining a second average similarity based on the similarities between the first matching score and the plurality of second matching scores, wherein the second average similarity is used to represent the similarity between the guide frame and the target video.

[0229] Exemplarily, a target text is extracted from a text prompt, where the target text is a text segment for specifying the image style of the guide frame. A first matching score between the guide frame and the target text, and a second matching score between each frame and the target text can be determined by a multimodal pre-trained model. Based on the similarity between the first matching score and each second matching score, a second average similarity can be determined. The second average similarity can represent the matching degree between the guide frame and the video.

[0230] The calculation formula can be expressed as follows:

[0231]

[0232] Among them, Ip represents the guide frame and Ts represents the target text.

[0233] In an optional embodiment, the target video includes a target character, the reference animation video clip includes a reference character, and the matching degree may include a matching degree between the target character and the reference character.

[0234] Correspondingly, step S904 may include:

[0235] Step S1500: Acquire the features of the benchmark character based on the benchmark animation video clip.

[0236] Step S1502: Acquire multiple sample frames from the multiple frames.

[0237] Step S1504: input the multiple sample frames into a pre-trained role detection model to obtain features of multiple target roles.

[0238] Step S1506: determining role similarity based on the similarity between the feature of the benchmark role and the features of the plurality of target roles, wherein the role similarity is used to indicate the degree of matching between the target role and the benchmark role.

[0239] Exemplarily, GroundingDino and SAM can be applied to detect, segment and identify each frame (or multiple sample frames) of the target video to extract the mask of the target character in each frame (or multiple sample frames). The mask of the target character is input into the character detection model to obtain the features of multiple target characters. The features of the multiple target characters and the features of the pre-stored benchmark characters (obtained based on the benchmark animation video clip) are similarly calculated to obtain the character similarity, which can represent the character consistency, that is, the matching degree between the target character and the benchmark character.

[0240] The calculation formula can be expressed as follows:

[0241]

[0242] Among them, S represents the number of sample frames, Mi represents the mask obtained from GroundingDino and SAM, and feac represents the features of the benchmark character.

[0243] In some embodiments, in addition to the above quantitative analysis, the model can also be evaluated by humans, for example, by asking 20 volunteers to give scores based on the above multiple indicators (such as visual smoothness, motion attributes, etc.).

[0244] Figure 4 The benchmark test results obtained by evaluating the trained target video generation model according to the evaluation benchmark provided in the embodiment of the present application are provided. It can be seen that the human evaluation results and the benchmark test results are highly correlated, indicating that the evaluation benchmark provided in the embodiment of the present application is feasible and effective and can be used for standardized testing. In addition, single-frame guidance can achieve better video generation effects, and adding more guide frames can further improve character consistency and motion stability, and produce animation videos with a larger motion range and more realistic. Figure 5 and Figure 6 As shown, the motion area control of the generated video can effectively control the movable area and non-movable area (character and background) of the video, improve the video animation effect, and ensure alignment with various storylines. Even without motion control, the video generation model provided in the embodiment of the present application still shows a certain degree of control. Among them, AnimateAnything is a controllable video generation technology.

[0245] Embodiment 2

[0246] Figure 7 The block diagram of the animation video generating device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 7 As shown, the apparatus 1000 may include: an acquisition module 1100, a determination module 1200 and a generation module 1300, wherein:

[0247] An acquisition module 1100 is used to acquire a target guide image and a target text prompt;

[0248] A determination module 1200, configured to determine a spatiotemporal guidance condition based on the target guidance image;

[0249] The generation module 1300 is used to generate a series of frames that are associated with the target guidance image in time and space and match the target text prompt according to the spatiotemporal guidance conditions and the guidance of the target text prompt to form the animation video clip.

[0250] As an optional embodiment, the method is implemented by a pre-trained video generation model, the video generation model comprising: a spatiotemporal mask module, an encoding module and a transformer module, wherein:

[0251] Determining a spatiotemporal guidance condition based on the target guidance image by the spatiotemporal mask module, wherein the spatiotemporal guidance condition includes a guidance sequence and a mask sequence, wherein the guidance sequence is used to specify a position of the target guidance image in the animation video segment, and the mask sequence is used to define whether the series of frames are displayed in the animation video segment;

[0252] Encoding the guide sequence and the mask sequence through an encoding module to obtain a guide feature sequence and a mask encoding sequence;

[0253] The animated video clip is generated by the transformer module based on the guiding feature sequence, the mask encoding sequence, noise and the target text prompt.

[0254] As an optional embodiment, the video generation model is obtained by the following operations:

[0255] Acquire a training data set, the training data set comprising a plurality of target animation video clips, each target animation video clip having a corresponding text prompt, the target animation video clip being used to determine a guidance condition;

[0256] Based on the multiple target animation video clips and the corresponding text prompts and guiding conditions, the pre-trained base model is supervised and fine-tuned to obtain a target video generation model, and the target video generation model is used to generate animation videos.

[0257] As an optional embodiment, obtaining a training data set includes:

[0258] Acquire an original video set, wherein the original video set includes multiple animation videos;

[0259] Dividing the multiple animation videos into scenes to obtain multiple video clips;

[0260] Based on a preset filtering rule, determining the multiple target animation video segments from the multiple video segments;

[0261] Inputting the plurality of target animation video clips into a pre-trained video annotation model to obtain a plurality of text prompts through the video annotation model, each text prompt corresponding to a target animation video clip;

[0262] The training dataset is constructed based on the multiple target animation video clips and the multiple text prompts.

[0263] As an optional embodiment, the base model includes: a spatiotemporal mask module, an encoding module and a transformer module, and the target animation video clip includes a frame sequence consisting of a plurality of frames;

[0264] Correspondingly, based on the multiple target animation video clips and the corresponding text prompts and guiding conditions, the pre-trained base model is supervised and fine-tuned to obtain a target video generation model, including:

[0265] Inputting the frame sequence into the spatiotemporal mask module, so as to determine a guide frame from the plurality of frames through the spatiotemporal mask module, and determining a guide sequence and a mask sequence based on the guide frame; wherein the guide frame is used to guide the transformer module to generate a series of frames associated with the guide frame;

[0266] Inputting the guide sequence and the mask sequence into the encoding module to obtain a guide feature sequence and a mask encoding sequence through the encoding module;

[0267] Based on the guiding feature sequence, the mask encoding sequence, noise and corresponding text prompts, the transformer module is supervised fine-tuned to obtain the target video generation model.

[0268] As an optional embodiment, determining a guide sequence and a mask sequence based on the guide frame includes:

[0269] placing the guide frame at a target position of the guide sequence, other positions of the guide sequence corresponding to a series of frames associated with the guide frame generated by the converter module;

[0270] A first mask sequence is determined based on the guide sequence, the first mask sequence comprising a demasking mask, the demasking mask corresponding to the guide frame.

[0271] As an optional embodiment, determining a guide sequence and a mask sequence based on the guide frame includes:

[0272] placing the guide frame at a target position of the guide sequence, other positions of the guide sequence corresponding to a series of frames associated with the guide frame generated by the converter module;

[0273] Acquire a target mask based on the frame sequence, the target mask comprising a masking area and a demasking area, the target mask being used to control a motion area of ​​the series of frames, wherein the masking area corresponds to a non-motion area, and the demasking area corresponds to a motion area;

[0274] A second mask sequence is determined according to the guide sequence and the target mask, the second mask sequence and the guide sequence have the same length, and the second mask sequence includes a plurality of the target masks.

[0275] As an optional embodiment, acquiring a target mask based on the frame sequence includes:

[0276] Performing foreground detection on a first frame of the frame sequence to determine a foreground area of ​​the first frame;

[0277] Based on the foreground area of ​​the first frame, generating a foreground mask corresponding to each of the plurality of frames, wherein the foreground mask is used to represent a motion area and a non-motion area of ​​the corresponding frame;

[0278] The target mask is generated according to the foreground masks corresponding to each of the multiple frames.

[0279] As an optional embodiment, based on the multiple target animation video clips and the corresponding text prompts and guiding conditions, the pre-trained base model is supervised fine-tuned to obtain a target video generation model, including:

[0280] Determine the resolution and frame rate corresponding to each of the plurality of target animation video clips;

[0281] Based on the resolutions and frame rates corresponding to the multiple target animation video clips, the training data set is divided into multiple subsets, wherein at least one of the resolution and frame rate corresponding to each subset is different from that of other subsets;

[0282] Determine the training priority of each subset according to the resolution and frame rate of each subset;

[0283] According to the training priorities corresponding to the subsets, the base model is supervised and fine-tuned using different subsets in sequence to obtain the target video generation model;

[0284] The lower the resolution and / or frame rate of a subset, the higher the corresponding training priority.

[0285] As an optional embodiment, the device 1000 is further used for:

[0286] Acquire a benchmark data set, the benchmark data set comprising a plurality of benchmark animation video clips, each benchmark animation video clip having a corresponding text prompt;

[0287] Based on the target video generation model, the reference animation video clip and the corresponding text prompt, obtaining a target video;

[0288] Determining the visual quality and matching degree of the target video;

[0289] A benchmark test result is determined according to the visual quality and the matching degree, and the benchmark test result is used to evaluate the performance of the target visual model.

[0290] As an optional embodiment, the target video includes multiple frames, the visual quality includes visual smoothness, and the visual smoothness is used to represent the coherence between the multiple frames;

[0291] Correspondingly, determining the visual quality of the target video includes:

[0292] Acquire visual features of the multiple frames;

[0293] Based on the visual features of the multiple frames, sequentially obtaining similarities between two adjacent frames in the multiple frames;

[0294] Based on the similarity between two adjacent frames in the multiple frames, a first average similarity is determined, where the first average similarity is used to represent the visual smoothness.

[0295] As an optional embodiment, the visual quality includes motion attributes; correspondingly, determining the visual quality of the target video includes:

[0296] inputting the corresponding text prompt into a pre-trained motion scoring model to obtain a first motion score through the motion scoring model;

[0297] Inputting the target video into the motion scoring model to obtain a second motion score;

[0298] A similarity between the first motion score and the second motion score is determined, the similarity being used to represent the motion attribute.

[0299] As an optional embodiment, the target video includes a plurality of frames, and the visual quality includes visual appeal;

[0300] Correspondingly, determining the visual quality of the target video includes:

[0301] determining a plurality of key frames from the plurality of frames;

[0302] Aesthetic scores of the plurality of key frames are obtained to determine an average aesthetic score, wherein the average aesthetic score is used to represent the visual appeal.

[0303] As an optional embodiment, the target video includes a plurality of frames, and the matching degree includes a matching degree between the corresponding text prompt and the target video;

[0304] Correspondingly, obtaining the matching degree of the target video includes:

[0305] Inputting the corresponding text prompt and the plurality of frames into a multimodal pre-trained model to obtain a plurality of matching scores, each matching score representing a degree of matching between a corresponding frame and the text prompt;

[0306] An average matching score is determined based on the multiple matching scores, where the average matching score is used to represent the matching degree between the corresponding text prompt and the target video.

[0307] As an optional embodiment, the target video includes multiple frames, the multiple frames include a guide frame, the guide frame is used to guide the target video generation model to generate the target video, and the matching degree includes the matching degree between the guide frame and the target video;

[0308] Correspondingly, determining the matching degree of the target video includes:

[0309] Acquire a target text from the corresponding text prompt, wherein the target text is used to specify an image style of the guide frame;

[0310] Inputting the guide frame and the target text into a multimodal pre-trained model to obtain a first matching score;

[0311] Inputting the multiple frames and the target text into the multimodal pre-trained model to obtain multiple second matching scores, each second matching score corresponding to one frame;

[0312] Based on the similarities between the first matching score and the plurality of second matching scores, a second average similarity is determined, where the second average similarity is used to represent the similarity between the guide frame and the target video.

[0313] As an optional embodiment, the target video includes a plurality of frames, the target video includes a target character, the reference animation video clip includes a reference character, and the matching degree includes a matching degree between the target character and the reference character;

[0314] Correspondingly, determining the matching degree of the target video includes:

[0315] Acquire the features of the benchmark character based on the benchmark animation video clip;

[0316] Acquire a plurality of sample frames from the plurality of frames;

[0317] Inputting the plurality of sample frames into a pre-trained role detection model to obtain features of the plurality of target roles;

[0318] Based on the similarity between the feature of the reference character and the features of the plurality of target characters, a role similarity is determined, wherein the role similarity is used to indicate a degree of matching between the target character and the reference character.

[0319] Embodiment 3

[0320] Figure 8 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the animation video generation method according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 8 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them:

[0321] The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 may also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as the program code of the animation video generation method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0322] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0323] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0324] It should be pointed out that Figure 8 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0325] In this embodiment, the animation video generation method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0326] Embodiment 4

[0327] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the animation video generation method in the embodiment are implemented.

[0328] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEP ROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the animation video generation method in the embodiment, etc. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0329] Embodiment 5

[0330] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0331] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0332] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating an animation video, characterized in that: The method comprises: Obtain target guidance images and target text prompts; determining spatiotemporal guidance conditions based on the target guidance image; According to the spatiotemporal guidance conditions and the guidance of the target text prompt, a series of frames associated with the target guidance image in time and space and matched with the target text prompt are generated to constitute the animation video clip.

2. The method according to claim 1, characterized in that The method is implemented by a pre-trained video generation model, which includes: a spatiotemporal mask module, an encoding module and a transformer module, wherein: Determining a spatiotemporal guidance condition based on the target guidance image by the spatiotemporal mask module, wherein the spatiotemporal guidance condition includes a guidance sequence and a mask sequence, wherein the guidance sequence is used to specify a position of the target guidance image in the animation video segment, and the mask sequence is used to define whether the series of frames are displayed in the animation video segment; Encoding the guide sequence and the mask sequence through an encoding module to obtain a guide feature sequence and a mask encoding sequence; The animated video clip is generated by the transformer module based on the guiding feature sequence, the mask encoding sequence, noise and the target text prompt.

3. The method according to claim 2, characterized in that The video generation model is obtained by the following operations: Acquire a training data set, the training data set comprising a plurality of target animation video clips, each target animation video clip having a corresponding text prompt, the target animation video clip being used to determine a guidance condition; Based on the multiple target animation video clips and the corresponding text prompts and guiding conditions, the pre-trained base model is supervised and fine-tuned to obtain a target video generation model, and the target video generation model is used to generate animation videos.

4. The method according to claim 3, characterized in that Get the training dataset, including: Acquire an original video set, wherein the original video set includes multiple animation videos; Dividing the multiple animation videos into scenes to obtain multiple video clips; Based on a preset filtering rule, determining the multiple target animation video segments from the multiple video segments; Inputting the plurality of target animation video clips into a pre-trained video annotation model to obtain a plurality of text prompts through the video annotation model, each text prompt corresponding to a target animation video clip; The training dataset is constructed based on the multiple target animation video clips and the multiple text prompts.

5. The method according to claim 3, characterized in that: The base model includes: a spatiotemporal mask module, an encoding module and a transformer module, and the target animation video clip includes a frame sequence composed of multiple frames; Correspondingly, based on the multiple target animation video clips and the corresponding text prompts and guiding conditions, the pre-trained base model is supervised and fine-tuned to obtain a target video generation model, including: Inputting the frame sequence into the spatiotemporal mask module, so as to determine a guide frame from the plurality of frames through the spatiotemporal mask module, and determining a guide sequence and a mask sequence based on the guide frame; wherein the guide frame is used to guide the transformer module to generate a series of frames associated with the guide frame; Inputting the guide sequence and the mask sequence into the encoding module to obtain a guide feature sequence and a mask encoding sequence through the encoding module; Based on the guiding feature sequence, the mask encoding sequence, noise and corresponding text prompts, the transformer module is supervised fine-tuned to obtain the target video generation model.

6. The method according to claim 5, characterized in that Determining a guide sequence and a mask sequence based on the guide frame includes: placing the guide frame at a target position of the guide sequence, other positions of the guide sequence corresponding to a series of frames associated with the guide frame generated by the converter module; A first mask sequence is determined based on the guide sequence, the first mask sequence comprising a demasking mask, the demasking mask corresponding to the guide frame.

7. The method according to claim 5, characterized in that Determining a guide sequence and a mask sequence based on the guide frame includes: placing the guide frame at a target position of the guide sequence, other positions of the guide sequence corresponding to a series of frames associated with the guide frame generated by the converter module; Acquire a target mask based on the frame sequence, the target mask comprising a masking area and a demasking area, the target mask being used to control a motion area of ​​the series of frames, wherein the masking area corresponds to a non-motion area, and the demasking area corresponds to a motion area; A second mask sequence is determined according to the guide sequence and the target mask, the second mask sequence and the guide sequence have the same length, and the second mask sequence includes a plurality of the target masks.

8. The method according to claim 7, characterized in that Acquiring a target mask based on the frame sequence includes: Performing foreground detection on a first frame of the frame sequence to determine a foreground area of ​​the first frame; Based on the foreground area of ​​the first frame, generating a foreground mask corresponding to each of the plurality of frames, wherein the foreground mask is used to represent a motion area and a non-motion area of ​​the corresponding frame; The target mask is generated according to the foreground masks corresponding to each of the multiple frames.

9. The method according to any one of claims 3 to 8, characterized in that: Based on the multiple target animation video clips and the corresponding text prompts and guiding conditions, the pre-trained base model is fine-tuned to obtain a target video generation model, including: Determine the resolution and frame rate corresponding to each of the plurality of target animation video clips; Based on the resolutions and frame rates corresponding to the multiple target animation video clips, the training data set is divided into multiple subsets, wherein at least one of the resolution and frame rate corresponding to each subset is different from that of other subsets; Determine the training priority of each subset according to the resolution and frame rate of each subset; According to the training priorities corresponding to the subsets, different subsets are used in turn to perform supervised fine-tuning on the base model to obtain the target video generation model.

10. The method according to any one of claims 3 to 8, characterized in that: Also includes: Acquire a benchmark data set, the benchmark data set comprising a plurality of benchmark animation video clips, each benchmark animation video clip having a corresponding text prompt; Based on the target video generation model, the reference animation video clip and the corresponding text prompt, obtaining a target video; Determining the visual quality and matching degree of the target video; A benchmark test result is determined according to the visual quality and the matching degree, and the benchmark test result is used to evaluate the performance of the target visual model.

11. The method according to claim 10, characterized in that The target video includes a plurality of frames, the visual quality includes visual smoothness, and the visual smoothness is used to represent the coherence between the plurality of frames; Correspondingly, determining the visual quality of the target video includes: Acquire visual features of the multiple frames; Based on the visual features of the multiple frames, sequentially obtaining similarities between two adjacent frames in the multiple frames; Based on the similarity between two adjacent frames in the multiple frames, a first average similarity is determined, where the first average similarity is used to represent the visual smoothness.

12. The method according to claim 10, characterized in that The visual quality includes motion attributes; correspondingly, determining the visual quality of the target video includes: inputting the corresponding text prompt into a pre-trained motion scoring model to obtain a first motion score through the motion scoring model; Inputting the target video into the motion scoring model to obtain a second motion score; A similarity between the first motion score and the second motion score is determined, the similarity being used to represent the motion attribute.

13. The method according to claim 10, characterized in that The target video includes a plurality of frames, and the visual quality includes visual appeal; Correspondingly, determining the visual quality of the target video includes: determining a plurality of key frames from the plurality of frames; Aesthetic scores of the plurality of key frames are obtained to determine an average aesthetic score, wherein the average aesthetic score is used to represent the visual appeal.

14. The method according to claim 10, characterized in that The target video includes a plurality of frames, and the matching degree includes a matching degree between the corresponding text prompt and the target video; Correspondingly, obtaining the matching degree of the target video includes: Inputting the corresponding text prompt and the plurality of frames into a multimodal pre-trained model to obtain a plurality of matching scores, each matching score representing a degree of matching between a corresponding frame and the text prompt; An average matching score is determined based on the multiple matching scores, where the average matching score is used to represent the matching degree between the corresponding text prompt and the target video.

15. The method according to claim 10, characterized in that The target video includes a plurality of frames, the plurality of frames include a guide frame, the guide frame is used to guide the target video generation model to generate the target video, and the matching degree includes the matching degree between the guide frame and the target video; Correspondingly, determining the matching degree of the target video includes: Acquire a target text from the corresponding text prompt, wherein the target text is used to specify an image style of the guide frame; Inputting the guide frame and the target text into a multimodal pre-trained model to obtain a first matching score; Inputting the multiple frames and the target text into the multimodal pre-trained model to obtain multiple second matching scores, each second matching score corresponding to one frame; Based on the similarities between the first matching score and the plurality of second matching scores, a second average similarity is determined, where the second average similarity is used to represent the similarity between the guide frame and the target video.

16. The method according to claim 10, characterized in that The target video includes a plurality of frames, the target video includes a target character, the reference animation video clip includes a reference character, and the matching degree includes a matching degree between the target character and the reference character; Correspondingly, determining the matching degree of the target video includes: Acquire the features of the benchmark character based on the benchmark animation video clip; Acquire a plurality of sample frames from the plurality of frames; Inputting the plurality of sample frames into a pre-trained role detection model to obtain features of the plurality of target roles; Based on the similarity between the feature of the reference character and the features of the plurality of target characters, a role similarity is determined, wherein the role similarity is used to indicate a degree of matching between the target character and the reference character.

17. An animation video generating device, characterized in that: The device comprises: An acquisition module, used to acquire a target guiding image and a target text prompt; A determination module, configured to determine a spatiotemporal guidance condition based on the target guidance image; A generation module is used to generate a series of frames that are associated with the target guidance image in time and space and match the target text prompt according to the guidance of the spatiotemporal guidance conditions and the target text prompt to constitute the animation video clip.

18. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 16 is implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 16 are implemented.

Images