Video generation model training method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a video generation model training method and device, electronic equipment and storage medium, belonging to the technical field of artificial intelligence. The method comprises: obtaining sample special effect videos and special effect prompt words of a plurality of special effects, wherein the special effect prompt words are used to indicate special effects in a video to be generated by a video generation model; for any special effect in the plurality of special effects, processing a first frame image in the sample special effect video of the special effect and the special effect prompt word of the special effect through the video generation model to obtain a first special effect video, wherein the first frame image is used to indicate a subject in the video to be generated by the video generation model; and training the video generation model based on the first special effect videos and the sample special effect videos corresponding to the plurality of special effects. The above technical solution improves the generation efficiency of special effect videos on the basis of ensuring the generation effect of special effect videos.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and particularly relates to a video generation model training method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the way of creating special effect videos based on artificial intelligence (AI) technology is becoming more and more popular. Among them, a trained video generation model is usually used to make special effect videos. The performance of the video generation model often depends on the training method of the video generation.

[0003] In related technologies, the commonly used training method is: inputting a base video, a piece of description information about a special effect, and a region prompt into the video generation model, and the video generation model generates a special effect in the specified region in the base video according to the description information, thereby obtaining a special effect video; and the model is trained by taking the special effect video more natural and conforming to the visual rules of the real world as the goal.

[0004] However, the video generation model trained by using the above technical solution requires the user to describe the special effect during the use of the model, so as to generate a corresponding special effect video. However, the description information about the special effect is usually complex, and it is difficult for general users to quickly input reasonable and accurate description information, thereby resulting in low generation efficiency of the special effect video. SUMMARY

[0005] The present disclosure provides a video generation model training method, device, electronic device, and storage medium, which improves the generation efficiency of the special effect video on the basis of ensuring the generation effect of the special effect video. The technical solution of the present disclosure is as follows.

[0006] According to an aspect of an embodiment of the present disclosure, a video generation model training method is provided, comprising:

[0007] Obtaining sample special effect videos and special effect prompt words of a plurality of special effects, wherein the special effect prompt words are used to indicate special effects in a video to be generated by the video generation model;

[0008] For any special effect in the plurality of special effects, processing a first frame image in the sample special effect video of the special effect and the special effect prompt word of the special effect by using the video generation model to obtain a first special effect video, wherein the first frame image is used to indicate a subject in the video to be generated by the video generation model;

[0009] Training the video generation model based on the first special effect videos and the sample special effect videos corresponding to the plurality of special effects.

[0010] According to another aspect of the embodiments of the present disclosure, a training device of a video generation model is provided, comprising:

[0011] An acquisition module configured to acquire a plurality of special effect videos each corresponding to a sample special effect and a special effect prompt word for indicating a special effect in a video to be generated by the video generation model;

[0012] A first processing module configured to, for any special effect in the plurality of special effects, process a first frame image in the sample special effect video of the special effect and the special effect prompt word of the special effect by the video generation model, to obtain a first special effect video, the first frame image being used to indicate a subject in the video to be generated by the video generation model.

[0013] A training module configured to train the video generation model based on the first special effect videos and the sample special effect videos corresponding to the plurality of special effects.

[0014] In some embodiments, the first processing module comprises:

[0015] A first encoding unit configured to, for any special effect in the plurality of special effects, encode the first frame image in the sample special effect video of the special effect by an image encoder in the video generation model, to obtain first image encoding features;

[0016] A second encoding unit configured to encode the special effect prompt word of the special effect by a text encoder in the video generation model, to obtain text encoding features;

[0017] A processing unit configured to process the first image encoding features and the text encoding features by a video generation network in the video generation model, to obtain the first special effect video.

[0018] In some embodiments, the processing unit is configured to process the first image encoding features and the text encoding features by the video generation network in the video generation model, to obtain first video features, the first video features being used to indicate the first special effect video.

[0019] The training module is configured to, for any special effect in the plurality of special effects, determine a first loss based on the first video features and sample video features of the sample special effect video of the special effect, the first loss being used to represent a difference between the first video features and the sample video features; and train the video generation model with a target of minimizing the first loss.

[0020] In some embodiments, the training module is configured to perform freezing parameters in structures other than a text encoder in the video generation model, and adjusting parameters of the text encoder in the video generation model.

[0021] In some embodiments, the apparatus further includes:

[0022] The special effect encoding module is configured to perform, for any special effect in the plurality of special effects, encoding, by a special effect encoder in the video generation model, sample video features of a sample special effect video of the special effect, to obtain special effect features, the special effect features being used to indicate attributes of the special effect in the sample special effect video.

[0023] The first processing module is further configured to perform processing, by the video generation model, on the special effect features, a first frame image in a sample special effect video of the special effect, and a special effect prompt word of the special effect, to obtain a second special effect video.

[0024] The training module is further configured to perform retraining the video generation model based on the second special effect videos and the sample special effect videos corresponding to the plurality of special effects.

[0025] In some embodiments, the first processing module is configured to perform, for any special effect in the plurality of special effects, processing, by an attention layer in the special effect encoder, on the sample video features, to obtain query features, key features, and value features of the attention layer, the query features and the key features being used to indicate positions of the special effect in the sample special effect video, and the value features being used to indicate attributes of the special effect; and fusing the query features, the key features, and the value features, to obtain the special effect features.

[0026] In some embodiments, the first processing module is configured to perform processing, by a video generation network in the video generation model, on the special effect features, first image encoding features corresponding to the first frame image, and text encoding features corresponding to the special effect prompt word, to obtain second video features, the second video features being used to indicate the second special effect video.

[0027] The training module is configured to perform, for any special effect in the plurality of special effects, determining a second loss based on the second video features and sample video features of a sample special effect video of the special effect, the second loss being used to represent a difference between the second video features and the sample video features; and training the video generation model with a target of minimizing the second loss.

[0028] In some embodiments, the training module is configured to perform freezing parameters in structures other than a special effect encoder in the video generation model, and adjusting parameters in the special effect encoder in the video generation model.

[0029] In some embodiments, the video generation model further comprises a plurality of special effect adapters corresponding to respective network layers in a video generation network in the video generation model, the video generation network being configured to generate a special effect video.

[0030] The apparatus further comprises:

[0031] The second processing module is configured to perform, for any special effect in the plurality of special effects, encoding, by an image encoder in the video generation model, any frame image in a sample special effect video of the special effect to obtain second image encoding features; for a special effect adapter corresponding to an i-th network layer in the video generation network, processing, by the special effect adapter, third video features and the second image encoding features to obtain fourth video features, the third video features being input of the i-th network layer, the third video features being configured to indicate a special effect video containing the special effect generated by an i-1-th network layer; fusing the fourth video features and output of the i-th network layer to obtain input of an i+1-th network layer; and training the plurality of special effect adapters in the video generation model based on target video features output by a last network layer in the video generation network and the sample special effect video.

[0032] In some embodiments, the first processing module is configured to perform, for a special effect adapter corresponding to an i-th network layer in the video generation network, broadcasting, by the special effect adapter, the second image encoding features into image features corresponding to respective frame images in the third video features to obtain the third video features.

[0033] In some embodiments, the plurality of special effects comprises an interactive special effect, the interactive special effect being configured to indicate a special effect in which both a subject and an environment in a video change.

[0034] According to another aspect of embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0035] one or more processors;

[0036] a memory for storing program code executable by the processor;

[0037] wherein the processor is configured to execute the program code to implement the method for training the video generation model.

[0038] According to another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which, when program codes in the computer readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method for training the video generation model.

[0039] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, which includes computer programs / instructions that, when executed by a processor, implement the method for training the video generation model.

[0040] The embodiments of the present disclosure provide a method for training a video generation model. The first special effect video of each special effect is generated by the video generation model based on the first frame image of the sample special effect video of each special effect and the special effect prompt word. The video generation model is trained based on the first special effect video of each special effect and the corresponding sample special effect video. The video generation model not only generates the special effect video more and more in line with the sample special effect video, i.e., the generated special effect video is more and more accurate, but also realizes the association between the special effect prompt word and the special effect video. The video generation model can learn this association. Therefore, the video generation model can accurately determine the style of the special effect based on the special effect prompt word. Compared with the user-defined description related to the special effect, the present solution does not require the user to consider the description of the special effect. The special effect prompt word can reduce semantic ambiguity related to the special effect. Therefore, the special effect video that meets the requirements can be automatically generated based on the special effect prompt word. On the basis of ensuring the generation effect of the special effect video, the generation efficiency of the special effect video is improved.

[0041] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0043] Figure 1 is a schematic diagram of an implementation environment of a method for training a video generation model according to an exemplary embodiment.

[0044] Figure 2 is a flowchart of a method for training a video generation model according to an exemplary embodiment.

[0045] Figure 3 is a flowchart of another method for training a video generation model according to an exemplary embodiment.

[0046] Figure 4 is a schematic diagram of special effect classification according to an exemplary embodiment.

[0047] Figure 5 is a schematic diagram of training a text encoder according to an example embodiment.

[0048] Figure 6 is a schematic diagram of training a special effect encoder according to an example embodiment.

[0049] Figure 7 is a schematic diagram of training a special effect adapter according to an example embodiment.

[0050] Figure 8 is a schematic diagram of a special effect adapter according to an example embodiment.

[0051] Figure 9 is a schematic diagram of a model inference stage according to an example embodiment.

[0052] Figure 10 is a block diagram of a training apparatus of a video generation model according to an example embodiment.

[0053] Figure 11 is a block diagram of a terminal according to an example embodiment.

[0054] Figure 12 is a block diagram of a server according to an example embodiment. DETAILED DESCRIPTION

[0055] In order to make the ordinary person skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0056] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following example embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0057] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present disclosure are authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the sample special effect videos and special effect prompt words of each of the multiple special effects involved in the present disclosure are obtained under full authorization.

[0058] Figure 1 is a schematic diagram of an implementation environment of a training method of a video generation model according to an example embodiment. Taking an electronic device provided as a server as an example, referring to Figure 1 , the implementation environment specifically includes a terminal 101 and a server 102.

[0059] The terminal 101 is at least one of a smartphone, a smart watch, a desktop computer, a laptop computer, an MP3 player, an MP4 player, a laptop computer, and the like. The terminal 101 has an application program installed and running thereon, which can be a multimedia application program, a clipping application program, a game application program, or a social application program, etc., and the present embodiment does not limit this. A user can log in to the application program through the terminal 101 to obtain the services provided by the application program. The terminal 101 can be connected to the server 102 through a wireless network or a wired network, and can then send sample special effect videos and special effect prompt words to the server 102, so that the server 102 trains a video generation model through the sample special effect videos and the special effect prompt words.

[0060] The terminal 101 generally refers to one of a plurality of terminals, and the present embodiment takes the terminal 101 as an example. Those skilled in the art can know that the number of the above terminals can be more or less. For example, the above terminals can be several, or the above terminals can be dozens or hundreds, or more, and the present embodiment does not limit the number and type of the terminals.

[0061] The server 102 is at least one of a server, a plurality of servers, a cloud computing platform, and a virtualization center. The server 102 can be connected to the terminal 101 and other terminals through a wireless network or a wired network, and the server 102 can receive the sample special effect videos and the special effect prompt words sent by the terminal 101, input the sample special effect videos and the special effect prompt words into a video generation model, and train the video generation model. In some embodiments, the number of the above servers can be more or less, and the present embodiment does not limit this. Of course, the server 102 also includes other functional servers in order to provide more comprehensive and diversified services.

[0062] Figure 2is a flowchart of a training method of a video generation model according to an example embodiment, referring to Figure 2 The training method of the video generation model is applied in a server and includes the following steps.

[0063] In step 201, the server obtains a plurality of special effect videos and special effect prompt words respectively corresponding to a plurality of special effects, and the special effect prompt words are used to indicate the special effect in the video generated by the video generation model.

[0064] In the embodiments of the present disclosure, the plurality of special effects can include petrification special effect, lightning special effect, explosion special effect, etc., and the embodiments of the present disclosure do not limit the style of the plurality of special effects. For any special effect, the server obtains the sample special effect video and the special effect prompt word of the special effect. The sample special effect video of each special effect is used to present the style of the special effect. The special effect prompt word of each special effect is used to indicate the special effect, which is equivalent to a special effect identifier.

[0065] In step 202, for any special effect in the plurality of special effects, the server processes the first frame image in the sample special effect video of the special effect and the special effect prompt word of the special effect through the video generation model to obtain a first special effect video, and the first frame image is used to indicate the subject in the video generated by the video generation model.

[0066] In the embodiments of the present disclosure, for any special effect in the plurality of special effects, the server inputs the first frame image in the sample special effect video of the special effect and the special effect prompt word into the video generation model, and processes the first frame image and the special effect prompt word through the video generation model to generate a first special effect video. The first special effect video is a special effect video automatically generated by the video generation model based on the first frame image and the special effect prompt word. In the process of generating the first special effect video, the video generation model determines the subject in the special effect video to be generated from the first frame image, and determines the special effect in the special effect video to be generated from the special effect prompt word.

[0067] The subject refers to the main carrier for conveying information in the video. The subject can be a character or an object in the video, and the embodiments of the present disclosure do not limit this.

[0068] In step 203, the server trains the video generation model based on the first special effect videos and the sample special effect videos corresponding to the plurality of special effects.

[0069] In the embodiments of the present disclosure, for any special effect in the plurality of special effects, the server can determine the model loss of the video generation model based on the first special effect video corresponding to the special effect and the sample special effect video of the special effect. The model loss is used to represent the difference between the first special effect video generated by the model and the sample special effect video. Then, the server trains the video generation model to minimize the model loss, so that the special effect video generated by the video generation model is closer and closer to the sample special effect video.

[0070] The method for training a video generation model provided in the embodiments of the present disclosure generates a first special effect video for each special effect by processing the first frame image of the sample special effect video of each special effect and the special effect prompt word of the special effect through the video generation model, and trains the video generation model through the first special effect video of each special effect and the corresponding sample special effect video. Not only does this make the video generation model generate special effect videos that are increasingly consistent with sample special effect videos, that is, the generated special effect videos are increasingly accurate, but it also achieves the association of special effect prompt words with special effect videos, and the video generation model can learn this association, thereby enabling the video generation model to accurately determine the style of a special effect through a special effect prompt word. In addition to the user defining descriptions related to special effects, the present solution does not require the user to consider the description of the special effect, and the special effect prompt word can reduce semantic ambiguity related to the special effect, thereby enabling the automatic generation of special effect videos that meet the requirements based on the special effect prompt word. On the basis of ensuring the generation effect of the special effect video, the generation efficiency of the special effect video is improved.

[0071] In some embodiments, for any special effect in the plurality of special effects, the processing of the first frame image in the sample special effect video of the special effect and the special effect prompt word of the special effect through the video generation model to obtain a first special effect video comprises:

[0072] For any special effect in the plurality of special effects, the first frame image in the sample special effect video of the special effect is encoded through an image encoder in the video generation model to obtain a first image encoding feature;

[0073] The special effect prompt word of the special effect is encoded through a text encoder in the video generation model to obtain a text encoding feature;

[0074] The first image encoding feature and the text encoding feature are processed through a video generation network in the video generation model to obtain the first special effect video.

[0075] The solution provided in the embodiments of the present disclosure includes an image encoder, a text encoder, and a video generation network in the video generation model. The first image encoding feature and the text encoding feature are obtained by encoding the first frame image in the sample special effect video and the special effect prompt word through the image encoder and the text encoder, respectively. Then, the video generation network generates a first special effect video based on the first image encoding feature and the text encoding feature, so that the pictures in the first special effect video can be as consistent as possible with the sample special effect video, which is conducive to achieving the desired training effect in a small number of training times.

[0076] In some embodiments, the processing of the first image encoding feature and the text encoding feature through the video generation network in the video generation model to obtain the first special effect video comprises:

[0077] The first image coding feature and the text coding feature are processed by a video generation network in the video generation model to obtain a first video feature, and the first video feature is used to indicate the first special effect video.

[0078] The video generation model is trained based on the first special effect video corresponding to the plurality of special effects and the sample special effect video.

[0079] For any special effect in the plurality of special effects, a first loss is determined based on the first video feature and a sample video feature of the sample special effect video of the special effect, and the first loss is used to represent a difference between the first video feature and the sample video feature.

[0080] The video generation model is trained with the first loss minimized as a target.

[0081] The scheme provided in the embodiments of the present disclosure calculates the difference between the sample special effect video and the generated first special effect video at the level of video features, and trains the video model without converting the video features into the video itself, which can improve the calculation efficiency of the first loss and thus improve the training efficiency of the video generation model.

[0082] In some embodiments, the training of the video generation model comprises:

[0083] Parameters in structures other than the text encoder in the video generation model are frozen, and parameters of the text encoder in the video generation model are adjusted.

[0084] The scheme provided in the embodiments of the present disclosure trains only the text encoder in the video generation model, which not only enables the text encoder to learn the relationship between the special effect prompt word and the special effect itself, so that the text encoder can accurately understand the special effect to be generated and thus extract more accurate features of the special effect prompt word, but also only adjusts the parameters of the text encoder, which is less, and thus improves the training efficiency.

[0085] In some embodiments, the method further comprises:

[0086] For any special effect in the plurality of special effects, a special effect feature is obtained by coding a sample video feature of a sample special effect video of the special effect by a special effect encoder in the video generation model, and the special effect feature is used to indicate an attribute of the special effect in the sample special effect video.

[0087] The video generation model is trained again by the special effect feature, a first frame image in the sample special effect video of the special effect, and a special effect prompt word of the special effect.

[0088] The scheme provided by the embodiments of the present disclosure encodes the sample video features of the sample special effect video through a special effect encoder, so that the features of the special effect in the sample special effect video can be obtained, and then the second special effect video is generated based on the first frame image in the sample special effect video and the special effect prompt word of the special effect, and the special effect features. The video generation model is trained again through the second special effect video and the sample special effect video. Since the special effect features are special effect information obtained from a visual perspective, and the special effect prompt word is special effect information obtained from a language perspective, the second special effect video is generated based on the special effect features and the special effect prompt word. The special effect in the second special effect video is more consistent with the special effect in the sample special effect video, that is, the special effect is more accurate. The model is trained on this basis, so that the video generation model can generate special effects from a visual perspective and a language perspective, which helps to improve the generation accuracy of the video generation model.

[0089] In some embodiments, for any special effect in the plurality of special effects, the sample video features of the sample special effect video of the special effect are encoded through a special effect encoder in the video generation model to obtain special effect features, including:

[0090] For any special effect in the plurality of special effects, the sample video features are processed through an attention layer in the special effect encoder to obtain query features, key features and value features of the attention layer. The query features and the key features are used to indicate the position of the special effect in the sample special effect video, and the value features are used to indicate the attributes of the special effect.

[0091] The query features, the key features and the value features are fused to obtain the special effect features.

[0092] The scheme provided by the embodiments of the present disclosure uses an attention layer to process sample video features to obtain query features, key features and value features of the attention layer, and fuses the three features to obtain special effect features. Since the query features and the key features are used to indicate the position of the special effect in the sample special effect video, and the value features are used to indicate the attributes of the special effect, the obtained special effect features can accurately reflect the position and style of the special effect in the sample special effect video, that is, the accuracy of the special effect features can be improved.

[0093] In some embodiments, the video generation model processes the special effect features, the first frame image in the sample special effect video of the special effect, and the special effect prompt word of the special effect to obtain a second special effect video, including:

[0094] The video generation network in the video generation model processes the special effect feature, a first image feature corresponding to the first frame image, and a text encoding feature corresponding to the special effect prompt word, to obtain a second video feature, where the second video feature is used to indicate the second special effect video.

[0095] The method further includes:

[0096] For any special effect in the plurality of special effects, a second loss is determined based on the second video feature and a sample video feature of a sample special effect video of the special effect, where the second loss is used to represent a difference between the second video feature and the sample video feature.

[0097] The video generation model is trained with the second loss minimized as a target.

[0098] The scheme provided in the embodiments of the present disclosure calculates the difference between the sample special effect video and the generated second special effect video at the level of the video feature, and trains the video model without converting the video feature into the video itself, which can improve the calculation efficiency of the second loss and thus improve the training efficiency of the video generation model.

[0099] In some embodiments, the retraining of the video generation model includes:

[0100] Parameters in structures other than the special effect encoder in the video generation model are frozen, and parameters in the special effect encoder in the video generation model are adjusted.

[0101] The scheme provided in the embodiments of the present disclosure trains only the special effect encoder in the video generation model, which not only enables the special effect encoder to learn the features of the special effect and accurately understand the special effect to be generated, so as to extract more accurate special effect features and provide a guarantee for subsequent accurate generation of the special effect video, but also adjusts only the parameters of the special effect encoder, which is less in the number of parameters and is conducive to improving the training efficiency.

[0102] In some embodiments, the video generation model further includes a plurality of special effect adapters, which one-to-one correspond to respective network layers in a video generation network in the video generation model, and the video generation network is used to generate a special effect video.

[0103] The method further includes:

[0104] For any special effect in the plurality of special effects, any frame image in a sample special effect video of the special effect is encoded by an image encoder in the video generation model to obtain a second image encoding feature.

[0105] For the special effect adapter corresponding to the i-th network layer in the video generation network, the third video feature and the second image coding feature are processed through the special effect adapter to obtain a fourth video feature, the third video feature being an input of the i-th network layer, and the third video feature being used to indicate a special effect video containing the special effect generated by the i-1-th network layer.

[0106] The fourth video feature is fused with an output of the i-th network layer to obtain an input of an i+1-th network layer.

[0107] Based on a target video feature output by a last network layer in the video generation network and a sample special effect video, the plurality of special effect adapters in the video generation model are trained.

[0108] The scheme provided in the embodiments of the present disclosure can decouple the video subject and the special effect in the sample special effect video, learn the physical motion law of the special effect and migrate it to any input picture, accurately capture and reproduce the physical motion law of the special effect, ensure that the generated video remains consistent in visual effect, and accurately reflect the characteristics of the special effect.

[0109] In some embodiments, the processing of the third video feature and the second image coding feature through the special effect adapter to obtain the fourth video feature includes:

[0110] For the special effect adapter corresponding to the i-th network layer in the video generation network, the second image coding feature is broadcast to image features corresponding to respective image frames in the third video feature through the special effect adapter to obtain the third video feature.

[0111] The scheme provided in the embodiments of the present disclosure can reflect the video subject through the second image coding feature, broadcast the second image coding feature to all images, use it as appearance guidance during training, make the model learned subject move according to the expected motion pattern in the existing video, decouple the complete special effect video and the random video frame spatial information, more purely capture the physical visual effect that does not exist in real life, and further enhance the performance of the special effect generation result after being injected into the generation model.

[0112] In some embodiments, the plurality of special effects include interactive special effects, and the interactive special effects are used to indicate special effects in which both the subject and the environment in the video change.

[0113] The scheme provided by the embodiments of the present disclosure includes interactive special effects in which the subject and the environment both change, which facilitates the video generation model to learn the interactive special effects, thereby enabling the video generation model to generate special effects that interact with the subject, and enriching the ability of the video generation model to generate special effects.

[0114] The above Figure 2 The above Figure 3 FIG. 8 is a flowchart of another method for training a video generation model according to an example embodiment. The method is described below with reference to the electronic device as a server. As shown in FIG. 8, the method includes the following steps. Figure 3

[0115] In step 301, the server obtains a plurality of special effects each having a sample special effect video and a special effect prompt word. The special effect prompt word is used to indicate a special effect in a video generated by the video generation model.

[0116] In the embodiments of the present disclosure, the plurality of special effects can include petrification special effects, lightning special effects, explosion special effects, etc. The embodiments of the present disclosure do not limit the styles of the plurality of special effects. For any special effect, the server obtains a sample special effect video and a special effect prompt word of the special effect. The sample special effect video of each special effect is used to present the style of the special effect. The special effect prompt word of each special effect is used to indicate the special effect, which is equivalent to a special effect identifier.

[0117] In some embodiments, the plurality of special effects can include at least one of a subject special effect, a background special effect, and an interactive special effect. The subject special effect is used to indicate a special effect in which a subject in a video changes. The background special effect is used to indicate a special effect in which a background (environment) in a video changes. The interactive special effect is used to indicate a special effect in which a subject and an environment (background) in a video both change. The scheme provided by the embodiments of the present disclosure includes interactive special effects in which the subject and the environment both change, which facilitates the video generation model to learn the interactive special effects, thereby enabling the video generation model to generate special effects that interact with the subject, and enriching the ability of the video generation model to generate special effects.

[0118] ​Each of the above-mentioned special effects can also include multiple sub-classes. For example, the subject special effect can include a subject physical change special effect, a subject material change special effect, a subject transformation special effect, and the like. The subject physical change special effect refers to a special effect in which the form of the subject changes. The subject transformation special effect refers to a special effect in which the subject is transformed into a preset role. The background special effect can include a real special effect and a virtual special effect, and the like. The background in the real special effect actually exists in real life. The background in the virtual special effect does not exist in real life, but is fictitious. The interactive special effect can include a motion following special effect, a subject static special effect, and a subject dynamic special effect. The motion following special effect refers to a special effect in which the display position of the special effect changes following the motion of the subject. In the subject static special effect, the subject is in a static state. For example, in the “hair growing” special effect, the display position of the head as the subject does not change. In the subject dynamic special effect, the subject is in a dynamic state. For example, in the “pinching the shoulder” special effect, the shoulder as the subject deforms in response to the pinching motion.

[0119] For example, the present scheme balances the construction of a special effect data set of about 50 types according to the type of special effect, each type of special effect containing 10 special effect videos, and the total amount of special effect data sets exceeding 500, for subsequent training of a special effect video generation model. See Figure 4 , Figure 4 is a schematic diagram of a special effect classification according to an example embodiment.

[0120] In step 302, for any special effect in the plurality of special effects, the server processes the first frame image in the sample special effect video of the special effect and the special effect prompt word of the special effect through the video generation model to obtain a first special effect video. The first frame image is used to indicate the subject in the video generated by the video generation model.

[0121] In the embodiments of the present disclosure, for any special effect in the plurality of special effects, the server inputs the first frame image in the sample special effect video of the special effect and the special effect prompt word into the video generation model, determines the subject in the special effect video to be generated from the first frame image in the sample special effect video of the special effect, and determines the special effect in the special effect video to be generated from the special effect prompt word, so as to generate a first special effect video based on the subject and the special effect.

[0122] In some embodiments, the video generation model comprises an image encoder, a text encoder, and a video generation network. The image encoder is configured to extract encoded features of a first frame image in a sample special effect video. The text encoder is configured to extract encoded features of a special effect prompt word. The video generation network is configured to generate a special effect video based on the encoded features of the first frame image and the encoded features of the special effect prompt word. Accordingly, the process of generating the first special effect video based on the video generation model comprises: for any special effect in the plurality of special effects, the server encodes the first frame image in the sample special effect video of the special effect by the image encoder in the video generation model to obtain first image encoded features. The server encodes the special effect prompt word of the special effect by the text encoder in the video generation model to obtain text encoded features. Then, the server processes the first image encoded features and the text encoded features by the video generation network in the video generation model to obtain the first special effect video.

[0123] The scheme provided by the embodiments of the present disclosure comprises an image encoder, a text encoder, and a video generation network in the video generation model. The first image encoded features and the text encoded features are obtained by respectively encoding the first frame image in the sample special effect video and the special effect prompt word by the image encoder and the text encoder. Then, the first special effect video is generated based on the first image encoded features and the text encoded features by the video generation network. As a result, the pictures in the first special effect video can be as close to the sample special effect video as possible, which is beneficial to achieving the desired training effect in a small number of training times.

[0124] In the process of generating the first special effect video, the server can process the first image encoded features and the text encoded features by the video generation network in the video generation model to obtain first video features, which are used to indicate the first special effect video. That is, the server only needs to generate the video features for subsequent model training, without the need to convert to the video level.

[0125] In step 303, the server trains the video generation model based on the first special effect videos corresponding to the plurality of special effects and the sample special effect videos.

[0126] In the embodiments of the present disclosure, for any special effect in the plurality of special effects, the server can determine a model loss of the video generation model based on the first special effect video corresponding to the special effect and the sample special effect video of the special effect. The model loss is used to represent the difference between the first special effect video generated by the video generation model and the sample special effect video. Then, the server trains the video generation model to minimize the model loss, so that the special effect video generated by the video generation model is closer and closer to the sample special effect video.

[0127] In the process of calculating the model loss, the server can calculate the model loss based on the video feature of the first special effect video and the video feature of the sample special effect video. Accordingly, for any special effect in the plurality of special effects, the server determines a first loss based on the first video feature and the sample video feature of the sample special effect video of the special effect. The first loss is used to represent the difference between the first video feature and the sample video feature. Then, the server trains the video generation model with the objective of minimizing the first loss. The video feature of the sample special effect video can be extracted by a video encoder independent of the video generation model. The disclosure does not limit the manner of calculating the first loss. The scheme provided by the disclosure calculates the difference between the sample special effect video and the generated first special effect video at the level of the video feature, and trains the video model without converting the video feature into the video itself, which can improve the calculation efficiency of the first loss and thus improve the training efficiency of the video generation model.

[0128] In the above training process of the video generation model, the server can adjust all parameters in the video generation model (including the parameters of the image encoder, the parameters of the text encoder, and the parameters of the video generation network); or the server can only adjust the parameters of the text encoder. The disclosure does not limit this.

[0129] In some embodiments, in the above training process, the server only adjusts the parameters of the text encoder. Accordingly, the server freezes the parameters in the structures other than the text encoder in the video generation model and adjusts the parameters of the text encoder in the video generation model. That is, the above training process is essentially a training process of the text encoder. The scheme provided by the disclosure trains only the text encoder in the video generation model, which not only enables the text encoder to learn the relationship between the special effect prompt and the special effect itself, so that the text encoder can accurately understand the special effect to be generated and thus extract more accurate features of the special effect prompt, providing a guarantee for subsequent accurate generation of the special effect video, but also only adjusts the parameters of the text encoder, which is less likely to affect the training efficiency.

[0130] For example, Figure 5 is a schematic diagram of training a text encoder according to an exemplary embodiment. Referring to Figure 5 In the above training process, for any special effect, the server inputs the special effect prompt of the special effect to the text encoder (text encoder) in the video generation model inputs the first image in the sample special effect video of the special effect to the image encoder (Image encoder) in the video generation model, extracts the special effect prompt The text encoding features are extracted by using an image encoder to obtain the first image encoding features of the first frame image; then, special effects prompts are used. The text encoding features and the first image encoding features of the first frame image are fed into the video generation network in the video generation model. The video generation network processes these features to obtain the video features of the first special effects video. The server can also input sample special effects videos into the video encoder, which extracts the video features of the sample special effects videos. Then, based on the video features of the first special effects video and the sample video features of the sample special effects videos, the server determines the first loss. Finally, by freezing components such as the video generation network, image encoder, and video encoder, the server aims to minimize the first loss and trains only those with special effects prompts. A text encoder as input.

[0131] In the training process described above, there can be multiple sample videos for each special effect. The main subject in different sample videos of the same special effect is different. That is, the server can train the model using multiple sample videos of the same special effect, enabling the video generation model to distinguish between special effects and the main subject, accurately learn the features of the special effect, and improve the accuracy of the video generation model.

[0132] After training the video generation model on multiple special effects, the server is able to combine the special effects video with the special effects prompts. Matching is performed to address the problem that special effects videos are difficult to describe in text, leading to errors in the video generation network due to insufficient understanding. Special effects cue words are introduced into the special effects video generation task. As the core carrier of text description, it can effectively solve the problems of descriptive ambiguity and generation instability in traditional text-driven generation, and bring multiple technical advantages. By extracting the visual essential features of special effects (such as "dynamic particle halo" and "fluid distortion and deformation"), a standardized descriptive terminology library is established, which can accurately anchor the physical properties and dynamic laws of special effects, reducing semantic ambiguity compared with free text description.

[0133] In step 304, for any one of the multiple special effects, the server encodes the sample video features of the sample special effect video using the special effect encoder in the video generation network to obtain special effect features, which are used to indicate the attributes of the special effect in the sample special effect video.

[0134] In the embodiments of the present disclosure, the special effect encoder is further included in the video generation model. The special effect encoder is configured to extract a special effect feature of a sample special effect video of a special effect. For any special effect in the plurality of special effects, the server inputs the sample video feature of the sample special effect video of the special effect into the special effect encoder in the video generation model, encodes the sample video feature by the special effect encoder, and obtains the special effect feature of the special effect. The special effect feature can reflect the characteristics or style of the special effect in the visual level. The extraction manner of the special effect feature is not limited in the embodiments of the present disclosure.

[0135] In some embodiments, the server can extract the special effect feature of the special effect by using the attention mechanism. Accordingly, step 304 includes: for any special effect in the plurality of special effects, the server processes the sample video feature by using an attention layer in the special effect encoder, and obtains a query feature, a key feature and a value feature of the attention layer. The query feature and the key feature are used to indicate the position of the special effect in the sample special effect video. The value feature is used to indicate the attribute of the special effect. Then, the server fuses the query feature, the key feature and the value feature to obtain the special effect feature. The query feature, the key feature and the value feature refer to the matrices corresponding to Q, K and V in the attention mechanism.

[0136] The scheme provided by the embodiments of the present disclosure uses the attention layer to process the sample video feature, and obtains the query feature, the key feature and the value feature of the attention layer. The query feature and the key feature are used to indicate the position of the special effect in the sample special effect video, and the value feature is used to indicate the attribute of the special effect. Therefore, the obtained special effect feature can accurately reflect the position and style of the special effect in the sample special effect video, that is, the accuracy of the special effect feature can be improved.

[0137] In step 305, the server processes the special effect feature, the first frame image in the sample special effect video of the special effect and the special effect prompt word of the special effect by using the video generation network to obtain a second special effect video.

[0138] In the embodiments of the present disclosure, the server processes the special effect feature of any special effect, the first image encoding feature of the first frame image in the sample special effect video of the special effect and the text encoding feature corresponding to the special effect prompt word by using the video generation network in the video generation model to obtain a second special effect video. The video generation model determines the subject in the generated special effect video from the first frame image in the sample special effect video of the special effect, and determines the special effect in the generated special effect video from the special effect prompt word and the special effect, so as to generate the second special effect video based on the subject and the special effect. That is, the video generation model determines the special effect to be generated from the textual description level and the image visual level.

[0139] In some embodiments, the server processes the special effect feature, the first image corresponding to the first frame image, and the text encoding feature corresponding to the special effect prompt word through a video generation network in the video generation network to obtain a second video feature. The second video feature is used to indicate a second special effect video. That is, the server only generates the video feature, and then performs subsequent model training, without the need to convert to the video level.

[0140] In step 306, the server re-trains the video generation model based on the second special effect video corresponding to the plurality of special effects and the sample special effect video.

[0141] In the embodiments of the present disclosure, for any special effect in the plurality of special effects, the server can re-determine a model loss of the video generation model based on the second special effect video corresponding to the special effect and the sample special effect video of the special effect. The model loss is used to represent the difference between the second special effect video generated by the video generation model and the sample special effect video. Then, the server re-trains the video generation model to minimize the model loss, so that the special effect video generated by the video generation model is closer and closer to the sample special effect video.

[0142] In the process of calculating the model loss, the server can calculate the model loss based on the video feature of the second special effect video and the video feature of the sample special effect video. Accordingly, for any special effect in the plurality of special effects, the server determines a second loss based on the second video feature and the sample video feature of the sample special effect video of the special effect. The second loss is used to represent the difference between the second video feature and the sample video feature; the server re-trains the video generation model to minimize the second loss. The scheme provided in the embodiments of the present disclosure calculates the difference between the sample special effect video and the generated second special effect video at the level of the video feature, and trains the video model without converting the video feature into the video itself, which can improve the calculation efficiency of the second loss, thereby facilitating the improvement of the training efficiency of the video generation model.

[0143] In the above training process of the video generation model, the server can adjust all parameters in the video generation model at the same time; or the server can also only adjust the parameters of the special effect encoder. The embodiments of the present disclosure do not limit this.

[0144] In some embodiments, in the training process described above, the server only adjusts the parameters of the special effect encoder. Accordingly, the server freezes the parameters in the structures other than the special effect encoder in the video generation network, and adjusts the parameters in the special effect encoder in the video generation network. That is, the training process in steps 304 to 306 is essentially a training process of the special effect encoder. The scheme provided by the embodiments of the present disclosure trains only the special effect encoder in the video generation network, which not only enables the special effect encoder to learn the features of the special effect, so that the special effect encoder can accurately understand the special effect to be generated, thereby extracting more accurate special effect features, and providing a guarantee for subsequent accurate generation of the special effect video, but also only adjusts the parameters of the special effect encoder, which is less, and is beneficial to improve the training efficiency.

[0145] For example, Figure 6 is a schematic diagram of training a special effect encoder according to an exemplary embodiment. Referring to Figure 6 , by using the first frame image of the sample special effect video and the special effect prompt word as conditions, a special effect encoder (VFX encoder) is trained, which can extract sample video features of the sample special effect video and encode them into the form of embedding. For the extracted features, the scheme uses a linear mapping method to convert them into the form of QKV in the attention mechanism. Among them, embedding QK is converted into a one-dimensional vector, and is directly superimposed on the hidden space features of the video generation network to indicate the motion embedding in the time dimension, and embedding V is converted into a two-dimensional vector, and is also directly superimposed on the hidden space features of the video generation network to indicate the motion embedding in the space dimension, as shown in Figure 6 , then these features will be operated in the self-attention mechanism in the time attention layer, and the formula is as follows:

[0146]

[0147] Among them, Q, K and V are query, key and value matrices obtained from F, dk represents the dimension of the key vector, and is a scaling factor to maintain numerical stability in the softmax function. This time attention mechanism allows the updated features of each frame to collect information from other frames, enhances the inter-frame relationship and captures the time continuity necessary for video generation. The purpose of setting embedding QK to a one-dimensional vector is to extract only the features between the frames of the special effect video, and embedding V is set to two-dimensional because there are a large number of deformation changes in the special effect video, and two-dimensional embedding V can better extract the spatial deformation changes in the video.

[0148] In some embodiments, the video generation network further comprises a plurality of effect adapters. The plurality of effect adapters correspond to each network layer in the video generation network one by one. The video generation network is configured to generate an effect video.

[0149] For any one of the plurality of effects, the server encodes any one of the frame images in the sample effect video of the effect through an image encoder in the video generation network to obtain second image encoding features. For the effect adapter corresponding to the i th network layer in the video generation network, the server processes the third video features and the second image encoding features through the effect adapter to obtain fourth video features. The third video features are the input of the i th network layer. The third video features are configured to indicate the effect video containing the effect generated by the i-1 th network layer. Then, the server fuses the fourth video features with the output of the i th network layer to obtain the input of the i+1 th network layer. The server trains the plurality of effect adapters in the video generation network based on the target video features output by the last network layer in the video generation network and the sample effect video.

[0150] The scheme provided by the embodiments of the present disclosure can decouple the video subject and the effect in the sample effect video, learn the physical motion law of the effect and migrate it to any input picture, accurately capture and reproduce the physical motion law of the effect, ensure the consistency of the generated video in visual effect, and accurately reflect the characteristics of the effect.

[0151] The server calculates a third loss based on the target video features output by the last network layer in the video generation network and the sample video features of the sample effect video. The third loss is configured to represent the difference between the target video features and the sample video features. Then, the server trains the plurality of effect adapters in the video generation network with the objective of minimizing the third loss. In the case that the third loss is lower than a preset value or the number of parameter adjustments of the effect video adapter reaches a preset number, the server stops training the plurality of effect adapters.

[0152] For example, Figure 7 is a schematic diagram of training an effect adapter according to an exemplary embodiment. Referring to Figure 7 In order to further enable the model to understand the effect, an additional effect adapter is designed, and the input of the additional effect adapter includes a random frame image in the sample effect video. The purpose of this is to learn the physical motion pattern in the sample effect video and decouple the subject in the video that changes the effect.

[0153] In some embodiments, for the special effect adapter corresponding to the i-th network layer in the video generation network, the server broadcasts the second image encoding features into the image features corresponding to each image frame in the third video features through the special effect adapter, to obtain the third video features. The scheme provided by the embodiments of the present disclosure can reflect the video subject through the second image encoding features, and by broadcasting the second image encoding features into all images, it can be used as appearance guidance during training, so that the model can make the learned subject move according to the expected motion pattern in the existing video, thereby decoupling the complete special effect video from the random video space information, and more pure physical visual effects that do not exist in real life can be captured, and after being injected into the video generation network, the performance of the special effect generation result is further enhanced.

[0154] For example, Figure 8 is a schematic diagram of a special effect adapter according to an exemplary embodiment. Referring to Figure 8 , the special effect adapter includes a down-sampling layer ( ), a nonlinear layer (Nonlinear), an up-sampling layer ( ), and a conditional layer ( ). The down-sampling layer is used to reduce the dimension or resolution of the feature map. Through the down-sampling operation, the channel number of the feature map after the nonlinear transformation can be reduced, thereby reducing the computational complexity while preserving the most important feature information. The nonlinear layer is usually a nonlinear activation function, such as ReLU (Rectified Linear Unit). The role of the nonlinear activation function is to introduce nonlinear characteristics, so that the model can learn more complex feature representations. Without the nonlinear activation function, the neural network will only learn linear transformations, which will greatly limit the expressive power of the model. The up-sampling layer is usually used to increase the dimension or resolution of the feature map. Through the up-sampling operation, the channel number of the input feature map can be increased, thereby providing more feature information for subsequent nonlinear transformation. The conditional layer is usually used to fuse the output of the adapter module with the original input. Through the conditional layer, the features extracted by the adapter module can be weighted and fused with the original input features, thereby enhancing the expressive power of the model. The output of the conditional layer is usually added element by element (such as the plus sign in Figure 8 ) with the original input feature map to realize the fusion of features.

[0155] where the effect adapter can be customized according to a class of videos (e.g., videos representing various dog motions), multiple videos showing the same motion, or even motion patterns extracted from a single video. Although the effect adapter is able to capture motion patterns, it inevitably learns the appearance of the subject in the input video during the training process. To separate the spatial and temporal information, we incorporate an appearance guide in the effect adapter, forcing it to learn pure motion. Specifically, we add a conditional linear layer with weights to integrate the appearance information into the temporal hidden state where is the hidden state at time t with dimension , is the batch size, is the size of the feature map, is the number of features, is some additional dimension (e.g., number of layers). Then, we randomly select a frame from the training video and obtain its image embedding through an image encoder. Here, is the image embedding obtained from the image encoder with dimension , is the number of features in the embedding. This image embedding is then broadcasted to all frames as the appearance guide during training. The forward process of the effect adapter is formulated as

[0156]

[0157]

[0158] where is the output of the effect adapter. is an activation function, typically a nonlinear function such as ReLU or sigmoid, used to introduce nonlinear characteristics; and are weight matrices used to further transform the hidden state, typically for dimension reduction and dimension increase operations to adapt to different layers or modules of the network. This operation first computes the matrix multiplication of e and , and then broadcasts the result to the same shape as so that element-wise addition can be performed. During inference, we randomly select a user-provided training image as the appearance condition input for the effect adapter.

[0159] In the model usage phase, the video generation model is able to generate effect customization based on the reference image input and the effect cue as a condition. For example, Figure 9is a schematic diagram of a model inference stage according to an example embodiment.

[0160] The method for training a video generation model provided in the embodiments of the present disclosure generates a first special effect video of each special effect through a video generation network based on the first frame image of the sample special effect video of each special effect and the special effect prompt word, trains the video generation network based on the first special effect video of each special effect and the corresponding sample special effect video, not only makes the video generation network generate the special effect video more and more in line with the sample special effect video, that is, the generated special effect video is more and more accurate, but also realizes the association of the special effect prompt word and the special effect video, and the video generation network can learn this association, so that the video generation network can accurately determine the style of the special effect through the special effect prompt word. Compared with the user defining the description related to the special effect, the present solution does not require the user to consider the description of the special effect, and the special effect prompt word can reduce the semantic ambiguity related to the special effect, so that the special effect video meeting the requirements can be automatically generated based on the special effect prompt word. On the basis of guaranteeing the generation effect of the special effect video, the generation efficiency of the special effect video is improved.

[0161] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described one by one here.

[0162] Figure 10 is a block diagram of a video generation model training device according to an example embodiment. Referring to Figure 10 The video generation model training device includes an acquisition module 1001, a first processing module 1002, and a training module 1003.

[0163] The acquisition module 1001 is configured to acquire sample special effect videos and special effect prompt words of a plurality of special effects, and the special effect prompt word is used to indicate the special effect in the video generated by the video generation network.

[0164] The first processing module 1002 is configured to, for any special effect in the plurality of special effects, process the first frame image in the sample special effect video of the special effect and the special effect prompt word of the special effect through the video generation network to obtain a first special effect video, and the first frame image is used to indicate the subject in the video generated by the video generation network.

[0165] The training module 1003 is configured to train the video generation network based on the first special effect video and the sample special effect video corresponding to the plurality of special effects.

[0166] In some embodiments, the first processing module 1002 includes:

[0167] The first encoding unit is configured to perform, for any special effect in the plurality of special effects, encoding, by an image encoder in the video generation network, a first frame image in a sample special effect video of the special effect to obtain first image encoded features;

[0168] The second encoding unit is configured to perform encoding, by a text encoder in the video generation network, a special effect prompt word of the special effect to obtain text encoded features;

[0169] The processing unit is configured to perform processing, by the video generation network in the video generation network, the first image encoded features and the text encoded features to obtain the first special effect video.

[0170] In some embodiments, the processing unit is configured to perform processing, by the video generation network in the video generation network, the first image encoded features and the text encoded features to obtain first video features, the first video features being used to indicate the first special effect video.

[0171] The training module is configured to perform, for any special effect in the plurality of special effects, determining a first loss based on the first video features and sample video features of a sample special effect video of the special effect, the first loss being used to represent a difference between the first video features and the sample video features; and training the video generation model with a target of minimizing the first loss.

[0172] In some embodiments, the training module 1003 is configured to perform freezing parameters in structures other than the text encoder in the video generation network, and adjusting parameters of the text encoder in the video generation network.

[0173] In some embodiments, the apparatus further comprises:

[0174] The special effect encoding module is configured to perform, for any special effect in the plurality of special effects, encoding, by a special effect encoder in the video generation network, sample video features of a sample special effect video of the special effect to obtain special effect features, the special effect features being used to indicate attributes of the special effect in the sample special effect video.

[0175] The first processing module 1002 is further configured to perform processing, by the video generation network, the special effect features, a first frame image in the sample special effect video of the special effect, and a special effect prompt word of the special effect to obtain a second special effect video.

[0176] The training module 1003 is further configured to perform training, by the video generation network, again based on the second special effect videos corresponding to the plurality of special effects and the sample special effect videos.

[0177] In some embodiments, the first processing module is configured to perform, for any special effect in the plurality of special effects, processing, by an attention layer in the special effect encoder, on the sample video feature to obtain a query feature, a key feature and a value feature of the attention layer, the query feature and the key feature being used to indicate a position of the special effect in the sample special effect video, and the value feature being used to indicate an attribute of the special effect; and fusing the query feature, the key feature and the value feature to obtain the special effect feature.

[0178] In some embodiments, the first processing module 1002 is configured to perform, by a video generation network in the video generation network, processing on the special effect feature, a first image encoding feature corresponding to the first frame image and a text encoding feature corresponding to the special effect prompt word to obtain a second video feature, the second video feature being used to indicate the second special effect video.

[0179] The training module 1003 is configured to perform, for any special effect in the plurality of special effects, determining, based on the second video feature and a sample video feature of a sample special effect video of the special effect, a second loss, the second loss being used to represent a difference between the second video feature and the sample video feature; and training the video generation model with a target of minimizing the second loss.

[0180] In some embodiments, the training module 1003 is configured to perform freezing parameters in structures other than the special effect encoder in the video generation network, and adjusting the parameters in the special effect encoder in the video generation network.

[0181] In some embodiments, the video generation network further comprises a plurality of special effect adapters, the plurality of special effect adapters corresponding to respective network layers in the video generation network in one-to-one correspondence, and the video generation network being used to generate the special effect video.

[0182] The apparatus further comprises:

[0183] The second processing module is configured to perform, for any special effect in the plurality of special effects, encoding, by an image encoder in the video generation network, on any frame image in a sample special effect video of the special effect to obtain a second image encoding feature; for a special effect adapter corresponding to an i-th network layer in the video generation network, processing, by the special effect adapter, on a third video feature and the second image encoding feature to obtain a fourth video feature, the third video feature being an input of the i-th network layer, and the third video feature being used to indicate a special effect video containing the special effect generated by an (i-1)-th network layer; fusing the fourth video feature and an output of the i-th network layer to obtain an input of an (i+1)-th network layer; and training the plurality of special effect adapters in the video generation network based on a target video feature output by a last network layer in the video generation network and the sample special effect video.

[0184] In some embodiments, the first processing module 1002 is configured to perform the special effect adapter corresponding to the i-th network layer in the video generation network, and broadcast the second image coding features into the image features corresponding to the image frames in the third video features through the special effect adapter, to obtain the third video features.

[0185] In some embodiments, the plurality of special effects includes interactive special effects, which are used to indicate special effects in which both the subject and the environment in the video change.

[0186] The embodiments of the present disclosure provide a training device of a video generation model. The first special effect video of each special effect is generated by the video generation network based on the first frame image of the sample special effect video of each special effect and the special effect prompt word. The video generation network is trained based on the first special effect video of each special effect and the corresponding sample special effect video. The video generation network not only generates the special effect video more and more in line with the sample special effect video, that is, the generated special effect video is more and more accurate, but also realizes the association of the special effect prompt word and the special effect video. The video generation network can learn this association relationship, so that the video generation network can accurately determine the style of the special effect through the special effect prompt word. Compared with the user defining the description related to the special effect, the present scheme does not require the user to consider the description of the special effect, and the special effect prompt word can reduce the semantic ambiguity related to the special effect, so that the special effect video meeting the requirements can be automatically generated based on the special effect prompt word. On the basis of ensuring the generation effect of the special effect video, the generation efficiency of the special effect video is improved.

[0187] It should be noted that the training device of the video generation model provided in the above embodiments is only used as an example to illustrate the division of the above functional units in training the video generation model. In actual application, the above functions can be completed by different functional units according to needs, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the training device of the video generation model provided in the above embodiments and the method for training the video generation model belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be described here.

[0188] As for the device in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0189] When the electronic device is provided as a terminal, Figure 11 is a block diagram of a terminal 1100 according to an example embodiment. The terminal Figure 11A structural block diagram of a terminal 1100 provided by one exemplary embodiment of the present disclosure is shown. The terminal 1100 can be a smartphone, a tablet computer, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a notebook computer, or a desktop computer. The terminal 1100 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0190] Generally, the terminal 1100 includes a processor 1101 and a memory 1102.

[0191] The processor 1101 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1101 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1101 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1101 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by a display screen. In some embodiments, the processor 1101 can further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

[0192] The memory 1102 can include one or more computer-readable storage media, which can be non-transitory. The memory 1102 can also include a high-speed random access memory and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one computer program for being executed by the processor 1101 to implement the training method of the video generation model provided by the method embodiments in the present application.

[0193] In some embodiments, terminal 1100 can also optionally include a peripheral device interface 1103 and at least one peripheral device. The processor 1101, the memory 1102 and the peripheral device interface 1103 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1103 through a bus, a signal line or a circuit board. Specifically, the peripheral devices include at least one of a radio frequency circuit 1104, a display screen 1105, a camera component 1106, an audio circuit 1107 and a power supply 1108.

[0194] The peripheral device interface 1103 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1101 and the memory 1102. In some embodiments, the processor 1101, the memory 1102 and the peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1101, the memory 1102 and the peripheral device interface 1103 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this regard.

[0195] The radio frequency circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1104 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. In some embodiments, the radio frequency circuit 1104 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1104 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1104 can also include NFC (Near Field Communication) related circuitry, which is not limited by the present application.

[0196] The display screen 1105 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1105 is a touch display screen, the display screen 1105 is further configured to capture touch signals on or above the surface of the display screen 1105. The touch signals can be input to the processor 1101 as control signals for processing. In this case, the display screen 1105 can also be configured to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1105 can be one, disposed on the front panel of the terminal 1100; in other embodiments, the display screen 1105 can be at least two, respectively disposed on different surfaces of the terminal 1100 or in a folding design; in other embodiments, the display screen 1105 can be a flexible display screen, disposed on a curved surface or a folding surface of the terminal 1100. Even, the display screen 1105 can also be disposed in an irregular shape, i.e., a special-shaped screen. The display screen 1105 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0197] The camera assembly 1106 is configured to capture images or videos. In some embodiments, the camera assembly 1106 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, the rear camera is at least two, which is any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function of the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function of the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 1106 can further include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0198] The audio circuit 1107 can include a microphone and a speaker. The microphone is used to collect sound waves of a user and an environment, and convert the sound waves into an electrical signal input to the processor 1101 for processing, or input to the radio frequency circuit 1104 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, respectively arranged at different parts of the terminal 1100. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert an electrical signal from the processor 1101 or the radio frequency circuit 1104 into sound waves. The speaker can be a traditional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can the electrical signal be converted into a sound wave audible to humans, but also can be converted into a sound wave inaudible to humans for ranging purposes. In some embodiments, the audio circuit 1107 can also include a headphone jack.

[0199] The power supply 1108 is used to supply power to each component in the terminal 1100. The power supply 1108 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 1108 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0200] Those skilled in the art can understand that Figure 10 The structure shown in the figure does not constitute a limitation on the terminal 1100, and can include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0201] When the electronic device is provided as a server, Figure 12 is a block diagram of a server 1200 according to an exemplary embodiment. The server 1200 can have a large difference in configuration or performance, and can include one or more processors (Central Processing Units, CPUs) 1201 and one or more memories 1202, wherein the memory 1202 stores at least one program code, which is loaded and executed by the processor 1201 to realize the training method of the video generation model provided by each method embodiment. Of course, the server can also have a wired or wireless network interface, a keyboard, and an input and output interface, etc. to perform input and output, and the server 1200 can also include other components for realizing the function of the device, which will not be described here.

[0202] In an example embodiment, a computer readable storage medium, for example, the memory 1102 or the memory 1202 including instructions, is also provided, which can be executed by the processor 1101 of the terminal 1100 or the processor 1201 of the server 1200 to complete the above-mentioned method for training the video generation model. Optionally, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0203] A computer program product including computer programs / instructions which, when executed by a processor, implement the above-mentioned method for training the video generation model.

[0204] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such departures from the present disclosure that come within known use or custom in the art to which the present disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0205] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A training method for a video generation model, characterized in that, The method includes: Obtain sample effect videos and effect prompts for multiple special effects, wherein the effect prompts are used to indicate the special effects in the video to be generated by the video generation model; For any one of the multiple special effects, the video generation model processes the first frame image of the sample special effect video of the special effect and the special effect prompt words of the special effect to obtain a first special effect video. The first frame image is used to indicate the main body in the video to be generated by the video generation model. The first special effect video is a special effect video automatically generated by the video generation model based on the first frame image and the special effect prompt words. Based on the first special effect video and sample special effect videos corresponding to the multiple special effects, the model loss of the video generation model is determined, and the video generation model is trained with the goal of minimizing the model loss.

2. The training method for the video generation model according to claim 1, characterized in that, For any one of the plurality of special effects, the video generation model processes the first frame image of the sample special effect video and the special effect prompt text to obtain a first special effect video, including: For any one of the multiple special effects, the first frame image in the sample special effect video of the special effect is encoded by the image encoder in the video generation model to obtain the first image encoding feature; The special effect prompts are encoded using the text encoder in the video generation model to obtain text encoding features; The first image encoding features and the first text encoding features are processed by the video generation network in the video generation model to obtain the first special effects video.

3. The training method for the video generation model according to claim 2, characterized in that, The step of processing the first image encoding features and the text encoding features through the video generation network in the video generation model to obtain the first special effects video includes: The first image encoding features and the first text encoding features are processed by the video generation network in the video generation model to obtain the first video features, which are used to indicate the first special effects video. The step of determining the model loss of the video generation model based on the first special effect video and sample special effect videos corresponding to the multiple special effects, and training the video generation model with the objective of minimizing the model loss, includes: For any one of the plurality of special effects, a first loss is determined based on the first video features and the sample video features of the sample special effect video of the special effect, wherein the first loss is used to represent the difference between the first video features and the sample video features; The video generation model is trained with the goal of minimizing the first loss.

4. The training method for the video generation model according to claim 2 or 3, characterized in that, Training the video generation model includes: Freeze the parameters in the structure other than the text encoder in the video generation model, and adjust the parameters of the text encoder in the video generation model.

5. The training method for the video generation model according to claim 1, characterized in that, The method further includes: For any one of the multiple special effects, the sample video features of the sample special effect video are encoded by the special effect encoder in the video generation model to obtain special effect features, which are used to indicate the attributes of the special effect in the sample special effect video. The video generation model is used to process the special effects features, the first frame image in the sample special effects video, and the special effects prompts to obtain a second special effects video. Based on the second special effects video and sample special effects video corresponding to the multiple special effects, the video generation model is trained again.

6. The training method for the video generation model according to claim 5, characterized in that, For any one of the plurality of special effects, the sample video features of the sample special effect video are encoded through the special effect encoder in the video generation model to obtain the special effect features, including: For any one of the multiple special effects, the sample video features are processed by the attention layer in the special effect encoder to obtain the query features, key features, and value features of the attention layer. The query features and key features are used to indicate the position of the special effect in the sample special effect video, and the value features are used to indicate the attributes of the special effect. The query feature, the key feature, and the value feature are fused to obtain the special effect feature.

7. The training method for the video generation model according to claim 5, characterized in that, The process of using the video generation model to process the special effects features, the first frame image of the sample special effects video, and the special effects prompts to obtain the second special effects video includes: The video generation network in the video generation model processes the special effects features, the first image encoding features corresponding to the first frame image, and the text encoding features corresponding to the special effects prompt words to obtain the second video features, which are used to indicate the second special effects video. The process of retraining the video generation model based on the second special effects video and sample special effects videos corresponding to the multiple special effects includes: For any one of the plurality of special effects, a second loss is determined based on the second video features and the sample video features of the sample special effect video of the special effect, wherein the second loss is used to represent the difference between the second video features and the sample video features; The video generation model is trained with the goal of minimizing the second loss.

8. The training method for the video generation model according to any one of claims 5-7, characterized in that, The retraining of the video generation model includes: Freeze the parameters in the other structures of the video generation model except for the special effects encoder, and adjust the parameters in the special effects encoder of the video generation model.

9. The training method for the video generation model according to claim 1, characterized in that, The video generation model also includes multiple special effects adapters, which correspond one-to-one with each network layer in the video generation network of the video generation model. The video generation network is used to generate special effects videos. The method further includes: For any one of the multiple special effects, the image encoder in the video generation model encodes any frame image in the sample special effect video of the special effect to obtain a second image encoding feature; For the special effects adapter corresponding to the i-th network layer in the video generation network, the third video feature and the second image coding feature are processed by the special effects adapter to obtain the fourth video feature. The third video feature is the input of the i-th network layer and is used to indicate the special effects video containing the special effects generated by the (i-1)-th network layer. The fourth video feature is fused with the output of the i-th network layer to obtain the input of the (i+1)-th network layer; Based on the target video features and sample effect videos output by the last network layer in the video generation network, multiple effect adapters in the video generation model are trained.

10. The training method for the video generation model according to claim 9, characterized in that, The effect adapter corresponding to the i-th network layer in the video generation network processes the third video feature and the second image coding feature to obtain the fourth video feature, including: For the special effects adapter corresponding to the i-th network layer in the video generation network, the second image coding feature is broadcast to the image features corresponding to each image frame in the third video feature through the special effects adapter, thereby obtaining the third video feature.

11. The training method for the video generation model according to claim 1, characterized in that, The multiple special effects include interactive special effects, which are used to indicate changes in both the subject and the environment in the video.

12. A training device for a video generation model, characterized in that, The device includes: The acquisition module is configured to acquire sample effect videos and effect prompts for multiple effects, wherein the effect prompts are used to indicate the effects in the video to be generated by the video generation model; The first processing module is configured to perform the following operation for any one of the plurality of special effects: using the video generation model, processing the first frame image of the sample special effect video of the special effect and the special effect prompt words of the special effect to obtain a first special effect video. The first frame image is used to indicate the main body in the video to be generated by the video generation model. The first special effect video is a special effect video automatically generated by the video generation model based on the first frame image and the special effect prompt words. The training module is configured to execute a first special effect video and sample special effect videos corresponding to the multiple special effects, determine the model loss of the video generation model, and train the video generation model with the goal of minimizing the model loss.

13. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the training method of the video generation model as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the training method of the video generation model as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the training method of the video generation model according to any one of claims 1 to 11.

Images