Verified video model training method, and method and device for generating video based on text

Abstract

The invention provides a text video model training method and a text-based video generation method and device, and relates to the technical field of computers, in particular to the artificial intelligence fields of large models, reinforcement learning, supervised learning and the like. The specific implementation scheme is as follows: inputting a description text into a first text video model to obtain a to-be-detected video; inputting the to-be-detected video into a video detection model to obtain a detection result corresponding to the to-be-detected video; and training the first text video model according to the detection result and a training sample to obtain a second text video model. According to the embodiment of the invention, the to-be-detected video generated by the text video model can be detected through the video detection model, and then the text video model is retrained according to the detection result, so that the quality of the video generated by the model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to artificial intelligence fields such as large models, reinforcement learning, and supervised learning. Background Art

[0002] Text-based video generation (abbreviated as text-based video) technology is a technology that uses artificial intelligence technology to convert text content into visual and auditory effects. In text-based video technology, a video can be generated by inputting a piece of text into the model. However, the video generated by the model may not necessarily conform to the description of the text. Summary of the invention

[0003] The present invention provides a text-based video model training method, a method, an apparatus, a device and a storage medium for generating a video based on text.

[0004] According to one aspect of the present disclosure, a method for training a Vincent video model is provided, comprising:

[0005] Input the description text into the first video model to obtain the video to be detected;

[0006] Input the video to be detected into the video detection model to obtain the detection result corresponding to the video to be detected;

[0007] According to the detection result, the first cultural video model is trained to obtain a second cultural video model.

[0008] According to another aspect of the present disclosure, a method for generating a video based on text is provided, comprising:

[0009] The description text is input into the Vincent video model to obtain a video to be detected; wherein the Vincent video model is trained based on the Vincent video model training method mentioned above.

[0010] According to another aspect of the present disclosure, a training device for a Vincent video model is provided, comprising:

[0011] A text processing module, used for inputting the description text into the first text video model to obtain a video to be detected;

[0012] The video processing module is used to input the video to be detected into the video detection model to obtain the detection result corresponding to the video to be detected;

[0013] The training module is used to train the first cultural video model according to the detection result to obtain a second cultural video model.

[0014] According to another aspect of the present disclosure, there is provided an apparatus for generating a video based on text, comprising:

[0015] The text processing module is used to input the description text into the Vincent video model to obtain the video to be detected; wherein, the Vincent video model is trained based on the above-mentioned Vincent video model training method.

[0016] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any embodiment of the present disclosure.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to any embodiment of the present disclosure.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the method according to any one of the embodiments of the present disclosure is implemented.

[0022] The disclosed embodiment can detect the to-be-detected video generated by the Vincent video model through the video detection model, and then retrain the Vincent video model according to the detection result, thereby improving the quality of the video generated by the model.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0025] Figure 1 is a flow chart of a Vincent video model training method according to an embodiment of the present disclosure;

[0026] Figure 2 is a flow chart of a Vincent video model training method according to another embodiment of the present disclosure;

[0027] Figure 3 is a flow chart of a Vincent video model training method according to another embodiment of the present disclosure;

[0028] Figure 4is a flow chart of a Vincent video model training method according to another embodiment of the present disclosure;

[0029] Figure 5 This is the overall flow chart of optimizing the Vincent video model based on reinforcement learning;

[0030] Figure 6 It is the overall flow chart of optimizing the Wensheng video model and rewriting the model based on reinforcement learning;

[0031] Figure 7 is a schematic block diagram of a training device for a Vincent video model according to an embodiment of the present disclosure;

[0032] Figure 8 is a schematic block diagram of a training device for a Vincent video model according to another embodiment of the present disclosure;

[0033] Fig. 9 is a schematic block diagram of an apparatus for generating a video based on text according to an embodiment of the present disclosure;

[0034] Fig.10 is a schematic block diagram of an apparatus for generating a video based on text according to another embodiment of the present disclosure;

[0035] Fig.11 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0037] The “and / or” in the embodiments of the present disclosure indicates that three relationships may exist. For example, A and / or B may indicate the following three situations: A exists alone, A and B exist at the same time, and B exists alone. The term “at least one” herein indicates any combination of at least two of any one or more of a plurality of. For example, at least one of A, B, and C may indicate any one or more elements selected from the set consisting of A, B, and C. The terms “first” and “second” herein refer to and distinguish between multiple similar technical terms, and do not mean to limit the order or to limit the meaning to only two. For example, the first feature and the second feature refer to two types / two features. The first feature may be one or more, and the second feature may also be one or more.

[0038] The training process of the Wensheng video model includes the training of the text-side model and the training of the video-side model. For example, the text-side can use pre-trained transformer models such as Text to Text Transfer Transformer (T5), Generative Pre-Training (GPT), etc. to model text input. The video-side model structure can use the Diffusion Transformer structure, which is modeled as an input of length L according to the tuple splitting method of the Video Vision Transformer (ViViT). The training method can be a diffusion modeling process, which can enable the model to be trained based on text guidance and to continuously restore the original video based on random noise. The text-side model and the video-side model are associated through the cross-attention mechanism.

[0039] Figure 1 is a flow chart of a method for training a Vincent video model according to an embodiment of the present disclosure, comprising:

[0040] S110, inputting the description text into the first text-generated video model to obtain a video to be detected;

[0041] S120, inputting the video to be detected into a video detection model to obtain a detection result corresponding to the video to be detected;

[0042] S130: According to the detection result, the first language video model is trained to obtain a second language video model.

[0043] According to the embodiments of the present disclosure, the description text may be input text or text sampled from historical data. The first Vincent video model may be a Vincent video model that needs to be trained or a pre-trained Vincent video model. The Vincent video model may be a large model or a model dedicated to Vincent videos. For example, models such as Sora, Pika, Stable Video Diffusion, and Transformer. The first Vincent video model may be pre-trained using supervised learning, and the first Vincent video model may be used as a hot start model. Further intensive training may be performed on this basis to obtain a second Vincent video model.

[0044] According to an embodiment of the present disclosure, after the description text is input into the first Vincent video model, multiple random videos can be generated using different random seeds, and the multiple random videos are multiple videos to be detected. For example, different random seeds can include different dropout layer (Dropout) positions in the Transformer model. Dropout is a regularization method in a neural network, which can randomly delete some neurons during the training process of the model. Other schemes can also be used to generate multiple videos to be detected, such as using uncertainty sampling when the Vincent video model performs result sampling to obtain multiple videos to be detected.

[0045] According to an embodiment of the present disclosure, a video detection model may be pre-trained as a reward model for a pre-trained first-language video model. The video detection model may be trained using a model such as ViViT, an encoder, a convolutional neural network (CNN), or a transformer. The training data of the video detection model may include training samples related to video quality. For example, training samples related to video text relevance, video aesthetics, video clarity, etc.

[0046] According to an embodiment of the present disclosure, the first Vincent video model can be trained and iterated according to the detection result output by the video detection model to obtain the second Vincent video model. For example, the training process of the Vincent video model can use the method of reinforcement learning with human feedback (RLHF) to train the iterative Vincent video model. RLHF is a method for training artificial intelligence systems that combines reinforcement learning with human feedback, by using human feedback to create a reward signal, and then using this signal to improve the optimization function of the model through reinforcement learning, so that the model can better capture complex human preferences.

[0047] According to the embodiment of the present disclosure, if a plurality of videos to be detected are obtained after the description text is input into the first video model, each video to be detected can be input into the video detection model to obtain the detection result corresponding to each video to be detected. Then, the corresponding optimization function can be obtained respectively according to the detection result corresponding to each video to be detected, or the optimization function can be obtained comprehensively. Then, the parameters of the first video model are adjusted according to the optimization function. The model after each adjustment is used as the first video model for the next training, and the second video model can be obtained after the optimization function converges.

[0048] Through the embodiments of the present disclosure, the video to be detected generated by the Vincent video model can be detected by the video detection model, and then the Vincent video model can be retrained according to the detection results, thereby improving the quality of the video generated by the model.

[0049] Figure 2 is a flow chart of a method for training a Wensheng video model according to another embodiment of the present disclosure. The method may include one or more features of the above method embodiments. In one embodiment, the video detection model includes a video text relevance model and a video aesthetics model; step S120 inputs the video to be detected into the video detection model to obtain a video detection result corresponding to the video to be detected, including:

[0050] S210, inputting the description text and the video to be detected into the video text correlation model to obtain the correlation of the video to be detected; and / or

[0051] S220: Input the video to be detected into the video aesthetics model to obtain the aesthetics of the video to be detected.

[0052] According to an embodiment of the present disclosure, the video detection model can be used as a reward model for the Vincent video model.

[0053] In some examples, the video detection model may include a video text relevance model. Video-text pairs can be used as training data for the video text relevance model. The text side can be modeled using structures such as bidirectional encoder representation (Bidirectional Encoder Representation from Transformers, BERT) based on the self-attention model, and the video side can be modeled by encoder structures such as ViViT, CNN or Transformer. The classification (Classification, CLS) feature vectors of the video side and the text side are extracted respectively as the classification vectors of the video side and the text side, and comparative learning training is performed. For example, a certain video A and its corresponding text are used as positive examples for comparative training, indicating that video A has a high correlation with the text. Other samples in the same batch that are not related to video A are used as negative examples, and the negative examples indicate that video A has no correlation with the text. One or more pairs of videos to be detected and description texts are input into the trained video text relevance model, and the relevance of the video to be detected is obtained by the cosine similarity of the video-side CLS classification and the text-side CLS classification of the model. The relevance can be represented by a score, so it can be called a relevance score. For example, the relevance score between video A1 and text T1 is 0.7, and the relevance score between video A2 and text T1 is 0.5.

[0054] In some examples, the video detection model may include a video aesthetics model. The video aesthetics model may be modeled using an encoder structure such as ViViT. Using video-side CLS classification may represent regression prediction of an aesthetics score. By inputting one or more videos to be detected into a trained video aesthetics model, the aesthetics of each video to be detected may be obtained. Aesthetics can be represented by a score, so it may be called an aesthetics score. For example, the aesthetics score of video A1 is 8, and the aesthetics score of video A2 is 9.

[0055] If the Vincent video model outputs multiple videos to be detected according to the description text, the multiple videos to be detected and the description text can be combined into multiple text-video pairs. The multiple text-video pairs are respectively input into the trained video-text relevance model to obtain the video-text relevance of each video to be detected. In addition, the multiple videos to be detected can be input into the trained video aesthetics model to obtain the aesthetics of each video to be detected. In S130, the first Vincent video model is trained according to the relevance and aesthetics of the videos to be detected to obtain the second Vincent video model. For example, an optimization function can be calculated according to the relevance and aesthetics of each video to be detected, and the parameters of the first Vincent video model can be adjusted according to the optimization function. The model after each adjustment is used as the first Vincent video model for the next training, and the second Vincent video model can be obtained after the optimization function converges.

[0056] Through the embodiments of the present disclosure, the correlation between the video and text generated by the Wensheng video model and / or the video aesthetics can be detected, which facilitates the selection of highly correlated and / or highly aesthetic videos for presentation, thereby improving satisfaction with the videos generated by the model.

[0057] In one embodiment, S130 trains the first Vincent video model according to the detection result to obtain the second Vincent video model, including: adjusting the parameters of the first Vincent video model according to the first optimization function to obtain the second Vincent video model; wherein the first optimization function is determined according to the detection result, the training sample and the first probability of generating the video to be detected based on the description text. This step can be performed multiple times, and each time it can be determined whether the model needs to be updated according to the calculation result of the optimization function, and if the model needs to be updated, the parameters of the first Vincent video model are adjusted. The adjusted model is used as the first Vincent video model for the next training, and the training continues. If the model does not need to be updated, the model updated last time can be used as the trained second Vincent video model. The first optimization function includes the first probability of generating the video to be detected based on the description text. By training the Vincent video model with the first optimization function, the probability of the model generating a video with a high aesthetic score and a high similarity score can be increased, and the probability of the model generating a video with a low aesthetic score and a low similarity score can be reduced. The first probability of generating the video to be detected based on the description text can be an output result of the Vincent video model. For example, the Wensheng video model can output, based on the input text T: video A and its corresponding first probability, video B and its corresponding first probability.

[0058] In one embodiment, the training samples include reinforcement learning samples and supervised learning samples, the reinforcement learning samples include unlabeled texts, and the supervised learning samples include labeled data of texts and videos.

[0059] According to an embodiment of the present disclosure, the training samples used in the first optimization function may include reinforcement learning samples and supervised learning samples. An example of the first optimization function is as follows:

[0060]

[0061] Where x represents the video generated by the first video model; z represents the description text input into the first video model; D unlabeled Represents samples for reinforcement learning (i.e. reinforcement learning samples, no corresponding video annotation is required); D labeled represents the sample for supervised learning (i.e., supervised learning sample, used to keep the output of the model stable for supervised samples). θ represents the model parameter to be optimized; p θ (x|z) represents the probability of generating x based on z under the current model (i.e., the first probability). simφ (x, z) represents the relevance score output by the video text relevance model, r aesφ (x) represents the aesthetic score output by the video aesthetic model. γ and β are adjustment coefficients. Represents the average value and is used to determine the overall optimization goal. The adjustment coefficient is generally a hyperparameter in the model training process.

[0062] D unlabeled The sample may include a text sample, such as text in historical data, and z may be a sample from D unlabeled D labeled The sample may include a video-text pair, may be training data used in a pre-trained first-text video model, may be data sampled from the training data, or may be other supervised learning samples.

[0063] According to an embodiment of the present disclosure, the training sample, the first probability, and the detection result output by the video detection model are substituted into the first optimization function to calculate the value of the first optimization function. According to the value of the first optimization function, it can be determined whether the model needs to be updated. For example, if the value of the first optimization function does not converge, the model needs to be updated. If the value of the first optimization function converges, a second text video model can be obtained.

[0064] Through the embodiment of the present disclosure, the Vincent video model can be iteratively trained through the first optimization function, thereby increasing the probability of generating videos with high detection result scores, reducing the probability of generating videos with low detection result scores, and improving the satisfaction of generated videos.

[0065] In one embodiment, Figure 3 As shown, the method also includes:

[0066] S310, inputting the initial text into the rewriting model to obtain a description text.

[0067] According to the embodiments of the present disclosure, the initial text may be an input text or a text sampled from historical data. A rewriting model may be added to rewrite the initial text, for example, the grammar and logic of the initial text may be modified, and the initial text may be expanded according to the keywords of the initial text. In this way, a more detailed, more reasonable, and more fluent description text may be generated using the rewriting model, and the rationality, accuracy, or richness of the text input into the Wensheng video model may be improved. The rewritten description text may be input into the Wensheng video model, and the model training efficiency may be improved, and the quality of the video output by the model may be improved.

[0068] In one embodiment, according to the detection result, the first Vincent video model is trained to obtain a second Vincent video model, including: according to the detection result, the first Vincent video model and the rewriting model are jointly trained to obtain a second Vincent video model and an updated rewriting model.

[0069] According to an embodiment of the present disclosure, the rewriting model can be trained using the training data of the first video model. The rewriting model can be modeled using a model such as a Transformer model.

[0070] Through the disclosed embodiments, the rewriting model can be pre-trained, and the training samples of the rewriting model can be the same as the training samples of the text end of the Wensheng video model, or other training samples can be used. After the initial text is input into the pre-trained rewriting model, the rewriting model can output the rewritten description text. After the description text is input into the first Wensheng video model, the first Wensheng video model can output multiple videos to be detected. After the multiple videos to be detected are respectively input into the video detection model, the video detection model can output the detection result of each video to be detected. An optimization function can be calculated according to the detection result output by the video detection model, and the parameters of the first Wensheng video model and the rewriting model can be modified according to the optimization function. For example, according to the relevance score output by the video text relevance model and the aesthetic score output by the video aesthetic model, the total score of the video detection result can be obtained. The total score can be equal to the sum or weighted sum of the relevance score and the aesthetic score. According to the total score, the optimization function for joint training can be calculated. There is no restriction on the order of modifying the parameters of the first Wensheng video model and the rewriting model using the optimization function, and they can be processed in parallel or in a sequential order. Using the same detection results to jointly train the two models can improve training efficiency and increase the relevance of the text rewritten by the rewriting model and the video generated by the text-generated video model.

[0071] In one embodiment, according to the detection result, the first Vincent video model and the rewriting model are jointly trained to obtain the second Vincent video model and the updated rewriting model, including: adjusting the parameters of the first Vincent video model according to the second optimization function to obtain the second Vincent video model and the updated rewriting model; wherein the second optimization function is based on the detection result, the training sample, the second probability of generating the video to be detected based on the description text, and the third probability of generating the rewriting text based on the description text. This step can be performed multiple times, and each time it is first determined whether the model needs to be updated according to the calculation result of the second optimization function, and if the model needs to be updated, the parameters of the first Vincent video model and / or the rewriting model are adjusted. The adjusted model is used as the first Vincent video model and / or the rewriting model for the next training to continue training. If the model does not need to be updated, the Vincent video model after the last update can be used as the trained second Vincent video model, and / or the rewriting model after the last update can be used as the trained rewriting model.

[0072] According to an embodiment of the present disclosure, the second probability of generating a video to be detected based on the description text may be an output result of the Vincent video model. For example, the Vincent video model may output: video A and its corresponding second probability, video B and its corresponding second probability based on the input text T. The third probability of generating a rewritten text based on the description text may be an output result of the rewriting model. For example, the rewriting model may output the rewritten text T2 and its corresponding third probability based on the input text T1.

[0073] According to an embodiment of the present disclosure, the training samples used in the second optimization function may include reinforcement learning samples and supervised learning samples. An example of a second optimization function for joint training is as follows:

[0074]

[0075] Among them, x represents the video generated by the second-language video model, z represents the description text of the second-language video model, and D unlabeled represents samples for reinforcement learning (no corresponding video annotation is required), D labeled represents the sample for supervised learning (used to keep the model's output stable for supervised samples). θ represents the model parameter to be optimized; z rew Represents the description text after rewriting the model; ori represents the initial text; p θ (x|z rew ) represents the rewritten description text z under the current model rew The probability of generating x (i.e. the second probability); p θ (z rew |z ori ) represents the description text z before rewriting based on the current model ori Generate rewritten description text z rew The probability of (i.e., the third probability); r φ (x,z ori ) represents the scoring weight of the aesthetics model and the video text relevance model; r simφ (x|z ori ) represents the relevance score of generating video x through the initial text; r simφ (x,z ori ) represents the relevance score output by the video text relevance model; r aesφ (x) represents the video aesthetics model score. γ and β are adjustment coefficients. Represents the average value. The adjustment coefficient is generally a hyperparameter in the model training process.

[0076] According to the embodiments of the present disclosure, by iteratively training the Vincent video model and the rewriting model through the second optimization function and using the same training samples to calculate the optimization function, it can be ensured that during the joint training process, the rewriting model and the Vincent video model will not deviate from the original intention of the user, thereby increasing the probability of generating videos with high detection result scores, reducing the probability of generating videos with low detection result scores, and improving the satisfaction of the generated videos.

[0077] In some examples, after adding the rewriting model, only the Vincent video model can be trained without training the rewriting model. Specifically, after the description text is rewritten by the rewriting model to the initial text, the description text is input into the first Vincent video model to generate the video to be detected. The initial text and the video to be detected are input into the video text correlation model to obtain the correlation score, and the video to be detected is input into the aesthetics model to obtain the aesthetics score. Then, the correlation score and the aesthetics score are used to calculate the above-mentioned first optimization function, and the first Vincent video model is trained using the first optimization function to obtain the second Vincent video model.

[0078] In one embodiment, Figure 3 As shown, it also includes:

[0079] S320: annotate the video to be detected to obtain samples for updating the video detection model.

[0080] According to the embodiments of the present disclosure, the detection results of the video training samples to be detected output by the Vincent video model can be annotated. For example, the videos output by the Vincent video model include videos A, B, and C, wherein the aesthetics of video A can be annotated as 8, the aesthetics of video B can be annotated as 7, and the aesthetics of video C can be annotated as 6. The specific annotation basis can refer to the output results of the video aesthetics model, or it can be annotated based on experience. For example, the output result of the video A input video aesthetics model is 7.5, and the viewing degree of video A can be annotated as 8. For another example, the aesthetics of video B can be annotated as 7 based on experience. The video to be detected can also be annotated with reference to the clarity, length, content, etc. of the video. The annotated samples can be used as new training samples for the video detection model; the video detection model is further trained, the video detection model is updated, and the accuracy of the retrieval results of the video detection model is improved, thereby improving the training efficiency of the Vincent video model and / or the rewriting model and the quality of the generated video.

[0081] Figure 4 A method for generating a video based on text according to an embodiment of the present disclosure may include:

[0082] S410, inputting the description text into a Vincent video model to obtain a video to be detected; wherein the Vincent video model is trained based on the Vincent video model training method described above.

[0083] According to an embodiment of the present disclosure, a description text may be input into a trained Vincent video model, and the Vincent video model may generate one or more videos A, B, C, ..., N to be detected based on the description text. The video to be detected may also be referred to as a candidate video. For example, the description text input into the model is T. The generated videos A, B, C, ..., N to be detected may contain one or more elements mentioned in the description text.

[0084] Through the embodiments of the present disclosure, by inputting description text into the Wensheng video model trained in the above embodiments, a high-quality video related to the description text can be generated.

[0085] In one embodiment, it further includes:

[0086] S420: Input the video to be detected into a video detection model to obtain a video detection result corresponding to the video to be detected.

[0087] Through the embodiments of the present disclosure, a video detection model can be used to detect one or more videos to be detected generated by the Wensheng video model. The quality of the video can be detected in a variety of ways. For example, a variety of video detection models can be pre-trained to detect and obtain a variety of detection results such as the text-video relevance, video aesthetics, and video clarity of the video.

[0088] In one implementation, the video detection model includes a text-video correlation model and a video aesthetics model. Step S420 inputs the video to be detected into the video detection model to obtain a detection result corresponding to the video to be detected, including:

[0089] Inputting the description text and the video to be detected into the text-video correlation model to obtain the correlation of the video to be detected; and / or

[0090] The video to be detected is input into the video aesthetics model to obtain the aesthetics of the video to be detected.

[0091] According to an embodiment of the present disclosure, one or more videos to be detected are respectively input into a text-video correlation model and / or a video aesthetics model to obtain the corresponding text-video correlation and video aesthetics of each video to be detected, and the text-video correlation and video aesthetics are used to characterize the detection results of the video to be detected.

[0092] Through the embodiments of the present disclosure, the text-video relevance score and the video aesthetics score are used as evaluation criteria for the video to be detected, which can improve the quality of the model-generated video and thus improve the satisfaction.

[0093] In one embodiment, it further includes:

[0094] According to the detection result, the multiple videos to be detected are sorted.

[0095] According to the embodiments of the present disclosure, if multiple detection results can be output through multiple video detection models, for example, the relevance scores of multiple videos to be detected are output through the video text relevance model, and the aesthetics scores are output through the video aesthetics model. The multiple videos to be detected can be sorted from high to low according to the relevance scores, or the multiple videos to be detected can be sorted from high to low according to the aesthetics scores, or the total score can be calculated by combining the relevance scores and the aesthetics scores before sorting. Through the embodiments of the present disclosure, by using the video detection results as the sorting criteria for the videos, high-quality videos with better detection results can be preferentially displayed in the display interface.

[0096] In one embodiment, it further includes:

[0097] The original text is rewritten into the description text using the rewriting model.

[0098] According to the disclosed embodiment, when the initial text cannot accurately and specifically generate a video, the rewriting model can be used to rewrite the initial text. The rewriting model can also be used to rewrite each initial text. The rewritten text is used as the description text of the input text video model. The rewriting model of this embodiment can use a pre-trained rewriting model, or can use the rewriting model updated after joint training in the above embodiment.

[0099] Through the embodiments of the present disclosure, a rewriting model can be used to generate a more detailed, more reasonable, and more fluent description text, thereby improving the rationality, accuracy, or richness of the text input into the Vincent video model. Inputting the rewritten description text into the Vincent video model can improve the quality of the video output by the model.

[0100] In some application scenarios, a video detection model can be trained as a reward model for the text video model. The video detection model includes a video text relevance model and a video aesthetics model. These two models can be constructed using contrastive learning training, supervised learning training, and video classification (CLS) techniques.

[0101] For example, the training data used for video-text relevance model training is video-text pairs. The video side can use the ViViT model (or other common video encoder structures, such as CNN or other variants of Transformer) for modeling, and the text side can use pre-trained models such as BERT. The CLS classification representation of the video side and the CLS classification representation of the text side are used for comparative learning and training. The video and the text corresponding to the video are used as positive examples of training samples, and the video and other texts in the same batch that do not correspond to the video are used as negative examples of training samples. The Cosine similarity of the CLS of the video side and the text side can be used as the relevance score of the video-text relevance model.

[0102] For example, the data used for video aesthetics model training is annotated data (for example, the data is in the form of a video and the aesthetics score of the video by its annotator). The video aesthetics model can be modeled using a video encoder structure such as ViViT. The CLS classification representation on the video side is used for regression prediction of the aesthetics score.

[0103] After completing the training of the video text relevance model and the video aesthetics model, they are trained as reward models. This solution can be trained using reinforcement learning of a multi-reward model.

[0104] Figure 5 This is the overall flow chart of optimizing the Wensheng video model based on reinforcement learning. Figure 5 As shown, the optimization process can also be understood as a training process. The models mainly used in the optimization process include a text video model 501, a video text correlation model 502 (also called a text video correlation model, etc.) and a video aesthetics model 503.

[0105] First, the text (request text) is input into the Vincent video model 501. The Vincent video model 501 can generate multiple random videos (or candidate videos) using different random seeds (for example, controlling different Dropout positions in the Transformer model). The multiple random videos generated by the Vincent video model 501 are respectively input into the video text relevance model 502 and the video aesthetics model 503 to obtain the text video relevance score and the video aesthetics score. The sum of the relevance score and the aesthetics score (which can be weighted) is used as the overall generated video reward. During the training process, the RLHF training process can be used to iterate the Vincent video model.

[0106] An example of an optimization formula is as follows:

[0107]

[0108] Where x represents the video output by the Wensheng video model, and z represents the input text. unlabeled Represents samples for reinforcement learning (i.e. reinforcement learning samples, no corresponding video annotation is required); D labeled represents the sample for supervised learning (used to keep the model's output stable for supervised samples). θ represents the model parameter to be optimized, p θ (x|z) represents the probability of generating x based on z under the current model. simφ (x, z) represents the relevance score of the text-video model, r aesφ (x) represents the video aesthetics model score. γ and β are adjustment coefficients.

[0109] The main purpose of model optimization is to enable the model to generate multiple candidate videos through reinforcement learning, increase the probability of generating videos with high aesthetics and similarity scores, and reduce the probability of generating videos with low aesthetics and similarity scores.

[0110] like Figure 6 As shown in Figure 2, the overall process of optimizing the Wensheng video model based on reinforcement learning can also be Figure 5 On this basis, a rewriting model 601 is added to rewrite the input initial text to generate a more detailed and more suitable input text for the text-generated video model.

[0111] The rewriting model 601 can be trained using annotated data. The annotated data can include input content and rewritten content that is identical or similar in meaning to the input content. The rewritten content can enable the Vincent model to generate better video results. The data is obtained by sampling the Vincent video model. If there is no annotated data, the same input (i.e., the request (Query) before and after the rewriting is the same) can be used as training data for the rewriting model. In one example, the rewriting model can use a Transformer model structure.

[0112] In addition to optimizing the Wensheng video model itself, the reinforcement learning process can also optimize the rewritten model. An example of an optimization formula is as follows:

[0113]

[0114] Among them, z rew Represents the rewritten input content, z ori represents the input content before rewriting. x represents the video generated by the second-language video model, z represents the description text of the second-language video model, and D unlabeled represents the sample for reinforcement learning, D labeled represents the sample for supervised learning. θ represents the model parameter to be optimized; p θ (x|z rew ) represents the rewritten description text z under the current model rew The probability of generating x; p θ (z rew |z ori ) represents the description text z before rewriting based on the current model ori Generate rewritten description text z rew The probability of φ (x,z ori ) represents the scoring weight of the aesthetics model and the video text relevance model; r simφ (x|z ori ) represents the relevance score of generating video x through the initial text; r simφ (x,z ori) represents the relevance score output by the video text relevance model; r aesφ (x) represents the video aesthetics model score. Y and β are adjustment coefficients. This optimization formula can use the original input content to calculate the similarity score and supervised training to ensure that the rewritten model does not deviate from the user's original intention.

[0115] After the rewriting model is introduced, the overall process of optimizing the Vincent video and the rewriting model based on reinforcement learning includes: inputting the initial text into the rewriting model 601. The rewriting model rewrites the initial text to obtain a rewritten description text (which can be referred to as the rewritten text). The rewritten text is input into the Vincent video model 501 to generate multiple videos to be detected (or multiple candidate videos). Multiple video-text pairs consisting of multiple videos to be detected and the initial text are respectively input into the video-text relevance model 502 to obtain the relevance score of each video to be detected. Multiple videos to be detected are respectively input into the video aesthetics model 503 to obtain the aesthetics score of the video to be detected. The relevance score and the aesthetics score gradient are returned to the Vincent video model 501 and the rewriting model 601 to train the Vincent video model 501 and the rewriting model 601.

[0116] In some examples, after the video generated by the Vincent video model is annotated, the annotated data can be used to train the text-video related model and / or the aesthetics model. That is, further reinforcement learning is performed based on the model after reinforcement learning to achieve cyclic iteration. By training the video detection model as the reward model of the Vincent video model through reinforcement learning technology, the video generation process can take into account the goals of video-text similarity and video aesthetics, and the generated video has higher satisfaction.

[0117] Figure 7 A training device for a Vincent video model according to an embodiment of the present disclosure includes:

[0118] The text processing module 701 is used to input the description text into the first text video model to obtain the video to be detected;

[0119] The video processing module 702 is used to input the video to be detected into a video detection model to obtain a detection result corresponding to the video to be detected;

[0120] The training module 703 is used to train the first cultural video model according to the detection result to obtain a second cultural video model.

[0121] In one embodiment, the video detection model includes a video text relevance model and a video aesthetics model, and the video processing module 702 is used to:

[0122] Inputting the description text and the video to be detected into the video text correlation model to obtain the correlation of the video to be detected; and / or

[0123] The video to be detected is input into the video aesthetics model to obtain the aesthetics of the video to be detected.

[0124] In one embodiment, the training module 703 is used to:

[0125] According to a first optimization function, the parameters of the first video model are adjusted to obtain a second video model; wherein the first optimization function is determined according to the detection result, the training sample and a first probability of generating the video to be detected based on the description text.

[0126] Figure 8 is a schematic block diagram of a training device for a Vincent video model according to another embodiment of the present disclosure. The device may include one or more features of the above-mentioned device. In one embodiment, the device further includes:

[0127] The rewriting module 801 is used to input the initial text into the rewriting model to obtain the description text.

[0128] In one embodiment, the training module 703 includes:

[0129] The joint training submodule 802 is used to jointly train the first language video model and the rewriting model according to the detection result to obtain a second language video model and an updated rewriting model.

[0130] In one embodiment, the joint training submodule 802 is used to:

[0131] According to the second optimization function, the parameters of the first language video model are adjusted to obtain the second language video model and the updated rewriting model; wherein the second optimization function is based on the detection result, the training sample, the second probability of generating the video to be detected based on the description text, and the third probability of generating the rewritten text based on the description text.

[0132] In one embodiment, the training samples include reinforcement learning samples and supervised learning samples, the reinforcement learning samples include unlabeled texts, and the supervised learning samples include labeled data of texts and videos.

[0133] In one embodiment, the device further comprises:

[0134] The labeling module 803 is used to label the video to be detected to obtain samples for updating the video detection model.

[0135] Fig. 9is a schematic block diagram of an apparatus for generating a video based on text according to an embodiment of the present disclosure, comprising:

[0136] The text processing module 901 is used to input the description text into the Vincent video model to obtain the video to be detected; wherein the Vincent video model is trained based on the Vincent video model training method of any one of claims 1 to 8.

[0137] Fig.10 is a schematic block diagram of an apparatus for generating a video based on text according to another embodiment of the present disclosure. The apparatus may include one or more features of the above apparatus. In one embodiment, the apparatus further includes:

[0138] The video processing module 1001 is used to input the video to be detected into a video detection model to obtain a detection result corresponding to the video to be detected.

[0139] In one implementation, the video detection model includes a text-video correlation model and a video aesthetics model, and the video processing module 1001 includes:

[0140] A correlation processing module 1002 is used to input the description text and the video to be detected into the text-video correlation model to obtain the correlation of the video to be detected; and / or

[0141] The aesthetics processing module 1003 is used to input the video to be detected into the video aesthetics model to obtain the aesthetics of the video to be detected.

[0142] In one embodiment, the device further comprises:

[0143] The sorting module 1004 is used to sort the multiple videos to be detected according to the detection result.

[0144] In one embodiment, the device further comprises:

[0145] The rewriting module 1005 is used to rewrite the initial text into the description text using the rewriting model.

[0146] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0147] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0148] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0149] Fig.11 A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0150] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0151] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0152] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the Vincent video model training method or the method for generating a video based on text. For example, in some embodiments, the Vincent video model training method or the method for generating a video based on text may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the Vincent video model training method or the method for generating a video based on text described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the text-based video model training method or the method for generating a video based on text in any other appropriate manner (eg, by means of firmware).

[0153] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0155] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0157] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0158] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0159] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0160] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a Vincent video model, comprising: Input the description text into the first video model to obtain the video to be detected; Input the video to be detected into a video detection model to obtain a detection result corresponding to the video to be detected; The first Vincent video model is trained according to the detection result to obtain a second Vincent video model.

2. The method according to claim 1, wherein: The video detection model includes a video text correlation model and a video aesthetics model. The video to be detected is input into the video detection model to obtain a detection result corresponding to the video to be detected, including: Inputting the description text and the video to be detected into the video text correlation model to obtain the correlation of the video to be detected; and / or The video to be detected is input into the video aesthetics model to obtain the aesthetics of the video to be detected.

3. The method according to claim 1 or 2, wherein: According to the detection result, the first Vincent video model is trained to obtain a second Vincent video model, including: According to a first optimization function, the parameters of the first Vincent video model are adjusted to obtain a second Vincent video model; wherein the first optimization function is determined according to the detection result, the training sample and a first probability of generating the video to be detected based on the description text.

4. The method according to claim 1 or 2, further comprising: The initial text is input into the rewriting model to obtain the description text.

5. The method according to claim 4, wherein: According to the detection result, the first Vincent video model is trained to obtain a second Vincent video model, including: According to the detection result, the first Vincent video model and the rewriting model are jointly trained to obtain a second Vincent video model and an updated rewriting model.

6. The method according to claim 5, wherein: According to the detection result, the first Vincent video model and the rewriting model are jointly trained to obtain a second Vincent video model and an updated rewriting model, including: According to the second optimization function, the parameters of the first cultural video model are adjusted to obtain the second cultural video model and the updated rewriting model; wherein the second optimization function is based on the detection results, training samples, a second probability of generating the video to be detected based on the description text, and a third probability of generating the rewritten text based on the description text.

7. The method according to any one of claims 1 to 6, wherein: The training samples include reinforcement learning samples and supervised learning samples, the reinforcement learning samples include unlabeled texts, and the supervised learning samples include labeled data of texts and videos.

8. The method according to any one of claims 1 to 7, wherein: Also includes: The video to be detected is labeled to obtain samples for updating the video detection model.

9. A method for generating a video based on text, comprising: The description text is input into the Vincent video model to obtain a video to be detected; wherein the Vincent video model is trained based on the Vincent video model training method described in any one of claims 1 to 8.

10. The method according to claim 9, further comprising: The video to be detected is input into a video detection model to obtain a detection result corresponding to the video to be detected.

11. The method according to claim 10, wherein the video detection model comprises a text-video correlation model and a video aesthetics model, and the video to be detected is input into the video detection model to obtain a detection result corresponding to the video to be detected, comprising: Inputting the description text and the video to be detected into the text-video correlation model to obtain the correlation of the video to be detected; and / or The video to be detected is input into the video aesthetics model to obtain the aesthetics of the video to be detected.

12. The method according to claim 10 or 11, further comprising: According to the detection result, the multiple videos to be detected are sorted.

13. The method according to any one of claims 9 to 12, further comprising: The initial text is rewritten into the description text using a rewriting model.

14. A training device for a Vincent video model, comprising: A text processing module, used for inputting the description text into the first text video model to obtain a video to be detected; A video processing module is used to input the video to be detected into a video detection model to obtain a detection result corresponding to the video to be detected; A training module is used to train the first Vincent video model according to the detection result to obtain a second Vincent video model.

15. The device according to claim 14, wherein: The video detection model includes a video text correlation model and a video aesthetics model, and the video processing module is used for: Inputting the description text and the video to be detected into the video text correlation model to obtain the correlation of the video to be detected; and / or The video to be detected is input into the video aesthetics model to obtain the aesthetics of the video to be detected.

16. The device according to claim 14 or 15, wherein: The training module is used to: According to a first optimization function, the parameters of the first Vincent video model are adjusted to obtain a second Vincent video model; wherein the first optimization function is determined according to the detection result, the training sample and a first probability of generating the video to be detected based on the description text.

17. The apparatus according to claim 14 or 15, further comprising: The rewriting module is used to input the initial text into the rewriting model to obtain the description text.

18. The device according to claim 17, wherein: The training module comprises: The joint training submodule is used to jointly train the first Vincent video model and the rewriting model according to the detection result to obtain a second Vincent video model and an updated rewriting model.

19. The device according to claim 18, wherein: The joint training submodule is used for: According to the second optimization function, the parameters of the first cultural video model are adjusted to obtain the second cultural video model and the updated rewriting model; wherein the second optimization function is based on the detection results, training samples, a second probability of generating the video to be detected based on the description text, and a third probability of generating the rewritten text based on the description text.

20. The device according to any one of claims 14 to 19, wherein: The training samples include reinforcement learning samples and supervised learning samples, the reinforcement learning samples include unlabeled texts, and the supervised learning samples include labeled data of texts and videos.

21. The method according to any one of claims 14 to 20, wherein: Also includes: The labeling module is used to label the video to be detected to obtain samples for updating the video detection model.

22. A device for generating a video based on text, comprising: The text processing module is used to input the description text into the Vincent video model to obtain the video to be detected; wherein, the Vincent video model is trained based on the Vincent video model training method described in any one of claims 1 to 8.

23. The apparatus according to claim 22, further comprising: The video processing module is used to input the video to be detected into the video detection model to obtain the detection result corresponding to the video to be detected.

24. The apparatus according to claim 23, wherein the video detection model comprises a text-video correlation model and a video aesthetics model, and the video processing module comprises: A correlation processing module, used for inputting the description text and the video to be detected into the text-video correlation model to obtain the correlation of the video to be detected; and / or The aesthetics processing module is used to input the video to be detected into the video aesthetics model to obtain the aesthetics of the video to be detected.

25. The apparatus according to claim 23 or 24, further comprising: A sorting module is used to sort the multiple videos to be detected according to the detection result.

26. The apparatus according to any one of claims 22 to 25, further comprising: The rewriting module is used to rewrite the initial text into the description text using the rewriting model.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

29. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images