Training method and device of model for audio and video content generation

Abstract

The invention discloses a training method and device for a model for audio and video content generation, and the method comprises the steps: inputting complete video data and complete audio data in sample data into a to-be-trained model for audio and video content generation, enabling the to-be-trained model to extract a first video feature and a first audio feature, obtaining a control factor of the sample data through the first video feature and the first audio feature, and inputting the complete video data and the first incomplete audio data of the complete audio data in the sample data into the to-be-trained model; inputting first incomplete video data and complete audio data of complete video data in the sample data into a to-be-trained model, so that the to-be-trained model fuses obtained video features and audio features with control factors of the sample data to pre-train the to-be-trained model to obtain a pre-trained model; and training of generating the content controllable model is performed based on the pre-training model. The method improves the generation capability of the model.

Description

Technical Field

[0001] The present invention relates to the field of generative artificial intelligence, and in particular, to a training method and device for a model for generating audio and video content. Background Art

[0002] With the development of generative artificial intelligence (AIGC) technology, its application potential has been explored, such as film production, advertising production, game computer animation (CG) production, commercial illustration production, short video content, music video (MV) creation, digital human generation, etc.

[0003] Existing models for audio and video content generation, such as generative models, require a large amount of sample data for training before they can be used. However, they currently face the following technical bottlenecks:

[0004] 1. Sample data is difficult to collect. For example, film production, advertising production, game CG, etc. belong to different fields. Different fields require different sample data. Collecting sample data is difficult and labor-intensive.

[0005] 2. The cost of manual labeling of sample data in each field is too high.

[0006] 3. Models in each field cannot communicate with each other, and the generation capacity of models in a single field is also very limited.

[0007] 4. The generated content is single and cannot be diversified. Because diversity will lead to the generation of risky content, we try to generate only a single scenario as much as possible.

[0008] 5. Unable to solve the zero-sample problem. Summary of the invention

[0009] The present invention provides a training method and device for a model for generating audio and video content, so as to improve the generation capability of the model for generating audio and video content.

[0010] The first aspect of the present application provides a training method for a model for generating audio and video content, the method comprising:

[0011] The complete video data and the complete audio data in the sample data are input into a to-be-trained model for generating audio and video content, so that the to-be-trained model extracts a first video feature and a first audio feature, and obtains a control factor of the sample data through the first video feature and the first audio feature, wherein the complete video data and the complete audio data are derived from the same video clip with sound, and the complete video data and the complete audio data correspond to each other,

[0012] The complete video data and the first incomplete audio data of the complete audio data in the sample data are input into the model to be trained, and the first incomplete video data and the complete audio data of the complete video data in the sample data are input into the model to be trained, so that the model to be trained fuses the acquired video features and audio features with the control factors of the sample data, so as to pre-train the model to be trained and obtain a pre-trained model.

[0013] The second incomplete video data of the complete video data in the sample data, the complete audio data, and the control factor category data of the sample data are input into the pre-trained model, and the complete video data, the second incomplete audio data of the complete audio data in the sample data, and the control factor category data of the sample data are input into the pre-trained model to perform controllable generation content training on the pre-trained model to obtain a controllable model with controllable generation content.

[0014] Preferably, the method further comprises:

[0015] The third incomplete video data of the complete video data in the sample data, the complete audio data, and the control factor category data of the sample data are input into the controllable model, and the complete video data, the third incomplete audio data of the complete audio data, and the control factor category data of the sample data are input into the controllable model, and the output result of the controllable model is judged to perform risk content training on the controllable model, and obtain a risk content controllable model as a trained model of the model to be trained; preferably, the sample data is obtained in the following manner:

[0016] Collect audio and video data in various fields,

[0017] Cut the video in the audio video data into segments according to the set segment length,

[0018] For each segment cut, an image frame is read from the segment according to the video frame rate, and the read image frame and audio data in the segment are used as sample data;

[0019] Preferably, the control factor is obtained in the following manner:

[0020] Inputting the first video feature and the first audio feature into a control factor extractor so that the control factor extractor extracts the control factor of the sample data;

[0021] The pre-trained model is trained in the following way:

[0022] The complete video data and the first incomplete audio data of the complete audio data in the sample data are input into the model to be trained, so that the model to be trained fuses the acquired second video features and the second audio features and the control factor of the sample data output by the control factor extractor, generates the first audio data, and calculates the first audio loss value between the first audio data and the complete audio data.

[0023] The first incomplete video data and the complete audio data of the complete video data in the sample data are input into the model to be trained, so that the model to be trained fuses the third video feature and the third audio feature acquired by the model and the control factor of the sample data output by the control factor extractor, generates the first video data, and calculates the first video loss value between the first video data and the complete video data.

[0024] According to the sum of the first audio loss value and the first video loss value, adjusting the model parameters of the to-be-trained model and the parameters of the control factor extractor,

[0025] Repeat this process until the model to be trained meets expectations and a pre-trained model is obtained.

[0026] Preferably, the control factor category is obtained in the following manner: the control factors of the extracted sample data are classified to obtain control factor category data of the sample data,

[0027] The controllable model is trained in the following way:

[0028] Input the second incomplete video data of the complete video data in the sample data, the complete audio data, and the control factor category data of the sample data into the pre-training model, so that the pre-training model generates the second video data, and calculates the second video loss value between the second video data and the complete video data.

[0029] The complete video data, the second incomplete audio data of the complete audio data, and the control factor category data of the sample data in the sample data are input into the pre-training model, so that the pre-training model generates the second audio data, and calculates the second audio loss value between the second audio data and the complete audio data.

[0030] According to the sum of the second audio loss value and the second video loss value, adjusting the model parameters of the pre-trained model,

[0031] Repeat this process until the pre-trained model meets expectations and a controllable model is obtained.

[0032] Preferably, the classifying the control factors of the extracted sample data includes: classifying the control factors according to a clustering algorithm;

[0033] Preferably, the risk content controllable model is trained in the following manner:

[0034] Input the third incomplete video data of the complete video data in the sample data, the complete audio data, and the control factor category data of the sample data into the controllable model, so that the controllable model generates the third video data, input the third video data into the trained discriminator, obtain the video discrimination result, and calculate the third video loss value between the video discrimination result and the target video discrimination result,

[0035] The complete video data, the third incomplete audio data of the complete audio data, and the control factor category data of the sample data in the sample data are input into the controllable model, so that the controllable model generates the third audio data, the third audio data is input into the trained discriminator to obtain the audio discrimination result, and the third audio loss value between the audio discrimination result and the target audio discrimination result is calculated.

[0036] According to the sum of the third audio loss value and the third video loss value, adjusting the model parameters of the controllable model,

[0037] Execute repeatedly until the controllable model meets expectations and obtains a controllable model of risk content.

[0038] Preferably, the method further comprises:

[0039] Build a discriminator for risky content identification,

[0040] The discriminator is trained using positive sample data and negative sample data to obtain a trained discriminator;

[0041] Among them, the positive sample data and negative sample data are obtained as follows:

[0042] For each field of sample data, determine the optional control factor category and the non-optional control factor category of the field sample data.

[0043] Inputting the fourth incomplete video data of the complete video data in the sample data, the complete audio data, and the optional control factor category of the sample data into the controllable model so that the controllable model generates video positive sample data,

[0044] Inputting the fourth incomplete video data of the complete video data in the sample data, the complete audio data, and the non-selectable control factor category of the sample data into the controllable model so that the controllable model generates video negative sample data,

[0045] Inputting the complete video data, the fourth incomplete audio data of the complete audio data, and the optional control factor category of the sample data in the sample data into the controllable model so that the controllable model generates audio positive sample data,

[0046] The complete video data, the fourth incomplete audio data of the complete audio data, and the non-selectable control factor category of the sample data in the sample data are input into the controllable model so that the controllable model generates audio negative sample data,

[0047] The unexpected positive sample data is regarded as negative sample data.

[0048] Preferably, the construction of a discriminator for risk content discrimination includes:

[0049] Constructing a video discriminator for video risk content identification and an audio discriminator for audio risk content identification respectively;

[0050] The method of training the discriminator using the positive sample data and the negative sample data to obtain the trained discriminator includes:

[0051] The video positive sample data and the video negative sample data are input into the video discriminator to train the video discriminator and obtain the trained video discriminator.

[0052] Inputting the audio positive sample data and the audio negative sample data into the audio discriminator to train the video discriminator to obtain a trained audio discriminator;

[0053] Preferably, the step of inputting the third video data into the trained discriminator comprises:

[0054] Inputting the third audio data into the trained audio discriminator to obtain an audio discrimination result,

[0055] The third video data is input into the trained video discriminator to obtain a video discrimination result.

[0056] Preferably, the incomplete audio data is obtained in the following manner:

[0057] The complete audio data is masked according to the set masking ratio, so that the masked data is replaced by empty data.

[0058] Preferably, the masking ratio of the first incomplete audio data is a first masking ratio,

[0059] Preferably, the masking ratio of the second incomplete audio data is a third masking ratio.

[0060] Preferably, the masking ratio of the third incomplete audio data is the fifth masking ratio.

[0061] Preferably, the masking ratio of the fourth incomplete audio data is the seventh masking ratio;

[0062] Preferably, the incomplete video data is obtained in the following manner:

[0063] Preferably, the complete video data is masked according to a set masking ratio so that the masked data is replaced by empty data.

[0064] Preferably, the masking ratio of the first incomplete video data is the second masking ratio,

[0065] Preferably, the masking ratio of the second incomplete video data is a fourth masking ratio.

[0066] Preferably, the masking ratio of the third incomplete video data is a sixth masking ratio.

[0067] Preferably, the masking ratio of the fourth incomplete video data is an eighth masking ratio;

[0068] Preferably, for each field sample data, determining the optional control factor category and the non-optional control factor category of the field sample data includes:

[0069] For each control factor category of the sample data in the field, count the number of sample data with this control factor category in the field, calculate the proportion of the counted number in the total number of all sample data of all control factor categories in the field, and obtain the proportion of this control factor category.

[0070] Among the proportions of various control factor categories in this field, the first M control factor categories with the largest proportions are selected as optional control factor categories.

[0071] The remaining control factor categories are regarded as non-selectable control factor categories.

[0072] Wherein, M is a set natural number.

[0073] A second aspect of the present application provides a method for generating audio and video content, the method comprising:

[0074] The video data and audio data are input into the trained model for audio and video content generation, and the model generates audio and video content output.

[0075] in,

[0076] The model for generating audio and video content is trained using any of the steps of the training method for generating audio and video content.

[0077] A third aspect of the present application provides a training device for a model for generating audio and video content, the device comprising:

[0078] a control factor extraction module, for inputting the complete video data and the complete audio data in the sample data into a to-be-trained model for generating audio and video content, so that the to-be-trained model extracts a first video feature and a first audio feature, and obtains the control factor of the sample data through the first video feature and the first audio feature, wherein the complete video data and the complete audio data are derived from the same audio video clip, and the complete video data and the complete audio data correspond to each other,

[0079] The first training module is used to input the complete video data and the first incomplete audio data of the complete audio data in the sample data into the model to be trained, and the first incomplete video data and the complete audio data of the complete video data in the sample data into the model to be trained, so that the model to be trained fuses the acquired video features and audio features with the control factors of the sample data to pre-train the model to be trained, and obtain a pre-trained model.

[0080] The second training module is used to input the second incomplete video data of the complete video data, the complete audio data, and the control factor category data of the sample data into the pre-training model, and input the complete video data, the second incomplete audio data of the complete audio data, and the control factor category data of the sample data into the pre-training model to perform controllable generation content training on the pre-training model to obtain a controllable model with controllable generation content.

[0081] A fourth aspect of the present application provides a device for generating audio and video content, the device comprising:

[0082] The audio and video content generation module is used to generate audio and video content output from the input video data and / or audio data using the trained model for audio and video content generation.

[0083] in,

[0084] The model for generating audio and video content is trained using any of the steps of the training method for generating audio and video content.

[0085] The present application provides a training method for a model for generating audio and video content. Through pre-training, the model to be trained is able to generate audio and video content based on control factors extracted from input sample data, which is conducive to forming multi-domain general capabilities and zero-sample generation capabilities. By generating controllable content training, the model to be trained is able to generate audio and video content that meets expectations, and is conducive to reducing the cost of sample data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 A flowchart of a method for training a model for generating audio and video content according to an embodiment of the present application.

[0087] Figure 2 A schematic diagram of a model structure of a model used for generating audio and video content in an embodiment of the present application.

[0088] Figure 3a-3b For this embodiment Figure 2 A schematic diagram of a process for training a model to be trained of a structure.

[0089] Figure 4 A schematic diagram of the model structure during pre-training in this embodiment.

[0090] Figure 5 A schematic diagram of the pre-training process of this embodiment.

[0091] Figure 6 A schematic diagram of the controllable fine-tuning training of this embodiment.

[0092] Figure 7 A schematic diagram of generating positive and negative samples based on a controllable model in this embodiment.

[0093] Figure 8 A schematic diagram of a discriminator for risk content discrimination in this embodiment.

[0094] Fig. 9 A schematic diagram of using the trained discriminator in this embodiment to discriminate the third video data and the third audio data output by the controllable model

[0095] Fig.10 A schematic diagram of a training device for a model for generating audio and video content according to an embodiment of the present application.

[0096] Fig.11 A schematic diagram of a device for generating audio and video content according to an embodiment of the present application.

[0097] Fig.12 Another schematic diagram of an apparatus for generating audio and video content according to an embodiment of the present application and / or another schematic diagram of an apparatus for training a model for generating audio and video content. DETAILED DESCRIPTION

[0098] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0099] See also Figure 1 As shown, Figure 1 The following is a flow chart of a training method for a model for generating audio and video content according to an embodiment of the present application. The training method includes:

[0100] Step 101: input the complete video data and the complete audio data in the sample data into a model to be trained for audio and video content generation, so that the model to be trained extracts a first video feature and a first audio feature, and obtains a control factor of the sample data through the first video feature and the first audio feature, wherein the complete video data and the complete audio data are derived from the same audio video clip, and the complete video data and the complete audio data correspond to each other.

[0101] Among them, the control factor is used to provide a basis for the generation of audio and video during the pre-training process to help the model to be trained to form generation capabilities.

[0102] Step 102: input the complete video data and the first incomplete audio data of the complete audio data in the sample data into the model to be trained, and input the first incomplete video data and the complete audio data of the complete video data in the sample data into the model to be trained, so that the model to be trained fuses the acquired video features and audio features with the control factors of the sample data to pre-train the model to be trained, and obtain a pre-trained model.

[0103] Among them, the pre-trained model can form the ability to generate audio and video content based on the control factors extracted from the input sample data, for example, input incomplete video and complete audio, or input complete video and incomplete audio, where "incomplete" refers to blank pictures or silent audio. The pre-trained model can generate different audio and video content in multiple fields.

[0104] Step 103: input the second incomplete video data of the complete video data in the sample data, the complete audio data, and the control factor category data of the sample data into the pre-trained model, input the complete video data, the second incomplete audio data of the complete audio data in the sample data, and the control factor category data of the sample data into the pre-trained model to perform controllable generation content training on the pre-trained model to obtain a controllable model that generates controllable content.

[0105] The control factor category is obtained by classifying the control factors for controlling the generated content. The controllable model can form a content-controllable audio and video content generation capability. For example, by inputting set style information, the controllable model can generate audio and video content with the set style. As an example, further, the training method also includes:

[0106] Step 104: input the third incomplete video data of the complete video data, the complete audio data, and the control factor category data of the sample data into the controllable model, input the third incomplete audio data of the complete video data, the complete audio data, and the control factor category data of the sample data into the controllable model, and judge the output result of the controllable model to perform risk content training on the controllable model, and obtain a risk content controllable model as a trained model of the model to be trained.

[0107] Among them, the risk content controllable model can form the ability to generate audio and video content with controllable risk content. For example, by inputting the set style information and the sensitive information that is not expected to be generated, the risk content controllable model can generate audio and video content with the set style, and the audio and video content does not contain sensitive information.

[0108] It should be understood that the model used for audio and video content generation in this embodiment may not be limited to the structure of the model, but may be applicable to generative models; complete video data and complete audio data refer to data that is not obscured, and incomplete video data and incomplete audio data refer to data that is partially obscured, for example, some image frames in incomplete video data are blank frames, and some frames in incomplete audio data are silent frames; risky content refers to content that is not expected to be generated, such as non-compliant content, sensitive content, etc.; sample data may be data from different fields such as various uses, various application scenarios, and various forms including image content.

[0109] The embodiments of the present application can help expand the control information of the input model and the diversity of sample data by extracting control factors of sample data. The training of pre-trained models can help form multi-domain general capabilities and zero-sample generation capabilities. The training of controllable models can reduce the sample data annotation cost and form the ability to generate controllable content. The training of risk content controllable models can solve the problem of uncontrollable risk content caused by diversity, making the trained model more robust.

[0110] To facilitate understanding of the present application, the following description is given in conjunction with a generative model. It should be understood that the embodiments of the present application are not limited to the specific structure of the model, but are also applicable to generative models of other structures.

[0111] See also Figure 2 As shown, Figure 2 This is a schematic diagram of a model structure of a model used for audio and video content generation in an embodiment of the present application. The model includes:

[0112] A video encoder is used to obtain video features of input video data.

[0113] The control encoder is used to obtain the control characteristics of the input control information, wherein the control information is used to control the effect or style of the target audio and video content to be generated. For example, the style of the target audio content may include rock, jazz, light music, etc.

[0114] An audio encoder is used to obtain the audio features of the input audio data.

[0115] The feature fuser is used to fuse the video features output by the video encoder, the control features output by the control encoder, and the audio features output by the audio encoder.

[0116] Feature encoder, used to obtain the feature encoding of the fused features output by the feature fuser,

[0117] A feature decoder is used to decode the feature code output by the feature encoder.

[0118] A video decoder, used for performing video decoding on the decoded data output by the feature decoder, and outputting video data,

[0119] The audio decoder is used to perform audio decoding on the decoded data output by the feature decoder and output audio data.

[0120] As an example, the video encoder can be the encoder part in UNET, the control information encoder is the fully connected layer (Dense), the audio encoder is the encoder part in Transformer, the feature encoder is the encoder part in Transformer, the feature decoder is the decoder part in Transformer, the video decoder is the decoder part in UNET, and the audio decoder is the decoder part in Transformer.

[0121] This model is used as the model to be trained.

[0122] Before model training, sample data for model training is collected. As an example, various audio videos are collected, such as audio videos in different fields such as film and television, animation, CG, MV, television, news broadcasts, etc., and can be classified and stored according to different fields.

[0123] The sample audio video is cut into segments according to the set segment length. For each segment, the image frames in the video segment are read according to the frame rate of the video, and the read image frames and the audio data in the segment are used as sample data. In this way, the video data and the audio data in the sample data correspond. For example, if the set segment length T is 2 seconds and the frame rate P of the video is 25 frames / second, each video is cut into 2-second segments, and each video segment corresponds to 50 image frames and 2 seconds of audio.

[0124] See also Figure 3a and 3b As shown, Figure 3a and 3b For this embodiment Figure 2 A flow chart of training a model to be trained of a structure. The training method includes:

[0125] Step 301, pre-training the model to be trained to obtain a pre-trained model,

[0126] See also Figure 4 As shown, Figure 4 A schematic diagram of the model structure during pre-training. A control factor extractor is added outside the model to be trained, and the video features output by the video encoder and the audio features output by the audio encoder in the model to be trained are respectively input into the control factor extractor. As an example, the control factor extractor adopts a recurrent neural network (LSTM).

[0127] See also Figure 5 As shown, Figure 5 A schematic diagram of the pre-training process.

[0128] Step 3011, input the complete video data in the sample data into a video encoder, input the complete audio data in the sample data into an audio encoder, input the obtained first video feature and first audio feature into a control factor extractor, and the control factor extractor outputs the control factor of the sample data, wherein the control factor is a vector.

[0129] Step 3012, masking the complete audio data in the sample data according to the set first masking ratio to obtain first incomplete audio data of the complete audio data, wherein the masked part of the first incomplete audio data is silent; inputting the complete video data into a video encoder, and inputting the first incomplete audio data into an audio encoder,

[0130] As an example, the first masking ratio is set in a range of 10% to 80%. For example, the last 1 second of content is replaced with an empty file to obtain the first incomplete audio.

[0131] The second audio feature output by the audio encoder, the second video feature output by the video encoder, and the control factor obtained in step 3011 are input to the feature fusion device. The fusion feature output by the feature fusion device is processed by the feature encoder and the feature decoder and then input to the audio decoder and the video decoder respectively.

[0132] Calculating a loss function value between the first audio data output by the audio decoder and the complete audio data to obtain a first audio loss value,

[0133] Step 3013, masking the complete video data in the sample data according to the set second masking ratio to obtain first incomplete video data of the complete video data, wherein the masked portion of the first incomplete video data is a blank frame; inputting the complete audio data into an audio encoder, and inputting the first incomplete video data into a video encoder,

[0134] As an example, the second masking ratio is set in a range of 10% to 80%. For example, the last 1 second of content is replaced with an empty file to obtain an incomplete video.

[0135] The third audio feature output by the audio encoder, the third video feature output by the video encoder, and the control factor obtained in step 3011 are input to the feature fusion device. The fusion feature output by the feature fusion device is processed by the feature encoder and the feature decoder and then input to the audio decoder and the video decoder respectively.

[0136] A loss function value between the first video data output by the video decoder and the complete video data is calculated to obtain a first video loss value.

[0137] It should be understood that there is no strict order between steps 3012 and 3013 and they can be executed in parallel.

[0138] Step 3014: adjust the parameters of all modules in the model according to the sum of the first audio loss value and the first video loss value.

[0139] Repeat steps 3011 to 3014 until the expected result is achieved and a pre-trained model is obtained.

[0140] Since the audio and video content generated by the pre-trained model has not yet reached a controllable level for generating the target audio and video content, it can still be trained in subsequent steps.

[0141] Step 302: extract the control factors of each sample data to obtain a number of control factors, and classify each control factor.

[0142] As an example, clustering algorithms such as K-means are used to classify the control factors into N categories. In this way, the category of each control factor is obtained, and the category of the control factor of each sample data is also obtained.

[0143] Step 303 , in order to obtain a controllable model with controllable generated content, the pre-trained model is subjected to controllable fine-tuning training.

[0144] See also Figure 6 As shown, Figure 6 This is a schematic diagram of controllable fine-tuning training. It can be seen that it is similar to the training of the pre-trained model, except that the complete video and complete audio already have control factor category labels.

[0145] Step 3031, mask the complete audio data in the sample data according to the third masking ratio to obtain the second incomplete audio data of the complete audio data, and send the complete video data, the second incomplete audio data and the control factor category of the sample data to the video encoder, audio encoder and control information encoder in the pre-trained model respectively, and send the obtained video features, audio features and control features to the feature encoder. After being processed by the feature encoder and feature decoder in sequence, the output of the feature decoder is input to the audio decoder and the video decoder respectively.

[0146] Calculate the loss function value between the second audio data output by the audio decoder and the complete audio data to obtain a second audio loss value. Step 3032: Mask the complete video data in the sample data according to the fourth masking ratio to obtain second incomplete video data of the complete video data, and send the second incomplete video data, the complete audio data, and the control factor category of the sample data to the video encoder, audio encoder, and control information encoder in the pre-trained model respectively, and send the obtained video features, audio features, and control features to the feature encoder together. After being processed by the feature encoder and feature decoder in sequence, the output of the feature decoder is input to the audio decoder and the video decoder respectively.

[0147] A loss function value between the second video data output by the video decoder and the complete video data is calculated to obtain a second video loss value.

[0148] It should be understood that there is no strict order between steps 3031 and 3032 and they can be executed in parallel.

[0149] Step 3033: adjust the parameters of the control information encoder according to the sum of the second audio loss value and the second video loss value.

[0150] Repeat steps 3031 to 3033 until the desired result is achieved and a controllable model is obtained. Step 304: Since the sample data involves various fields, all control factor categories of the sample data in each field can be obtained, as shown in Table 1.

[0151] For each control factor category of sample data in each field, the proportion of the number of all sample data with this control factor category in the field to the total number of all sample data of all control factor categories in the field is counted, thereby obtaining the proportion of each control factor category in the field.

[0152] Among the proportions of each control factor category in this field, the first M control factor categories with the largest proportion are taken to represent the optional control factor categories in this field, and the remaining V control factor categories are used to represent the unselectable control factor categories in this field, where V = NM, and N is the total number of all control factor categories in this field. In this way, the optional control factor categories and unselectable control factor categories of the sample data are obtained.

[0153] Table 1

[0154]

[0155] Step 305: Input the sample data, optional control factor categories, and non-optional control factor categories of each field into the controllable model to generate positive and negative samples.

[0156] See also Figure 7 As shown, Figure 7 A schematic diagram of generating positive and negative samples based on a controllable model. For any sample data in any field,

[0157] The optional control factor category of the sample data, the fourth incomplete video data in the sample data, and the complete audio data are input into the controllable model, and the video data output by the controllable model is used as a video positive sample,

[0158] The non-selectable control factor of the sample data, the fourth incomplete video data in the sample data, and the complete audio data are input into the controllable model, and the video data output by the controllable model is used as a video negative sample,

[0159] The optional control factor of the sample data, the fourth incomplete audio data in the sample data, and the complete video data are input into the controllable model, and the audio data output by the controllable model is used as an audio positive sample.

[0160] The non-selectable control factor of the sample data, the fourth incomplete audio data in the sample data, and the complete video data are input into the controllable model, and the audio data output by the controllable model is used as an audio negative sample.

[0161] Furthermore, samples that are not expected to be generated in the positive samples are taken as negative samples, for example, samples with risky content.

[0162] In this way, positive and negative samples can be obtained in each field.

[0163] Among them, the masking ratio of the fourth incomplete audio data is the seventh masking ratio, and the masking ratio of the fourth incomplete video data is the eighth masking ratio.

[0164] Step 306: construct and train a discriminator for risky content identification.

[0165] See also Figure 8 As shown, Figure 8 : is a schematic diagram of a discriminator for risk content discrimination. The discriminator includes a video discriminator for video risk content discrimination and an audio discriminator for audio risk content discrimination. As an example, the video discriminator and the audio discriminator can both be implemented using Resnet50.

[0166] The positive sample and negative sample data obtained in step 305 are used as training data and input into the discriminator for training, so that the discriminator forms a discriminant function. As an example, the discriminant result is reflected in a discriminant value. For example, the positive sample is scored from 0 to 100. The score of the negative sample is 0.

[0167] Specifically, the video positive samples and the video negative samples are input into the video discriminator, and the audio positive samples and the audio negative samples are input into the audio discriminator, and the video discriminator and the audio discriminator are trained respectively.

[0168] It should be understood that risky content refers to content that is non-compliant or sensitive.

[0169] Step 307, using the trained discriminator, discriminate the third video data and the third audio data output by the controllable model, and adjust the model parameters of the controllable model according to the loss value between the discrimination result and the target discrimination result to perform risk content control training and improve the generation quality of the model.

[0170] See also Fig. 9 As shown, Fig. 9 A schematic diagram of using a trained discriminator to discriminate the third video data and the third audio data output by the controllable model. The outputs of the video decoder and the audio decoder in the controllable model are respectively input to the video discriminator and the audio discriminator.

[0171] Step 3071: mask the complete audio in the sample data according to the fifth masking ratio to obtain a third incomplete audio; send the complete video data, the third incomplete audio data, and the control factor category of the sample data to the video encoder, the audio encoder, and the control information encoder in the controllable model respectively; send the obtained video features, audio features, and control features to the feature encoder; after being processed by the feature encoder and the feature decoder in sequence, the output of the feature decoder is input to the audio decoder and the video decoder respectively; the third audio data output by the audio decoder is input to the audio discriminator;

[0172] The loss function value between the audio discrimination result output by the audio discriminator and the target audio discrimination result is calculated to obtain a third audio loss value. For example, the obtained score is compared with the full score to calculate the loss value.

[0173] Step 3072: mask the complete video of the sample data according to the sixth masking ratio to obtain third incomplete video data; send the third incomplete video data, the complete audio data, and the control factor category of the sample data to the video encoder, the audio encoder, and the control information encoder in the controllable model respectively; send the obtained video features, audio features, and control features to the feature encoder together; after being processed by the feature encoder and the feature decoder in sequence, the outputs of the feature decoder are respectively input to the audio decoder and the video decoder; the third video data output by the video decoder is input to the video discriminator;

[0174] A loss function value between the video discrimination result output by the video discriminator and the target video discrimination result is calculated to obtain a third video loss value.

[0175] Step 3073: adjust the parameters of the control information encoder according to the sum of the third audio loss value and the third video loss value.

[0176] Steps 3071 to 3073 are repeatedly performed until the expected result is achieved, thereby obtaining a trained risk content controllable model.

[0177] It should be understood that there is no strict order between steps 3071 and 3072 and they can be executed in parallel.

[0178] The training method of the model for generating audio and video content in this embodiment reduces the difficulty of collecting sample data by pre-training the model to be trained, so that the model to be trained can form multi-domain general capabilities and zero-sample generation capabilities. The labeling cost of sample data is reduced by performing controllable fine-tuning training on the pre-trained model. The problem of uncontrollable risk content caused by diversity is solved by performing risk content control training on the controllable model.

[0179] Based on the trained model, audio and video content can be generated.

[0180] For example, in a short video creation application, the user can input a video into a trained model, select the field that requires music, and the trained model will generate music for the corresponding field based on the video content; or, the user can input a video and an audio with special effects into the trained model, and the trained model will output a video with special effects added.

[0181] For example, in a digital human generation application, the user inputs a photo and a voice broadcasting content into a trained model, and the model outputs a video of the person in the photo broadcasting the content.

[0182] For example, in a movie stunt production application, the user provides a pre-stunt video, a pre-stunt music, and a stunt mid-soundtrack to the trained model, and the model outputs the corresponding stunt mid-soundtrack based on the music.

[0183] See also Fig.10 As shown, Fig.10 A schematic diagram of a training device for a model for generating audio and video content according to an embodiment of the present application. The device includes:

[0184] a control factor extraction module, used for inputting the complete video data and the complete audio data in the sample data into a model to be trained for audio and video content generation, so that the model to be trained extracts the first video feature and the first audio feature, and obtains the control factor of the sample data through the first video feature and the first audio feature, wherein the complete video data and the complete audio data are derived from the same audio video clip, and the complete video data corresponds to the complete audio data,

[0185] The first training module is used to input the complete video data and the first incomplete audio data of the complete audio data in the sample data into the model to be trained, and the first incomplete video data and the complete audio data of the complete video data in the sample data into the model to be trained, so that the model to be trained fuses the acquired video features and audio features with the control factors of the sample data to pre-train the model to be trained, and obtain a pre-trained model.

[0186] The second training module is used to input the second incomplete video data of the complete video data, the complete audio data, and the control factor category data of the sample data into the pre-training model, and input the complete video data, the second incomplete audio data of the complete audio data, and the control factor category data of the sample data into the pre-training model to perform controllable generation content training on the pre-training model to obtain a controllable model with controllable generation content.

[0187] As an example, the device further includes:

[0188] The third training module is used to input the third incomplete video data of the complete video data, the complete audio data, and the control factor category data of the sample data into the controllable model, and input the third incomplete audio data of the complete video data, the complete audio data, and the control factor category data of the sample data into the controllable model, and judge the output result of the controllable model to perform risk content training on the controllable model to obtain a risk content controllable model as a trained model of the model to be trained.

[0189] See also Fig.11 As shown, Fig.11 A schematic diagram of a device for generating audio and video content according to an embodiment of the present application.

[0190] The device includes:

[0191] The audio and video content generation module is used to generate audio and video content output from the input video data and / or audio data using the trained model for audio and video content generation.

[0192] in,

[0193] The model for generating audio and video content is trained using the steps of the training method for generating audio and video content in the embodiment of the present application.

[0194] See also Fig.12 As shown, Fig.12 Another schematic diagram of an apparatus for generating audio and video content and / or another schematic diagram of an apparatus for training a model for generating audio and video content according to an embodiment of the present application. The apparatus includes: a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to perform the steps of generating audio and video content and / or the steps of a training method for a model for generating audio and video content.

[0195] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0196] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0197] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of generating audio and video content and / or the steps of a training method for a model for generating audio and video content.

[0198] As for the apparatus / network-side device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0199] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0200] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A training method for a model for audio and video content generation, characterized in that: The method includes: The complete video data and the complete audio data in the sample data are input into a to-be-trained model for generating audio and video content, so that the to-be-trained model extracts a first video feature and a first audio feature, and obtains a control factor of the sample data through the first video feature and the first audio feature, wherein the complete video data and the complete audio data are derived from the same video clip with sound, and the complete video data and the complete audio data correspond to each other, The complete video data and the first incomplete audio data of the complete audio data in the sample data are input into the model to be trained, and the first incomplete video data and the complete audio data of the complete video data in the sample data are input into the model to be trained, so that the model to be trained fuses the acquired video features and audio features with the control factors of the sample data, so as to pre-train the model to be trained and obtain a pre-trained model. The second incomplete video data of the complete video data in the sample data, the complete audio data, and the control factor category data of the sample data are input into the pre-trained model, and the complete video data, the second incomplete audio data of the complete audio data in the sample data, and the control factor category data of the sample data are input into the pre-trained model to perform controllable generation content training on the pre-trained model to obtain a controllable model with controllable generation content.

2. The training method according to claim 1, characterized in that: The method further comprises: The third incomplete video data of the complete video data, the complete audio data, and the control factor category data of the sample data in the sample data are input into the controllable model, and the complete video data, the third incomplete audio data of the complete audio data, and the control factor category data of the sample data are input into the controllable model, and the output result of the controllable model is judged to train the controllable model for risk content and obtain a risk content controllable model as a trained model of the model to be trained.

3. The training method according to claim 1, characterized in that The sample data is obtained in the following manner: Collect audio and video data in various fields, Cut the video in the audio video data into segments according to the set segment length, For each segment cut, an image frame is read from the segment according to the video frame rate, and the read image frame and audio data in the segment are used as sample data; The control factor is obtained as follows: Inputting the first video feature and the first audio feature into a control factor extractor so that the control factor extractor extracts the control factor of the sample data; The pre-trained model is trained in the following way: The complete video data and the first incomplete audio data of the complete audio data in the sample data are input into the model to be trained, so that the model to be trained fuses the acquired second video features and the second audio features and the control factor of the sample data output by the control factor extractor, generates the first audio data, and calculates the first audio loss value between the first audio data and the complete audio data. The first incomplete video data and the complete audio data of the complete video data in the sample data are input into the model to be trained, so that the model to be trained fuses the third video feature and the third audio feature acquired by the model and the control factor of the sample data output by the control factor extractor, generates the first video data, and calculates the first video loss value between the first video data and the complete video data. According to the sum of the first audio loss value and the first video loss value, adjusting the model parameters of the to-be-trained model and the parameters of the control factor extractor, Repeat this process until the model to be trained meets expectations and a pre-trained model is obtained.

4. The training method according to claim 2, characterized in that: The control factor category is obtained in the following manner: the control factors of the extracted sample data are classified to obtain the control factor category data of the sample data, The controllable model is trained in the following way: Input the second incomplete video data of the complete video data in the sample data, the complete audio data, and the control factor category data of the sample data into the pre-training model, so that the pre-training model generates the second video data, and calculates the second video loss value between the second video data and the complete video data. The complete video data, the second incomplete audio data of the complete audio data, and the control factor category data of the sample data in the sample data are input into the pre-training model, so that the pre-training model generates the second audio data, and calculates the second audio loss value between the second audio data and the complete audio data. According to the sum of the second audio loss value and the second video loss value, adjusting the model parameters of the pre-trained model, Repeat this process until the pre-trained model meets expectations and a controllable model is obtained.

5. The training method according to claim 4, characterized in that: The classifying the control factors of the extracted sample data includes: classifying the control factors according to a clustering algorithm; The risk content controllable model is trained in the following way: Input the third incomplete video data of the complete video data in the sample data, the complete audio data, and the control factor category data of the sample data into the controllable model, so that the controllable model generates the third video data, input the third video data into the trained discriminator, obtain the video discrimination result, and calculate the third video loss value between the video discrimination result and the target video discrimination result, The complete video data, the third incomplete audio data of the complete audio data, and the control factor category data of the sample data in the sample data are input into the controllable model, so that the controllable model generates the third audio data, the third audio data is input into the trained discriminator to obtain the audio discrimination result, and the third audio loss value between the audio discrimination result and the target audio discrimination result is calculated. According to the sum of the third audio loss value and the third video loss value, adjusting the model parameters of the controllable model, Execute repeatedly until the controllable model meets expectations and obtains a controllable model of risk content.

6. The training method according to claim 5, characterized in that: The method further comprises: Build a discriminator for risky content identification, The discriminator is trained using positive sample data and negative sample data to obtain a trained discriminator; Among them, the positive sample data and negative sample data are obtained as follows: For each field of sample data, determine the optional control factor category and the non-optional control factor category of the field sample data. Inputting the fourth incomplete video data of the complete video data in the sample data, the complete audio data, and the optional control factor category of the sample data into the controllable model so that the controllable model generates video positive sample data, Inputting the fourth incomplete video data of the complete video data in the sample data, the complete audio data, and the non-selectable control factor category of the sample data into the controllable model so that the controllable model generates video negative sample data, Inputting the complete video data, the fourth incomplete audio data of the complete audio data, and the optional control factor category of the sample data in the sample data into the controllable model so that the controllable model generates audio positive sample data, The complete video data, the fourth incomplete audio data of the complete audio data, and the non-selectable control factor category of the sample data in the sample data are input into the controllable model so that the controllable model generates audio negative sample data, The unexpected positive sample data is regarded as negative sample data.

7. The training method according to claim 6, characterized in that: The construction of a discriminator for risk content discrimination includes: Constructing a video discriminator for video risk content identification and an audio discriminator for audio risk content identification respectively; The method of training the discriminator using the positive sample data and the negative sample data to obtain the trained discriminator includes: The video positive sample data and the video negative sample data are input into the video discriminator to train the video discriminator and obtain the trained video discriminator. Inputting the audio positive sample data and the audio negative sample data into the audio discriminator to train the video discriminator to obtain a trained audio discriminator; The step of inputting the third video data into the trained discriminator comprises: Inputting the third audio data into the trained audio discriminator to obtain an audio discrimination result, The third video data is input into the trained video discriminator to obtain a video discrimination result.

8. The training method according to claim 6, characterized in that: The incomplete audio data is obtained in the following manner: The complete audio data is masked according to the set masking ratio, so that the masked data is replaced by empty data. The masking ratio of the first incomplete audio data is a first masking ratio, The masking ratio of the second incomplete audio data is a third masking ratio, The masking ratio of the third incomplete audio data is the fifth masking ratio, The masking ratio of the fourth incomplete audio data is a seventh masking ratio; The incomplete video data is obtained in the following manner: The complete video data is masked according to the set masking ratio, so that the masked data is replaced by empty data. The masking ratio of the first incomplete video data is a second masking ratio, The masking ratio of the second incomplete video data is a fourth masking ratio, The masking ratio of the third incomplete video data is a sixth masking ratio, The masking ratio of the fourth incomplete video data is an eighth masking ratio; For each field sample data, determining the optional control factor category and the non-optional control factor category of the field sample data includes: For each control factor category of the sample data in the field, count the number of sample data with this control factor category in the field, calculate the proportion of the counted number in the total number of all sample data of all control factor categories in the field, and obtain the proportion of this control factor category. Among the proportions of various control factor categories in this field, the first M control factor categories with the largest proportions are selected as optional control factor categories. The remaining control factor categories are regarded as non-selectable control factor categories. Wherein, M is a set natural number.

9. A method for generating audio and video content, characterized in that: The method includes: The video data and audio data are input into the trained model for audio and video content generation, and the model generates audio and video content output. in, The model for generating audio and video content is trained using the steps of the training method for generating audio and video content described in any one of claims 1 to 8.

10. A training device for a model for generating audio and video content, characterized in that: The device includes: a control factor extraction module, for inputting the complete video data and the complete audio data in the sample data into a to-be-trained model for generating audio and video content, so that the to-be-trained model extracts a first video feature and a first audio feature, and obtains the control factor of the sample data through the first video feature and the first audio feature, wherein the complete video data and the complete audio data are derived from the same audio video clip, and the complete video data and the complete audio data correspond to each other, The first training module is used to input the complete video data and the first incomplete audio data of the complete audio data in the sample data into the model to be trained, and the first incomplete video data and the complete audio data of the complete video data in the sample data into the model to be trained, so that the model to be trained fuses the acquired video features and audio features with the control factors of the sample data to pre-train the model to be trained, and obtain a pre-trained model. The second training module is used to input the second incomplete video data of the complete video data, the complete audio data, and the control factor category data of the sample data into the pre-training model, and input the complete video data, the second incomplete audio data of the complete audio data, and the control factor category data of the sample data into the pre-training model to perform controllable generation content training on the pre-training model to obtain a controllable model with controllable generation content.

11. A device for generating audio and video content, characterized in that: The device includes: The audio and video content generation module is used to generate audio and video content output from the input video data and / or audio data using the trained model for audio and video content generation. in, The model for generating audio and video content is trained using the steps of the training method for generating audio and video content described in any one of claims 1 to 8.

Images