Training evaluation model, method and device for evaluating video quality

By training and evaluation models, using encoder to separate the video audio and video, reducing the distance between video frames, solving the accuracy and generalization of video evaluation in the prior art, and realizing multi-dimensional evaluation of virtual image videos.

CN117012228BActive Publication Date: 2024-11-01ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310943184.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-07-28
Publication Date
2024-11-01
Estimated Expiration
2043-07-28

AI Technical Summary

Technical Problem

The prior art is difficult to evaluate video quality objectively and accurately, especially when automated production of live videos of virtual avatar content. Traditional methods are time-consuming and lack generalization, and cannot adapt to the evaluation of virtual avatar videos.

Method used

By training and evaluation model, the video samples are separated by audio and video using the first encoder and the second encoder, reducing the distance between different modes between image frames and audio frames and the homomodal between adjacent image frames Distance, achieve frame-level audio and video consistency and coherence evaluation of video.

Benefits of technology

It realizes multi-dimensional objective and accurate evaluation of video quality, and is suitable for virtual image videos, improving the accuracy and generalization of evaluation and reducing the evaluation cost.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117012228B_ABST
    Figure CN117012228B_ABST
Patent Text Reader

Abstract

The embodiments of this specification provide a method and device for training an evaluation model and evaluating video quality. In the method for training an evaluation model, a sample set is obtained, which includes a first video sample with synchronized audio and video. The first video sample is separated from the audio and video to obtain a sample image frame sequence and a corresponding sample audio frame sequence, and the sample image frame sequence and the corresponding sample audio frame sequence are input into the evaluation model for model processing. The evaluation model includes a first encoder and a second encoder. The above-mentioned model processing includes encoding each sample image frame using the first encoder to obtain each sample image representation, and encoding each sample audio frame using the second encoder to obtain each sample audio representation. The parameters of the first encoder and the second encoder are adjusted with the goal of minimizing the first loss. The first loss is positively correlated to the distance between different modes of each sample image representation and the corresponding sample audio representation, and the same-modal distance between adjacent sample image representations in each sample image representation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of machine learning, and more particularly, to a method and apparatus for training an evaluation model and evaluating video quality. Background Art

[0002] With the development of the entertainment industry and the rise of the concept of the metaverse, the demand for automated content production has become more urgent. For example, in live broadcast scenarios, it is hoped that content explanation videos or live broadcast videos of virtual images (e.g., virtual digital people) can be automatically produced. Generally speaking, compared with text, synthetic videos can automatically complete the explanation for users, which helps to improve the vividness of the content and enhance the user experience.

[0003] For the videos produced above, their quality usually needs to be evaluated. Summary of the invention

[0004] One or more embodiments of this specification describe a method and device for training an evaluation model and evaluating video quality, which can objectively and accurately evaluate the quality of a video.

[0005] According to a first aspect, a method for training an evaluation model is provided, comprising:

[0006] Acquire a sample set, including a first video sample with synchronized audio and video;

[0007] Performing audio and video separation on the first video sample to obtain a sample image frame sequence and a corresponding sample audio frame sequence, and inputting the sample image frame sequence and the corresponding sample audio frame sequence into the evaluation model for model processing, wherein the evaluation model includes a first encoder and a second encoder; the model processing includes encoding each sample image frame in the sample image frame sequence using the first encoder to obtain each sample image representation, and encoding each sample audio frame in the sample audio frame sequence using the second encoder to obtain each sample audio representation;

[0008] The parameters of the first encoder and the second encoder are adjusted with the goal of minimizing a first loss; the first loss is positively correlated to the distances between different modes of each sample image representation of the first video sample and the corresponding sample audio representation, and the same-mode distances between adjacent sample image representations in each sample image representation of the first video sample.

[0009] According to a second aspect, there is provided a method for evaluating video quality, comprising:

[0010] Obtain the target video to be evaluated;

[0011] Performing audio and video separation on the target video to obtain a target image frame sequence and a corresponding target audio frame sequence, and inputting the target image frame sequence and the corresponding target audio frame sequence into an evaluation model for model processing, wherein the evaluation model includes a first encoder and a second encoder; the model processing includes encoding each target image frame in the target image frame sequence using the first encoder to obtain each target image representation, and encoding each target audio frame in the target audio frame sequence using the second encoder to obtain each target audio representation;

[0012] Determining a first evaluation index based on different-modality distances between each target image representation and a corresponding target audio representation, and determining a second evaluation index based on same-modality distances between adjacent target image representations in each target image representation;

[0013] The quality of the target video is evaluated based at least on the first evaluation index and the second evaluation index.

[0014] According to a third aspect, a method for training a driving model is provided, comprising:

[0015] Acquire an initial sample set for training the driving model, wherein a single sample includes image description data of a person, an audio clip, and a sample label; the sample label is used to indicate a real image frame sequence obtained by driving the image description data of the person based on the audio clip;

[0016] For any first sample, evaluating a first synthetic video obtained based on the real image frame sequence and the audio clip according to the method of the second aspect;

[0017] If the evaluation result indicates that the audio and video of the first composite video are not synchronized, adding a weight value lower than a predetermined threshold to the first sample; the weight value is used to indicate that the first sample is a dirty sample containing noise;

[0018] The driving model is trained based on at least the first sample to which the weight value is added.

[0019] According to a fourth aspect, a method for training a driving model is provided, comprising:

[0020] Obtaining a clean sample set for training the driving model, wherein a single sample includes image description data of a person and an audio clip;

[0021] For any first sample, input the image description data and audio clip of the character therein into the driving model to obtain a predicted image frame sequence;

[0022] According to the method of the second aspect, a second synthetic video obtained based on the predicted image frame sequence and the audio clip is evaluated;

[0023] If the evaluation result indicates that the audio and video of the second composite video are not synchronized, a corresponding penalty value is added to the first sample; the penalty value is used to indicate that the first sample is an important sample;

[0024] The driving model is trained based on at least the first sample to which the penalty value is added.

[0025] According to a fifth aspect, there is provided a device for training an evaluation model, comprising:

[0026] An acquisition unit, configured to acquire a sample set, including a first video sample with synchronized audio and video;

[0027] a processing unit, configured to perform audio and video separation on the first video sample, obtain a sample image frame sequence and a corresponding sample audio frame sequence, and input the sample image frame sequence and the corresponding sample audio frame sequence into the evaluation model for model processing, wherein the evaluation model includes a first encoder and a second encoder;

[0028] The processing unit comprises:

[0029] A first encoding submodule, configured to encode each sample image frame in the sample image frame sequence using the first encoder to obtain each sample image representation;

[0030] A second encoding submodule, configured to encode each sample audio frame in the sample audio frame sequence using the second encoder to obtain each sample audio representation;

[0031] An adjustment unit is used to adjust parameters of the first encoder and the second encoder with the goal of minimizing a first loss; the first loss is positively correlated to the distances between different modes of each sample image representation of the first video sample and the corresponding sample audio representation, and the same-mode distances between adjacent sample image representations in each sample image representation of the first video sample.

[0032] According to a sixth aspect, there is provided a device for evaluating video quality, comprising:

[0033] An acquisition unit, used for acquiring a target video to be evaluated;

[0034] A processing unit, configured to perform audio and video separation on the target video, obtain a target image frame sequence and a corresponding target audio frame sequence, and input the target image frame sequence and the target audio frame sequence into an evaluation model for model processing, wherein the evaluation model includes a first encoder and a second encoder;

[0035] The processing unit comprises:

[0036] A first encoding submodule, configured to encode each target image frame in the target image frame sequence using the first encoder to obtain each target image representation;

[0037] A second encoding submodule, configured to encode each target audio frame in the target audio frame sequence using the second encoder to obtain each target audio representation;

[0038] a determining unit, configured to determine a first evaluation index based on the different-modality distances between each target image representation and the corresponding target audio representation, and to determine a second evaluation index based on the same-modality distances between adjacent target image representations in each target image representation;

[0039] An evaluation unit is used to evaluate the quality of the target video based at least on the first evaluation index and the second evaluation index.

[0040] According to a seventh aspect, there is provided a device for training a driving model, comprising:

[0041] An acquisition unit is used to acquire an initial sample set for training the driving model, wherein a single sample includes image description data of a person, an audio clip and a sample label; the sample label is used to indicate a real image frame sequence obtained by driving the image description data of the person based on the audio clip;

[0042] An evaluation unit, configured to evaluate, for any first sample, a first synthetic video obtained based on the real image frame sequence and the audio segment according to the method of the second aspect;

[0043] an adding unit, configured to add a weight value lower than a predetermined threshold to the first sample if the evaluation result indicates that the audio and video of the first composite video are not synchronized; the weight value is used to indicate that the first sample is a dirty sample containing noise;

[0044] A training unit is used to train the driving model based on at least the first sample to which the weight value is added.

[0045] According to an eighth aspect, there is provided a device for training a driving model, comprising:

[0046] An acquisition unit, used to acquire a clean sample set for training the driving model, wherein a single sample includes image description data of a person and an audio clip;

[0047] An input unit, for inputting the image description data and the audio clip of the character in any first sample into the driving model to obtain a predicted image frame sequence;

[0048] an evaluation unit, configured to evaluate a second synthesized video obtained based on the predicted image frame sequence and the audio segment according to the method of the second aspect;

[0049] an adding unit, configured to add a corresponding penalty value to the first sample if the evaluation result indicates that the audio and video of the second composite video are not synchronized; the penalty value is used to indicate that the first sample is an important sample;

[0050] A training unit is used to train the driving model based on at least the first sample to which the penalty value is added.

[0051] According to a ninth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect or the second aspect.

[0052] According to the tenth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect or the second aspect is implemented.

[0053] The training evaluation model, method and device for evaluating video quality provided by one or more embodiments of this specification can, when training the evaluation model, reduce the inter-modal distance between the sample image representation and the corresponding sample audio representation, so that the trained evaluation model can evaluate the consistency of audio and video at the frame level of the video. In addition, by reducing the same-modal distance between adjacent sample image representations, the present solution can enable the trained evaluation model to evaluate the coherence of the video. In summary, the present solution can objectively and accurately evaluate the quality of the video from multiple dimensions. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;

[0056] Figure 2 A flowchart of a method for training an evaluation model according to one embodiment is shown;

[0057] Figure 3 A schematic diagram showing a method for evaluating video quality according to an embodiment is shown;

[0058] Figure 4 A schematic diagram showing a method of training a driving model in one example is shown;

[0059] Figure 5A schematic diagram of a method for training a driving model in another example is shown;

[0060] Figure 6 A schematic diagram of a device for training an evaluation model according to an embodiment is shown;

[0061] Figure 7 A schematic diagram of a device for evaluating video quality according to an embodiment is shown;

[0062] Figure 8 A schematic diagram of a device for training a driving model according to an embodiment is shown;

[0063] Fig. 9 A schematic diagram of a device for training a driving model according to another embodiment is shown. DETAILED DESCRIPTION

[0064] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0065] As mentioned above, the quality of the produced video needs to be evaluated. In traditional technology, there are mainly the following evaluation methods:

[0066] The first type is manual evaluation, which involves manually scoring the synthesized video from the following aspects: whether the audio and video are synchronized, the accuracy of lip movements, the smoothness of the picture, the degree of semantic matching, the degree of emotional expression, etc. There is no objective standard for these, and different people may give very different scores, and it is relatively time-consuming.

[0067] The second method is to evaluate based on a fixed test set, that is, to calculate the error between the synthetic video and the real video frame by frame. It has the following two main disadvantages: First, this method strictly requires that every pixel of the synthetic video must be as identical as possible to the real video, but this requirement is one-sided, because a person can say the same sentence in many ways, and the rhythm, expression, and movement may change. There is no single correct label, as long as there is no sense of violation as a whole, and this method cannot achieve high generalization. Second, this test set requires videos recorded by real people, and cartoon characters themselves do not have such data.

[0068] To this end, the present application proposes to evaluate the video quality by training an evaluation model, wherein the evaluation model specifically evaluates the quality of the synthesized video by feature matching. The training process of the evaluation model is described below.

[0069] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification. Figure 1 In the embodiment, the evaluation model includes a first encoder and a second encoder. The evaluation model is used to evaluate the video quality.

[0070] For the above evaluation model, training can be performed through the following process:

[0071] For the first video sample with synchronized audio and video in the sample set, the audio and video can be separated, and the obtained sample image frame sequence and sample audio frame sequence can be input into the evaluation model, so that the sample image frame sequence and the sample audio frame sequence are respectively encoded by the first encoder and the second encoder in the evaluation model to obtain each sample image representation and each sample audio representation.

[0072] Afterwards, the evaluation model is trained by reducing the inter-modal distance between the sample image representation and the corresponding sample audio representation, and reducing the same-modal distance between adjacent sample image representations. This allows the trained evaluation model to evaluate both the frame-level audio-visual consistency of the video and the coherence of the video. In other words, the video quality can be objectively and accurately evaluated from multiple dimensions.

[0073] It should be understood that Figure 1 This is just an exemplary description. In actual applications, the evaluation model can also be trained by increasing the distance between different modes of each sample image representation and the corresponding sample audio representation of the second video sample with asynchronous audio and video, etc. This manual does not limit this.

[0074] Figure 2 A flow chart of a method for training an evaluation model according to an embodiment is shown. The method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 2 As shown, the method may include the following steps:

[0075] Step S202: Acquire a sample set, which includes a first video sample with synchronized audio and video.

[0076] In one example, the first video sample with synchronized audio and video is a real-person video with synchronized audio and video. The synchronization of audio and video here may mean that the sound is consistent with the picture.

[0077] In one embodiment, the sample set may further include a second video sample whose audio and video are not synchronized.

[0078] It should be understood that, in the case where the sample set further includes a second video sample, the first video sample is used as a positive sample, and the second video sample is used as a negative sample.

[0079] Step S204, performing audio and video separation on the first video sample to obtain a sample image frame sequence and a corresponding sample audio frame sequence, and inputting them into the evaluation model for model processing.

[0080] Each sample image frame in the sample image frame sequence corresponds to each sample audio frame in the sample audio frame sequence one by one.

[0081] The above evaluation model may include a first encoder and a second encoder. The first and second encoders here may be implemented as Transformer encoders, that is, they may encode input data based on a self-attention mechanism.

[0082] The above-mentioned model processing may include using a first encoder to encode each sample image frame in a sample image frame sequence to obtain each sample image representation, and using a second encoder to encode each sample audio frame in a sample audio frame sequence to obtain each sample audio representation.

[0083] It should be understood that the encoding methods of the sample image frame and the sample audio frame are similar, so the encoding process is described below by taking the sample image frame as an example.

[0084] Specifically, the first encoder may include an embedding layer and an attention layer, and the encoding of each sample image frame in the sample image frame sequence may include:

[0085] Determine, using the embedding layer, an initial representation of each sample image frame based on the embedding vector and the sequence number of each sample image frame;

[0086] Using the attention layer, different weights are given to the initial representations of each sample image frame to obtain the corresponding representations of each sample image.

[0087] Similarly, the above-mentioned audio representations of each sample can be obtained through the above-mentioned second encoder.

[0088] Step S206, adjusting parameters of the first encoder and the second encoder with the goal of minimizing the first loss.

[0089] The first loss is positively correlated to the distances between different modalities of each sample image representation and the corresponding sample audio representation of the first video sample, and the distances between adjacent sample image representations in the same modality of each sample image representation of the first video sample.

[0090] In one embodiment, the distance between different modes or the distance between the same mode may be, for example, Euclidean distance or cosine distance.

[0091] It should be noted that since the above-mentioned first loss is positively correlated with the distance between different modes of each sample image representation and the corresponding sample audio representation of the first video sample, when the evaluation model is trained with the goal of minimizing the first loss (i.e., adjusting the parameters of the first encoder and the second encoder), the distance between different modes of the image frames and the corresponding audio frames of the video with synchronized audio and video will be reduced, that is, the image frames and the corresponding audio frames of the video with synchronized audio and video will obtain similar representations.

[0092] In addition, since the above-mentioned first loss is also positively correlated with the isomodal distance of adjacent sample image representations in each sample image representation of the first video sample, when the evaluation model is trained with the goal of minimizing the first loss, the isomodal distance between adjacent image frames of the audio and video synchronized video will be reduced, that is, the adjacent image frames of the audio and video synchronized video will obtain similar representations.

[0093] Secondly, when the sample set used to train the evaluation model can also include a second video sample with unsynchronized audio and video, the above-mentioned first loss can also be negatively correlated with the distance between different modalities of each sample image representation and the corresponding sample audio representation of the second video sample. In other words, when the evaluation model is trained with the goal of minimizing the first loss, the distance between different modalities of the image frame and the corresponding audio frame of the video with unsynchronized audio and video will increase, that is, the representation of the image frame and the corresponding audio frame of the video with unsynchronized audio and video will be moved away from each other.

[0094] It should be noted that, considering that the characters in the actually produced videos may be virtual images, such as cartoon images. Although cartoon images and real people generally have some similar features, such as eyes, mouths, heads, and limbs, the differences between image domains will lead to inaccurate evaluation of virtual image videos (such as digital human videos) by the evaluation model trained only with real-life videos. Therefore, the sample set used to train the evaluation model may also include a third video sample. The third video sample can be separated into audio and video to obtain a cartoon image frame sequence.

[0095] In addition, considering that the sample audio frame sequence obtained by separating the video sample may correspond to a synthetic audio (ext to Speech, TTS) frame sequence. Since there are certain differences between the synthetic audio and real-person speech, the sample set for training the evaluation model may also include a fourth video sample. The fourth video sample may be separated into audio and video to obtain a synthetic audio frame sequence.

[0096] Regarding the third video sample and the fourth video sample, there is no requirement for whether the audio and video are synchronized. Therefore, it is relatively easy to collect such samples.

[0097] When the above-mentioned sample set also includes the third and fourth video samples, the above-mentioned first loss is also positively correlated to the first target distance between the first image comprehensive representation of each sample image representation corresponding to the cartoon image frame sequence and the second image comprehensive representation of each sample image representation of the first video sample, and the second target distance between the first audio comprehensive representation of each sample audio representation corresponding to the synthetic audio frame sequence and the second audio comprehensive representation of each sample audio representation of the first video sample.

[0098] First, the first target distance or the second target distance may be, for example, a Euclidean distance or a cosine distance.

[0099] Next, each sample image representation corresponding to the cartoon image frame sequence may be obtained by encoding each cartoon image in the cartoon image frame sequence through the first encoder, and each sample audio representation corresponding to the synthetic audio frame sequence may be obtained by encoding each synthetic audio frame in the synthetic audio frame sequence through the second encoder.

[0100] Secondly, the first image comprehensive representation may be obtained by averaging or summing the sample image representations corresponding to the cartoon image frame sequence. Similarly, the second image comprehensive representation may be obtained. And, the first audio comprehensive representation may be obtained by averaging or summing the sample audio representations corresponding to the synthetic audio frame sequence. Similarly, the second audio comprehensive representation may be obtained.

[0101] It should be noted that, since the first loss is positively correlated with the first target distance and the second target distance, when the evaluation model is trained with the goal of minimizing the first loss, the distance between the sample image frame and the cartoon image frame will be reduced, that is, the sample image frame and the cartoon image frame will obtain similar representations. Also, the distance between the sample audio frame and the synthesized audio frame will be reduced, that is, the sample audio frame and the synthesized audio frame will obtain similar representations.

[0102] It should also be noted that the third video sample and the fourth video sample can be understood as target domain samples, and the number of them can be small. And the evaluation model trained based on the target domain samples can be understood as a cross-domain evaluation model. It should be understood that the cross-domain evaluation model can accurately evaluate both real-life videos (including real digital human videos) and cartoon digital human videos.

[0103] It should be understood that in order to ensure the accuracy of the evaluation model, the number of the first, second, third and fourth video samples may be multiple, so that the above steps S204 to S206 may be repeatedly performed.

[0104] After repeatedly executing steps S204 to S206, the final evaluation model can be obtained.

[0105] In summary, the method for training an evaluation model provided in the embodiments of this specification can train an evaluation model based on the different modal distances between the image frame and the corresponding audio frame of the video sample, as well as the same modal distance between adjacent image frames, so that the trained model can evaluate the consistency of audio and video at the frame level of the video, as well as the coherence of the video, that is, it can objectively and accurately evaluate the video quality from multiple dimensions. In addition, this solution can also use the method of transfer learning to achieve data domain migration, that is, only a small number of samples in the target domain are needed to easily train a cross-domain evaluation model. Finally, the evaluation method based on the evaluation model does not require the video to be evaluated to have accurate labels, so it has strong generalization.

[0106] It should be noted that the evaluation model trained in the above embodiments of this specification is specifically used to evaluate video quality, and the evaluation method is described below.

[0107] Figure 3 A schematic diagram of a method for evaluating video quality according to an embodiment is shown. Figure 3 In the method, the target video to be evaluated can be first obtained, and the audio and video of the target video can be separated to obtain a target image frame sequence and a corresponding target audio frame sequence. Then the two sequences can be input into the evaluation model for model processing. The model processing includes encoding each target image frame in the target image frame sequence using a first encoder to obtain each target image representation. And encoding each target audio frame in the target audio frame sequence using a second encoder to obtain each target audio representation.

[0108] Afterwards, a first evaluation index may be determined based on the different modality distances between each target image representation and the corresponding target audio representation, and a second evaluation index may be determined based on the same modality distances between adjacent target image representations in each target image representation. And the quality of the target video may be evaluated based on at least the first evaluation index and the second evaluation index.

[0109] In one embodiment, the target image frame sequence may be obtained by using a driving model to drive the character's image description data based on an audio segment formed by the target audio frame sequence.

[0110] The image description data of the character here can be, for example, an image, a video or a three-dimensional model, etc. The three-dimensional model can be a three-dimensional model of a digital human. It should be understood that when the image description data of the character is a three-dimensional model of a digital human, the target video can be a digital human video.

[0111] Regarding the above-mentioned driving model, it may correspond to the image description data of the character. Specifically, if the image description data of the character is a three-dimensional model, the above-mentioned driving model may be implemented based on a cross-modal conversion algorithm. For example, the cross-modal conversion algorithm is used to convert text or audio into a series of continuous pictures (i.e., image frame sequence). If the image description data of the character is an image or video, the above-mentioned driving model may be implemented based on a generative adversarial network (GAN, Generative Adversarial Networks).

[0112] Regarding the first evaluation index and the second evaluation index, in one embodiment, the distances between the target image representation and the corresponding target audio representation in multiple different modes can be averaged or summed, and the average result or summation result can be used as the first evaluation index. The first evaluation index can also be called a consistency evaluation index, which is a quantitative value.

[0113] In addition, the distances between multiple isomodalities of adjacent target image representations in each target image representation can be averaged or summed, and the average result or summation result can be used as the second evaluation index. The second evaluation index can also be called a fluency index, which is also a quantitative value. Generally speaking, if the second evaluation index is small, it means that the picture motion between adjacent image frames is smoother and the overall fluency is high. Otherwise, it means that the picture has a sudden change and the overall fluency is low.

[0114] The above-mentioned evaluation of the quality of the target video may include that if the first evaluation index and the second evaluation index are both less than their corresponding index thresholds, then the evaluation result may be that the audio and video of the target video are synchronized. If any one of the first evaluation index and the second evaluation index is not less than the corresponding index threshold, then the evaluation result may be that the audio and video of the target video are not synchronized.

[0115] In summary, this scheme can objectively and quantitatively evaluate the accuracy and coherence of videos from two dimensions.

[0116] It should be noted that the evaluation results obtained through the above method embodiment can be used to assist in training the driving model, thereby greatly improving the accuracy of the driving model.

[0117] Figure 4 A schematic diagram of a method for training a driving model in one example is shown. Figure 4 In the method, an initial sample set for training a driving model can be obtained, wherein a single sample may include image description data of a person, an audio clip and a sample label, and the sample label can be used to indicate a real image frame sequence obtained by driving the image description data of the person based on the audio clip.

[0118] For any first sample in the initial sample set, by executing Figure 3 In the method steps shown, a first synthetic video obtained based on an audio clip and a real image frame sequence is evaluated. If the evaluation result indicates that the audio and video of the first synthetic video are not synchronized, a weight value lower than a predetermined threshold is added to the first sample, and the weight value is used to indicate that the first sample is a dirty sample containing noise. For example, there are multiple people talking or background sound in the first synthetic video. Based on at least the first sample with the weight value added, the driving model is trained.

[0119] It should be noted that by setting a lower weight for the dirty sample, the driving model can reduce its dependence on the dirty sample, thereby filtering the dirty samples in the sample set. Therefore, the method of training the driving model in this example can solve the problem that the sample set includes dirty samples, and further improve the accuracy of the trained driving model.

[0120] Figure 5 A schematic diagram of a method for training a driving model in another example is shown. Figure 5 In the dataset, a clean sample set for training the driving model can be obtained, where a single sample includes a person's image description data and an audio clip. In addition, a sample label indicating a real image frame sequence can also be included.

[0121] For any first sample in the clean sample set, the character image description data and audio clips are input into the driving model to obtain a predicted image frame sequence. Figure 3 In each method step shown, a second synthesized video obtained based on the predicted image frame sequence and the audio clip is evaluated. If the evaluation result indicates that the audio and video of the second synthesized video are not synchronized, a corresponding penalty value is added to the first sample, and the penalty value is used to indicate that the first sample is an important sample. Based on at least the first sample with the penalty value added, the driving model is trained.

[0122] It should be noted that the method for training the driving model provided in this other example is based on a clean sample set, that is, the sample set does not include dirty samples.

[0123] In addition, by setting penalty values ​​for important samples, the learning of important samples by the driving model can be strengthened.

[0124] It should also be noted that in the method for training the driving model provided in this other example, the evaluation results of the video are used as an additional supervisory signal to assist the training of the driving model. This can solve the problem that when the training of the driving model is directly supervised by the real image frame sequence, the supervision method is too single, which makes the training process more difficult.

[0125] Corresponding to the above-mentioned method for training an evaluation model, an embodiment of the present specification also provides a device for training an evaluation model, such as Figure 6As shown, the device may include:

[0126] The acquisition unit 602 is used to acquire a sample set, including a first video sample with synchronized audio and video.

[0127] The processing unit 604 is used to separate the audio and video of the first video sample to obtain a sample image frame sequence and a corresponding sample audio frame sequence, and input them into the evaluation model for model processing. The evaluation model includes a first encoder and a second encoder.

[0128] The processing unit 604 includes:

[0129] A first encoding submodule 6042 is used to encode each sample image frame in the sample image frame sequence using a first encoder to obtain each sample image representation;

[0130] The second encoding submodule 6044 is used to encode each sample audio frame in the sample audio frame sequence by using a second encoder to obtain each sample audio representation.

[0131] The adjustment unit 606 is used to adjust the parameters of the first encoder and the second encoder with the goal of minimizing the first loss, where the first loss is positively correlated to the distance between different modes of each sample image representation of the first video sample and the corresponding sample audio representation, and the distance between the same mode of adjacent sample image representations in each sample image representation of the first video sample.

[0132] In one embodiment, the sample set further includes a second video sample in which the audio and the video are not synchronized;

[0133] The first loss is negatively correlated to the distance between different modalities of each sample image representation and the corresponding sample audio representation of the second video sample.

[0134] In one embodiment, the sample set further includes a third video sample and a fourth video sample, wherein the third video sample is subjected to audio and video separation to obtain a cartoon image frame sequence, and the fourth video sample is subjected to audio and video separation to obtain a synthetic audio frame sequence;

[0135] The above-mentioned first loss is also positively correlated to the first target distance between the first image comprehensive representation of each sample image representation corresponding to the cartoon image frame sequence and the second image comprehensive representation of each sample image representation of the first video sample, and the second target distance between the first audio comprehensive representation of each sample audio representation corresponding to the synthetic audio frame sequence and the second audio comprehensive representation of each sample audio representation of the first video sample.

[0136] The functions of the functional modules of the device in the above-mentioned embodiment of this specification can be implemented through the steps of the above-mentioned method embodiment. Therefore, the specific working process of the device provided by one embodiment of this specification will not be repeated here.

[0137] An embodiment of the present specification provides a device for training an evaluation model, and the trained evaluation model can objectively and accurately evaluate video quality from multiple dimensions.

[0138] Corresponding to the above-mentioned method for evaluating video quality, an embodiment of the present specification also provides a device for evaluating video quality, such as Figure 7 As shown, the device may include:

[0139] The acquisition unit 702 is used to acquire the target video to be evaluated.

[0140] In one embodiment, the target video is a digital human video.

[0141] The processing unit 704 is used to separate the audio and video of the target video, obtain a target image frame sequence and a corresponding target audio frame sequence, and input them into the evaluation model for model processing. The evaluation model includes a first encoder and a second encoder.

[0142] The processing unit 704 includes:

[0143] A first encoding submodule 7042 is used to encode each target image frame in the target image frame sequence using a first encoder to obtain each target image representation;

[0144] The second encoding submodule 7044 is used to encode each target audio frame in the target audio frame sequence by using a second encoder to obtain each target audio representation.

[0145] The determination unit 706 is used to determine a first evaluation index based on the different modality distances between each target image representation and the corresponding target audio representation, and to determine a second evaluation index based on the same modality distances between adjacent target image representations in each target image representation.

[0146] The evaluation unit 708 is configured to evaluate the quality of the target video based at least on the first evaluation index and the second evaluation index.

[0147] In one embodiment, the target image frame sequence is obtained by using a driving model to drive the character's image description data based on an audio segment formed by a target audio frame sequence.

[0148] The above-mentioned character image description data includes any one of the following: an image, a video, and a three-dimensional model.

[0149] The functions of the functional modules of the device in the above-mentioned embodiment of this specification can be implemented through the steps of the above-mentioned method embodiment. Therefore, the specific working process of the device provided by one embodiment of this specification will not be repeated here.

[0150] An embodiment of the present specification provides a device for evaluating video quality, which objectively and quantitatively evaluates the accuracy and coherence of a video from two dimensions.

[0151] Corresponding to the above-mentioned method for training a driving model, an embodiment of the present specification also provides a device for training a driving model, such as Figure 8 As shown, the device may include:

[0152] The acquisition unit 802 is used to acquire an initial sample set for training the driving model, wherein a single sample includes a character image description data, an audio clip and a sample label, and the sample label is used to indicate a real image frame sequence obtained by driving the character image description data based on the audio clip.

[0153] The evaluation unit 804 is used for evaluating any first sample by executing Figure 3 The method steps shown evaluate a first synthetic video obtained based on a real image frame sequence and an audio clip.

[0154] The adding unit 806 is configured to add a weight value lower than a predetermined threshold to the first sample if the evaluation result indicates that the audio and video of the first composite video are not synchronized, and the weight value is used to indicate that the first sample is a dirty sample containing noise.

[0155] The training unit 808 is used to train the driving model based on at least the first sample with the weight value added.

[0156] The functions of the functional modules of the device in the above-mentioned embodiment of this specification can be implemented through the steps of the above-mentioned method embodiment. Therefore, the specific working process of the device provided by one embodiment of this specification will not be repeated here.

[0157] An apparatus for training a driving model provided in an embodiment of the present specification can improve the accuracy of the trained driving model.

[0158] Corresponding to the above-mentioned method for training a driving model, an embodiment of the present specification also provides a device for training a driving model, such as Fig. 9 As shown, the device may include:

[0159] The acquisition unit 902 is used to acquire a clean sample set for training the driving model, wherein a single sample includes image description data of a character and an audio clip.

[0160] The input unit 904 is used to input the image description data and the audio clip of the character in any first sample into the driving model to obtain a predicted image frame sequence.

[0161] Evaluation unit 906 is used to execute Figure 3The method steps shown evaluate a second synthetic video based on the predicted image frame sequence and the audio segment.

[0162] The adding unit 908 is configured to add a corresponding penalty value to the first sample if the evaluation result indicates that the audio and video of the second composite video are not synchronized, and the penalty value is used to indicate that the first sample is an important sample.

[0163] The training unit 910 is used to train the driving model based on at least the first sample to which the penalty value is added.

[0164] The functions of the functional modules of the device in the above-mentioned embodiment of this specification can be implemented through the steps of the above-mentioned method embodiment. Therefore, the specific working process of the device provided by one embodiment of this specification will not be repeated here.

[0165] An apparatus for training a driving model provided in an embodiment of the present specification can improve the accuracy of the trained driving model.

[0166] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 2 or Figure 3 The method described.

[0167] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Figure 2 or Figure 3 The method described.

[0168] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0169] The steps of the method or algorithm described in conjunction with the disclosure of this specification can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, mobile hard disk, CD-ROM or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a server. Of course, the processor and the storage medium can also be present in the server as discrete components.

[0170] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention may be implemented in hardware, software, firmware, or any combination thereof. When implemented using software, the functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium may be any available medium that a general or special-purpose computer can access.

[0171] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0172] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this specification. It should be understood that the above description is only the specific implementation method of this specification and is not intended to limit the protection scope of this specification. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of this specification should be included in the protection scope of this specification.

Claims

1. A method for training an evaluation model, comprising: Acquire a sample set, including a first video sample and a third video sample with synchronized audio and video, wherein audio and video separation is performed on the third video sample to obtain a cartoon image frame sequence; Performing audio and video separation on the first video sample to obtain a sample image frame sequence and a corresponding sample audio frame sequence, and inputting the sample image frame sequence and the corresponding sample audio frame sequence into the evaluation model for model processing, wherein the evaluation model includes a first encoder and a second encoder; the model processing includes encoding each sample image frame in the sample image frame sequence using the first encoder to obtain each sample image representation, and encoding each sample audio frame in the sample audio frame sequence using the second encoder to obtain each sample audio representation; The parameters of the first encoder and the second encoder are adjusted with the goal of minimizing a first loss; the first loss is positively correlated to the distances between different modalities of each sample image representation and the corresponding sample audio representation of the first video sample, and the distances between the same modalities of adjacent sample image representations in each sample image representation of the first video sample; the trained evaluation model is used to evaluate the quality of the synthesized video; The first loss is also positively correlated to a first target distance between a first image comprehensive representation of each sample image representation corresponding to the cartoon image frame sequence and a second image comprehensive representation of each sample image representation of the first video sample; the first and second image comprehensive representations are obtained by averaging or summing the corresponding sample image representations.

2. The method according to claim 1, wherein: The sample set also includes a second video sample whose audio and video are not synchronized; The first loss is negatively correlated to the distance between different modalities of each sample image representation and the corresponding sample audio representation of the second video sample.

3. The method according to claim 1, wherein: The sample set further includes a fourth video sample, and audio and video separation is performed on the fourth video sample to obtain a synthetic audio frame sequence; The first loss is also positively correlated to a second target distance between a first audio comprehensive representation of each sample audio representation corresponding to the synthetic audio frame sequence and a second audio comprehensive representation of each sample audio representation of the first video sample; the first and second audio comprehensive representations are obtained by averaging or summing the corresponding sample audio representations.

4. A method for evaluating video quality, comprising: Obtain the target video to be evaluated; The target video is a synthesized video; Separating the target video from the audio and the video to obtain a target image frame sequence and a corresponding target audio frame sequence, and inputting the target image frame sequence and the corresponding target audio frame sequence into the evaluation model trained according to the method of claim 1 for model processing; the model processing includes encoding each target image frame in the target image frame sequence using a first encoder to obtain each target image representation, and encoding each target audio frame in the target audio frame sequence using a second encoder to obtain each target audio representation; Determining a first evaluation index based on different-modality distances between each target image representation and a corresponding target audio representation, and determining a second evaluation index based on same-modality distances between adjacent target image representations in each target image representation; The quality of the target video is evaluated based at least on the first evaluation index and the second evaluation index.

5. The method according to claim 4, wherein: The target image frame sequence is obtained by using a driving model to drive the character's image description data based on an audio segment formed by a target audio frame sequence.

6. The method according to claim 5, wherein: The image description data includes any one of the following: an image, a video, and a three-dimensional model.

7. The method according to claim 4, wherein: The target video is a digital human video.

8. A method for training a driving model, comprising: Acquire an initial sample set for training the driving model, wherein a single sample includes image description data of a person, an audio clip, and a sample label; The sample label is used to indicate a real image frame sequence obtained by driving the character's image description data based on the audio clip; For any first sample, evaluating a first synthetic video obtained based on the real image frame sequence and the audio clip according to the method of claim 4; If the evaluation result indicates that the audio and video of the first composite video are not synchronized, adding a weight value lower than a predetermined threshold to the first sample; the weight value is used to indicate that the first sample is a dirty sample containing noise; The driving model is trained based on at least the first sample to which the weight value is added.

9. A method for training a driving model, comprising: Obtaining a clean sample set for training the driving model, wherein a single sample includes image description data of a person and an audio clip; For any first sample, input the image description data and audio clip of the character therein into the driving model to obtain a predicted image frame sequence; According to the method of claim 4, evaluating a second synthetic video obtained based on the predicted image frame sequence and the audio segment; If the evaluation result indicates that the audio and video of the second composite video are not synchronized, a corresponding penalty value is added to the first sample; the penalty value is used to indicate that the first sample is an important sample; The driving model is trained based on at least the first sample to which the penalty value is added.

10. A device for training an evaluation model, comprising: An acquisition unit is used to acquire a sample set, including a first video sample and a third video sample with synchronized audio and video, wherein audio and video separation is performed on the third video sample to obtain a cartoon image frame sequence; a processing unit, configured to perform audio and video separation on the first video sample, obtain a sample image frame sequence and a corresponding sample audio frame sequence, and input the sample image frame sequence and the corresponding sample audio frame sequence into the evaluation model for model processing, wherein the evaluation model includes a first encoder and a second encoder; The processing unit comprises: A first encoding submodule, configured to encode each sample image frame in the sample image frame sequence using the first encoder to obtain each sample image representation; A second encoding submodule, configured to encode each sample audio frame in the sample audio frame sequence using the second encoder to obtain each sample audio representation; an adjustment unit, configured to adjust parameters of the first encoder and the second encoder with the goal of minimizing a first loss; the first loss is positively correlated to the distances between different modalities of each sample image representation and the corresponding sample audio representation of the first video sample, and the distances between the same modalities of adjacent sample image representations in each sample image representation of the first video sample; and the trained evaluation model is used to evaluate the quality of the synthesized video; The first loss is also positively correlated to a first target distance between a first image comprehensive representation of each sample image representation corresponding to the cartoon image frame sequence and a second image comprehensive representation of each sample image representation of the first video sample; the first and second image comprehensive representations are obtained by averaging or summing the corresponding sample image representations.

11. A device for evaluating video quality, comprising: An acquisition unit, used for acquiring a target video to be evaluated; The target video is a synthesized video; A processing unit, configured to perform audio and video separation on the target video, obtain a target image frame sequence and a corresponding target audio frame sequence, and input the target image frame sequence and the target audio frame sequence into an evaluation model trained by the method according to claim 1 for model processing; The processing unit comprises: A first encoding submodule, configured to encode each target image frame in the target image frame sequence using a first encoder to obtain each target image representation; A second encoding submodule, configured to encode each target audio frame in the target audio frame sequence using a second encoder to obtain each target audio representation; a determining unit, configured to determine a first evaluation index based on the different-modality distances between each target image representation and the corresponding target audio representation, and to determine a second evaluation index based on the same-modality distances between adjacent target image representations in each target image representation; An evaluation unit is used to evaluate the quality of the target video based at least on the first evaluation index and the second evaluation index.

12. A device for training a driving module, comprising: An acquisition unit is used to acquire an initial sample set for training the driving model, wherein a single sample includes image description data of a person, an audio clip and a sample label; the sample label is used to indicate a real image frame sequence obtained by driving the image description data of the person based on the audio clip; An evaluation unit for, for any first sample, According to the method of claim 4, evaluating a first synthetic video obtained based on the real image frame sequence and the audio clip; an adding unit, configured to add a weight value lower than a predetermined threshold to the first sample if the evaluation result indicates that the audio and video of the first composite video are not synchronized; The weight value is used to indicate that the first sample is a dirty sample containing noise; A training unit is used to train the driving model based on at least the first sample to which the weight value is added.

13. A device for training a driving model, comprising: An acquisition unit, used for acquiring a clean sample set for training the driving model, wherein a single sample includes image description data of a person and an audio clip; An input unit, for inputting the image description data and the audio clip of the character in any first sample into the driving model to obtain a predicted image frame sequence; an evaluation unit, configured to evaluate a second synthesized video obtained based on the predicted image frame sequence and the audio segment according to the method of claim 4; an adding unit, configured to add a corresponding penalty value to the first sample if the evaluation result indicates that the audio and video of the second composite video are not synchronized; the penalty value is used to indicate that the first sample is an important sample; A training unit is used to train the driving model based on at least the first sample to which the penalty value is added.

14. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 9.

15. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 9 is implemented.