Video generation method and apparatus, model training method and apparatus, device, storage medium, computer program, and computer program product

By obtaining voice features and head features, and generating videos matching audio, the problem of unreasonable actions of speakers in the prior art is solved and the video quality is improved.

WO2024244554A1PCT designated stage expired Publication Date: 2024-12-05MOORE THREADS TECH CO LTD

Patent Information

Application Number
PCT/CN2024/078798
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-02
Filing Date
2024-02-27
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

In the prior art, the speaking actions of the characters in the voice-driven speaker video are not reasonable and realistic enough, resulting in low video quality.

Method used

By obtaining voice features and head features, generating videos that match audio, using mask processing to improve the accuracy of head features and the rationality of voice features, and thus improving video quality.

Benefits of technology

It improves the rationality and fidelity of the characters' speech actions in the video and improves the quality of the video.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024078798_05122024_PF_FP_ABST
    Figure CN2024078798_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present disclosure provide a video generation method and apparatus, a model training method and apparatus, a device, a storage medium, a computer program, and a computer program product. The video generation method comprises: acquiring a first voice feature and a first head feature of a first character, wherein the first head feature is obtained by performing mask processing on a mouth area of the first character in a first video, and the first voice feature is obtained on the basis of a first audio; on the basis of the first head feature and the first voice feature, generating a second video containing the first character, wherein a speaking action of the first character in the second video matches the first audio.
Need to check novelty before this filing date? Find Prior Art

Description

Video generation and model training method, device, equipment, storage medium, computer program and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The embodiments of the present disclosure are based on and claim the priority of Chinese patent application with application number 202310646787.8, application date June 2, 2023, and application name “Video Generation and Model Training Method, Device, Equipment, Storage Medium”. The entire content of the Chinese patent application is hereby introduced into the present disclosure as a reference. Technical Field

[0003] The present disclosure relates to, but is not limited to, the field of artificial intelligence technology, and in particular to a video generation and model training method, apparatus, device, storage medium, computer program, and computer program product. Background Art

[0004] Speech-driven speaker synthesis uses voice information to drive a target person's facial movements to produce realistic-sounding speaking videos. With the development of artificial intelligence (AI), speech-driven speaker synthesis is becoming increasingly popular. However, in related technologies, the speech movements of the speaker in speech-driven generated videos are not realistic enough, resulting in low video quality.

[0005] Summary of the Invention

[0006] In view of this, the embodiments of the present disclosure provide at least one video generation and model training method, apparatus, device, storage medium, computer program, and computer program product, which can improve the rationality and realism of the overall speaking movements of the characters in the generated video, thereby improving the quality of the generated video.

[0007] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0008] An embodiment of the present disclosure provides a video generation method, which is executed by an electronic device and includes:

[0009] Obtaining a first voice feature and a first head feature of a first person; wherein the first head feature is obtained by masking the mouth area of ​​the first person in the first video, and the first voice feature is obtained based on the first audio;

[0010] Based on the first head feature and the first voice feature, a second video containing the first person is generated; the speaking action of the first person in the second video is matched with the first audio.

[0011] An embodiment of the present disclosure provides a model training method, which is performed by an electronic device and includes:

[0012] Obtaining first voice training data and first head training data of a second person; wherein the first head training data is obtained by masking the mouth area of ​​the second person in the training video, and the first voice training data is obtained based on the training audio;

[0013] Inputting the first voice training data and the first head training data into the to-be-trained model to obtain a fifth video including the second person;

[0014] Determining a target loss using the fifth video, and adjusting parameters of the to-be-trained model based on the target loss;

[0015] When the speaking action of the second character in the fifth video matches the training audio, it is determined that the training of the model to be trained is completed.

[0016] An embodiment of the present disclosure provides a video generation device, the device comprising:

[0017] a first acquisition section configured to acquire a first voice feature and a first head feature of a first person; wherein the first head feature is obtained by masking a mouth region of the first person in the first video, and the first voice feature is obtained based on the first audio;

[0018] The first generating part is configured to generate a third video containing the first person based on the first head feature and the first voice feature; the speaking action of the first person in the third video matches the first audio.

[0019] The present disclosure provides a model training device, comprising:

[0020] The second acquisition part is configured to acquire first voice training data and first head training data of a second person; wherein the first head training data is obtained by masking the mouth area of ​​the second person in the training video; and the first voice training data is obtained based on the training audio;

[0021] The second generating part is configured to input the first voice training data and the first head training data into the to-be-trained model to obtain a fifth video containing the second person;

[0022] an adjustment part, configured to determine a target loss using the fifth video, and adjust parameters of the to-be-trained model based on the target loss;

[0023] The first determination part is configured to determine that the training of the model to be trained is completed when the speaking action of the second character in the fifth video matches the training audio.

[0024] An embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, it implements some or all of the steps in the above-mentioned video generation method or model training method.

[0025] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, some or all of the steps in the above-mentioned video generation method or model training method are implemented.

[0026] An embodiment of the present disclosure provides a computer program, including computer-readable codes. When the computer-readable codes are executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0027] An embodiment of the present disclosure provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0028] An embodiment of the present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, the above method is implemented.

[0029] In the embodiment of the present disclosure, by obtaining the first voice feature and the first head feature of the first person, and based on the first head feature and the first voice feature, a second video containing the first person is generated, and the speaking action of the first person in the second video matches the first audio. In this way, since the first head feature is obtained by masking the mouth area of ​​the first person in the first video, the first head feature is not affected by the mouth area and is more reasonable and accurate, and the first voice feature is obtained based on the first audio, so that in the second video generated based on the first head feature and the first voice feature, the head appearance posture of the first person will be more accurate, and the head appearance posture will be more matched with the first audio, thereby making the overall speaking action of the person more reasonable and realistic. In this way, the rationality and realism of the overall speaking action of the first person in the generated second video can be improved, thereby improving the quality of the generated second video.

[0030] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0032] FIG1 is a schematic diagram of a first implementation flow of a video generation method provided by an embodiment of the present disclosure;

[0033] FIG2 is a second schematic diagram of an implementation flow of a video generation method provided by an embodiment of the present disclosure;

[0034] FIG3 is a third schematic diagram of an implementation flow of a video generation method provided by an embodiment of the present disclosure;

[0035] FIG4 is a schematic diagram of a first implementation flow of a model training method provided by an embodiment of the present disclosure;

[0036] FIG5 is a second schematic diagram of an implementation flow of a model training method provided by an embodiment of the present disclosure;

[0037] FIG6A is a fourth schematic diagram of an implementation flow of a video generation method provided by an embodiment of the present disclosure;

[0038] FIG6B is a schematic diagram illustrating an implementation of training a content encoder and a sentiment encoder using a cross-reconstruction decoupling network according to an embodiment of the present disclosure;

[0039] FIG. 7A is a schematic diagram of the structure of a video generating device provided by an embodiment of the present disclosure.

[0040] FIG7B is a schematic diagram of the structure of a model training device provided by an embodiment of the present disclosure;

[0041] FIG8 is a schematic diagram of a hardware entity of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure are further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0043] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] The terms "first / second / third" involved are used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with the specific order or sequence where permitted so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are for the purpose of describing the present disclosure only and are not intended to limit the present disclosure.

[0046] In related art, a feature encoder is typically used to extract features from the reference video and the driving audio separately, obtaining features from both the video and audio modalities. These features are then combined at a feature decoder to generate a video that matches the driving audio. However, these video generation methods in related art neglect modeling of head shape and posture, resulting in a poor fit between the head shape and posture of the character in the generated video and the driving audio. Consequently, the character's speech movements lack plausibility and realism, resulting in suboptimal video quality.

[0047] In view of this, embodiments of the present disclosure provide a video generation method that can be executed by a processor of a computer device. The computer device may refer to a server, laptop, tablet computer, desktop computer, smart TV, set-top box, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or other device with data processing capabilities.

[0048] In order to better understand the embodiments of the present disclosure, some of the terms involved in the embodiments of the present disclosure are explained below:

[0049] First video: a pre-set video containing the first person;

[0050] Second video: the video to be generated containing the first person;

[0051] The third video is a video obtained by masking the mouth area of ​​the first person in the first video;

[0052] Fourth video: a video to be generated containing the first person, wherein the second video can be obtained by performing image fusion on the face region and the background region in at least one video frame in the fourth video;

[0053] First audio: a preset audio used to drive the first person to speak in the second video to be generated;

[0054] Training video: video used for model training, which contains a second person;

[0055] Fifth video: a video containing the second person generated during model training;

[0056] Sixth video: a video obtained by masking the mouth area of ​​the second person in the training video;

[0057] Training audio: pre-set audio used to drive the second person's speech in the fifth video to be generated;

[0058] Reference video: A pre-set video tag corresponding to the training audio. The reference video includes a second person, and the speaking movements of the second person in the reference video match the training audio.

[0059] FIG1 is a schematic diagram of an implementation flow of a video generation method provided by an embodiment of the present disclosure. As shown in FIG1 , the method may include the following steps S101 to S102:

[0060] Step S101, obtaining a first voice feature and a first head feature of a first person; wherein the first head feature is obtained by masking the mouth area of ​​the first person in the first video, and the first voice feature is obtained based on the first audio.

[0061] Here, the first audio can be pre-set audio used to drive the speech of the character in the second video to be generated. The first audio can include any suitable voice content. During implementation, those skilled in the art can use any suitable method to obtain the first audio according to actual circumstances, and the embodiments of the present disclosure are not limited to this. For example, the first audio can be pre-collected audio, or the first audio can be audio synthesized based on a set text.

[0062] The first speech feature can be obtained by extracting features from the first audio. During implementation, any suitable feature extraction network can be used to extract features from the first audio to obtain the first speech feature, and this disclosure is not limited to this. The feature extraction network used to extract features from the first audio can include, but is not limited to, at least one of a CNN, an RNN, a Transformer encoder, and the like.

[0063] In some implementations, the first speech feature may include, but is not limited to, at least one of a first content feature representing speech content, a first emotion feature representing emotions contained in the speech, and the like.

[0064] The first video may include at least one video frame containing the first person. It is understandable that the first video may include a plurality of pre-collected or synthesized continuous video frames, or the first video may include at least one video frame sampled from a plurality of continuous video frames.

[0065] A first head feature can be obtained by masking the mouth region of the first person in the first video. The first head feature may include, but is not limited to, at least one of a first head action feature characterizing the head movement of the first person, a first non-mouth region feature characterizing the entire head region excluding the mouth region, and a first expression feature characterizing the facial expression of the first person.

[0066] In some implementations, a mask process may be performed on the mouth region of the first person in the first video to obtain a third video, and the first head feature of the first person may be extracted from the third video.

[0067] Here, the mouth area of ​​the first person contained in each video frame of the first video may be masked to obtain the third video.

[0068] In some embodiments, the mouth region of the first person in each video frame of the first video can be detected, and the corresponding mouth region in each video frame can be masked. During implementation, an object detection algorithm can be used to identify key points of the mouth in the video frame to obtain the mouth region of the first person in the video frame.

[0069] In some embodiments, the mouth area of ​​the first person in at least one video frame of the first video can be detected, and based on the mouth area of ​​the first person in the at least one video frame, the mouth areas corresponding to the respective video frames in the first video are determined, and the mouth areas in the respective video frames are masked. During implementation, those skilled in the art can use any suitable method to determine the mouth areas corresponding to the respective video frames in the first video based on the mouth area of ​​the first person in at least one video frame according to actual conditions. For example, the union or intersection of the mouth areas of the first person in at least one detected video frame can be determined as the mouth areas corresponding to the respective video frames in the first video. For another example, the mouth area of ​​the first person in a detected video frame can be determined as the mouth areas corresponding to the respective video frames in the first video.

[0070] By using any suitable feature extraction network to perform feature extraction on the third video, a first head feature of the first person can be obtained. Because the mouth region of the first person included in each video frame of the third video is masked, feature extraction on the third video can obtain the head feature of the first person in the third video excluding the mouth region, i.e., the first head feature.

[0071] During implementation, the feature extraction network used to extract features from the third video may include, but is not limited to, at least one of a convolutional neural network (CNN), a recurrent neural network (RNN), a transformer-based encoder network, etc. For example, a VGG network (such as VGG16, VGG19, etc.) may be used to extract features from the third video to obtain the first head features.

[0072] It can be understood that the above-mentioned process of extracting features from the first audio to obtain the first voice feature, the process of masking the mouth area of ​​the first person in the first video to obtain the third video, and the process of extracting the first head feature of the first person from the third video can be executed in the processor that executes the video generation method, or can be executed by other processors, which is not limited here.

[0073] Step S102: Generate a second video containing the first person based on the first head feature and the first voice feature, wherein the speaking action of the first person in the second video matches the first audio.

[0074] Here, a video frame sequence containing the first person can be rendered based on the first head feature and the first voice feature to generate a second video, and the speaking action of the first person in the second video matches the first audio. The speaking action of the first person may include but is not limited to at least one of lip shape, facial expression, head movement, etc.

[0075] During implementation, the generated second video may be without sound or with sound, and the disclosed embodiments do not limit this. For example, if the generated second video is without sound, those skilled in the art can add the first audio to the second video according to the actual application scenario to obtain a second video with sound.

[0076] In some implementations, after rendering and generating a video frame sequence containing the first person, the video frame sequence may be synthesized with the first audio to obtain a synthesized second video with sound.

[0077] In some embodiments, a pre-trained rendering generation network can be used to render and generate the second video based on the first head feature and the first voice feature. The rendering generation network can be any suitable neural network used by those skilled in the art based on actual conditions, and the embodiments of the present disclosure are not limited thereto.

[0078] In some embodiments, the first head feature and the first voice feature may be fused to obtain a first fused feature, and then a pre-trained rendering generation network may be used to render and generate a second video based on the first fused feature.

[0079] It is understood that the first head feature can be used to render the image portion corresponding to the non-mouth area in each video frame of the second video, and the first voice feature can be used to generate the image portion corresponding to the mouth area in each video frame of the second video, thereby generating a second video containing the first person based on the first head feature and the second head feature. In implementation, the image portion corresponding to the non-mouth area and the image portion corresponding to the mouth area in each video frame of the second video can be rendered separately and then fused together, or they can be rendered together.

[0080] In the embodiment of the present disclosure, by obtaining the first voice feature and the first head feature of the first person, and based on the first head feature and the first voice feature, a second video containing the first person is generated, and the speaking action of the first person in the second video matches the first audio. In this way, since the first head feature is obtained by masking the mouth area of ​​the first person in the first video, the first head feature is not affected by the mouth area and is more reasonable and accurate, and the first voice feature is obtained based on the first audio, so that in the second video generated based on the first head feature and the first voice feature, the head appearance posture of the first person will be more accurate, and the head appearance posture will be more matched with the first audio, thereby making the overall speaking action of the person more reasonable and realistic. In this way, the rationality and realism of the overall speaking action of the first person in the generated second video can be improved, thereby improving the quality of the generated second video.

[0081] In some embodiments, the first head feature includes a first non-mouth area feature and a first head motion feature, the first non-mouth area feature is obtained by encoding a third video, and the third video is obtained by masking the mouth area of ​​the first person in the first video; the first head motion feature is obtained by mapping the first non-mouth area feature to a preset motion feature space.

[0082] In some embodiments, the process of extracting the first head feature of the first person from the third video may include the following steps S111 to S112:

[0083] Step S111: Encode the third video to obtain a first non-mouth area feature.

[0084] Here, any appropriate encoder may be used to encode the third video to extract visual features of the first person in the third video except for the mouth area, and obtain first non-mouth area features.

[0085] During implementation, the encoder used to encode the third video may be pre-trained or may be initialized and untrained, which is not limited in the embodiments of the present disclosure.

[0086] In some embodiments, a trained first encoder can be used to encode each video frame in the third video to obtain a first non-mouth region feature. The first non-mouth region feature can include a non-mouth region sub-feature corresponding to each video frame. The first encoder can include, but is not limited to, at least one of a CNN-based encoder, an RNN-based encoder, a Transformer-based encoder, and the like. For example, the first encoder can include an encoder based on a VGG network (e.g., VGG16, VGG19, etc.).

[0087] Step S112: Map the first non-mouth area feature to a preset motion feature space to obtain a first head motion feature.

[0088] Here, the motion feature space can be used to define the motion posture representation of the character's head. Using a predetermined mapping subnetwork, the first non-mouth region feature can be mapped into the motion feature space to obtain a first head motion feature representing the motion of the first character's head. The mapping subnetwork may include, but is not limited to, at least one of a mapping layer, an embedding layer, and a fully connected layer, and the present disclosure is not limited thereto.

[0089] During implementation, the motion feature space can be pre-defined by those skilled in the art based on actual circumstances. For example, the motion feature space can have 12 dimensions, and each head motion feature in the motion feature space can be a 12-dimensional vector, where each head motion feature can include a 9-dimensional rotation matrix, a 2-dimensional translation representation, and a 1-dimensional scale representation.

[0090] In the above embodiment, the third video is encoded to obtain first non-mouth area features, which are then mapped to a preset motion feature space to obtain first head motion features. In this way, the visual features of the first person's non-mouth area and the first person's head movements can be modeled separately, so that the extracted first non-mouth area features can better characterize the visual features of the first person's non-mouth area, and the extracted first head motion features can better characterize the first person's head movements, thereby further improving the rationality and realism of the first person's overall speaking movements in the generated second video, thereby improving the quality of the generated second video.

[0091] In some embodiments, the above step S102 may include the following steps S121 to S122:

[0092] Step S121: Generate a fourth video containing the first person based on the first head feature and the first voice feature.

[0093] Step S122: performing image fusion on the face area and the background area in at least one fourth video frame in the fourth video to obtain the second video.

[0094] Here, after rendering and generating a fourth video containing the first person based on the first head feature and the first voice feature, an image fusion algorithm can be used to fuse the face area and the background area of ​​at least one fourth video frame in the fourth video to obtain a second video.

[0095] During implementation, any suitable image fusion method may be used to fuse the face region and the background region in at least one fourth video frame of the fourth video, and the present disclosure is not limited thereto. For example, the image fusion method may include, but is not limited to, at least one of Poisson fusion, Laplacian pyramid fusion, and multi-band fusion.

[0096] In the above embodiment, a fourth video containing the first person is generated based on the first head feature and the first voice feature; and image fusion is performed on the face region and background region of at least one fourth video frame in the fourth video to generate the second video. This allows for a more natural transition between the face region and background region in each video frame of the generated second video, thereby reducing subtle artifacts in the second video and subtle variations in the appearance of the first person across each video frame of the second video, thereby improving the consistency of the appearance of the first person across each video frame of the second video.

[0097] The present disclosure provides a video generation method, which can be executed by a processor of a computer device. FIG2 is a schematic diagram of an implementation flow of a video generation method provided by the present disclosure. As shown in FIG2 , the method may include the following steps S201 to S203:

[0098] Step S201, obtaining a first voice feature and a first head feature of a first person; wherein the first head feature is obtained by masking the mouth area of ​​the first person in the first video, and the first voice feature is obtained based on the first audio.

[0099] Here, step S201 corresponds to step S101 in the aforementioned embodiment, and the implementation may refer to the aforementioned implementation of step S101.

[0100] Step S202: Fusing the first head feature and the first speech feature to obtain a first fused feature.

[0101] Here, the process of fusing the first head feature and the first voice feature may include but is not limited to at least one of processes such as concatenation and convolution fusion of the first head feature and the first voice feature.

[0102] In some implementations, the first head feature and the first voice feature may be integrated and fused to obtain a first fused feature.

[0103] In some embodiments, the first head feature may include a first head sub-feature corresponding to each video frame in the first video, the first voice feature may include a first voice sub-feature corresponding to each video frame in the first video, and the first fusion feature may include a first fusion sub-feature corresponding to each video frame in the first video. During implementation, each first head sub-feature in the first head feature may be fused with each first voice sub-feature in the first voice feature at the frame level to obtain a fused first fusion feature. For example, for each video frame in the first video, the first head sub-feature and the first voice sub-feature corresponding to the video frame may be spliced ​​to obtain the first fusion sub-feature corresponding to the video frame, thereby obtaining the first fusion feature.

[0104] Step S203: Generate a second video based on the first fusion feature, wherein the speaking action of the first person in the second video matches the first audio.

[0105] In some embodiments, a pre-trained rendering generation network can be used to render and generate the second video based on the first fused features. The rendering generation network can be any suitable neural network used by those skilled in the art based on practical circumstances, and the present disclosure is not limited thereto. For example, the rendering generation network can include, but is not limited to, an encoder-decoder network, a decoder network, or a generative adversarial network.

[0106] In the disclosed embodiment, the first head feature and the first voice feature are first fused to obtain a first fused feature, and then based on the first fused feature, a second video can be generated more accurately so that the speaking action of the first person in the second video can better match the first audio.

[0107] In some embodiments, the first speech feature includes a first content feature and a first emotion feature. The above step S202 may include the following step S211:

[0108] Step S211: Fusing the first header feature and the first content feature to obtain a first fused feature.

[0109] Here, the first speech feature may include a first content feature representing the speech content and a first emotion feature representing the emotion contained in the speech. In implementation, the first content feature and the first emotion feature may be extracted from the first audio respectively.

[0110] In some implementations, two encoders may be used to respectively extract features from the first audio to obtain a first content feature and a first emotion feature.

[0111] The process of fusing the first head feature and the first content feature may include, but is not limited to, at least one of splicing, convolution fusion, and the like.

[0112] In some implementations, the first header feature and the first content feature may be integrated to obtain a first fused feature.

[0113] In some embodiments, multiple first head sub-features corresponding to different video frames in the first head feature and first content sub-features corresponding to different video frames in the first content feature can be frame-level fused to obtain a fused first fused feature.

[0114] The above step S203 may include the following step S212:

[0115] Step S212: Generate the second video based on the first fusion feature and the first emotion feature.

[0116] Here, a pre-trained rendering generation network can be used to render and generate the second video based on the first fusion feature and the first emotion feature. For example, the rendering generation network can include but is not limited to at least one of an encoder-decoder network, a generator in a generative adversarial network, and the like.

[0117] In some embodiments, the first fused feature and the first emotion feature may be encoded to obtain a feature code, which is then decoded to generate the second video. For example, the rendering generation network may include a fourth encoder and a first decoder. The fourth encoder may be used to encode the first fused feature and the first emotion feature to obtain a feature code, which is then decoded by the first decoder to generate the second video.

[0118] In some implementations, the first fusion feature may be encoded to obtain a feature code, and then the feature code and the first emotion feature may be decoded to generate a second video.

[0119] In the above embodiment, the first content feature and the first emotion feature can be extracted separately from the first audio; the first head feature and the first content feature are fused to obtain a first fused feature, and the second video is generated based on the first fused feature and the first emotion feature. In this way, the content features and the emotion features in the first audio can be decoupled, so that while the first content feature drives the generation of mouth movements, the first person's facial expression can be adaptively adjusted in combination with the first emotion feature. This allows the head appearance and posture of the first person in the generated second video to better match the speech content and emotion in the first audio.

[0120] In some embodiments, the above step S212 may include the following steps S221 to S222:

[0121] Step S221: Encode the first fusion feature to obtain the encoded first fusion feature.

[0122] Step S222: Decode the first emotion feature and the encoded first fusion feature to obtain the second video.

[0123] In some embodiments, the first fusion feature can be encoded using a fourth encoder to obtain an encoded first fusion feature, and then the first emotion feature and the encoded first fusion feature can be decoded using a first decoder to obtain a first video.

[0124] The present disclosure provides a video generation method, which can be executed by a processor of a computer device. FIG3 is a schematic diagram of an implementation flow of a video generation method provided by the present disclosure. As shown in FIG3 , the method may include the following steps S301 to S305:

[0125] Step S301: Detect the face of a first person in a first video to obtain facial key points corresponding to at least one first video frame in the first video.

[0126] Step S302 : for each of the first video frames, determine the mouth area corresponding to the first video frame based on the facial key points corresponding to the first video frame.

[0127] Step S303: Mask the mouth area corresponding to each of the first video frames to obtain a third video.

[0128] Here, a pre-trained face detection network may be used to detect the face of the first person in at least one first video frame of the first video, and obtain facial key points respectively contained in each first video frame.

[0129] Facial key points may include, but are not limited to, at least one of mouth key points, eye key points, nose key points, and eyebrow key points. It will be appreciated that, for each first video frame, various regions of the face of the first person in the first video frame may be determined based on the facial key points included in the first video frame. For example, the mouth region of the first person in the first video frame may be determined based on the mouth key points included in the first video frame; and the eye region of the first person in the first video frame may be determined based on the eye key points included in the first video frame.

[0130] In some embodiments, for each first video frame in the first video, the mouth area of ​​the first character in the first video frame can be segmented according to the corresponding mouth key points in the first video frame to obtain the mouth area and non-mouth area corresponding to the first video frame.

[0131] After determining the mouth region corresponding to each first video frame, mask processing can be performed on the mouth region corresponding to each first video frame to obtain the masked first video frames, thereby obtaining a third video including the masked first video frames.

[0132] Step S304: Acquire a first voice feature and a first head feature of the first person; wherein the first head feature is obtained based on the third video, and the first voice feature is obtained based on the first audio.

[0133] Here, step S304 corresponds to step S101 in the aforementioned embodiment, and the implementation may refer to the aforementioned implementation of step S101.

[0134] Step S305: Generate a second video containing the first person based on the first head feature and the first voice feature, wherein the speaking action of the first person in the second video matches the first audio.

[0135] Here, steps S304 to S305 correspond to steps S101 to S102 in the aforementioned embodiment, respectively. When implementing, reference may be made to the implementation of the aforementioned steps S101 to S102.

[0136] In the disclosed embodiment, the face of a first person in a first video is detected to obtain facial key points corresponding to at least one first video frame in the first video; for each first video frame, the mouth region corresponding to the first video frame is determined based on the facial key points corresponding to the first video frame; and the mouth region corresponding to each first video frame is masked to obtain a third video. In this way, the mouth region of the first person in the first video can be masked quickly and accurately to obtain the third video.

[0137] In some embodiments, the method further includes the following step S311:

[0138] Step S311: Determine a first reference video frame from the first video.

[0139] Here, the first reference video frame can be determined from the first video using any suitable method, and the embodiments of the present disclosure are not limited thereto. For example, the first video frame with the highest image quality in the first video can be determined as the first reference video frame. For another example, the first first video frame in the first video can be determined as the first reference video frame. For another example, a video frame can be randomly selected from the first video as the first reference video frame.

[0140] The above step S305 may include the following step S312:

[0141] Step S312: Generate a second video containing the first person based on the first head feature, the first identity feature of the first person, and the first voice feature; wherein the first identity feature is extracted from the first reference video frame.

[0142] Here, the identity information of the first person in the first reference video frame can be identified to obtain the first identity feature of the first person. A video frame sequence containing the first person can be rendered based on the first head feature, the first identity feature, and the first voice feature to obtain a third video.

[0143] During implementation, a pre-trained rendering generation network can be used to render and generate the second video based on the first head feature, the first identity feature, and the first voice feature.

[0144] In the above embodiment, a first reference video frame is determined from the first video, and the first identity feature of the first person is extracted from the first reference video frame; then, based on the first head feature, the first identity feature, and the first voice feature, a second video containing the first person is generated. In this way, the first identity feature of the first person is comprehensively considered during the generation of the second video, thereby better maintaining the overall visual features of the first person in the second video and the identity consistency of the first person across all video frames. This makes the video generation method more suitable for generating second videos corresponding to different people.

[0145] In some embodiments, the facial key points include a first key point representing the upper lip and a second key point representing the lower lip. The above step S311 may include the following steps S321 to S322:

[0146] Step S321 : For each of the first video frames, determine the degree of closure of the mouth in the first video frame based on the first key point and the second key point corresponding to the first video frame.

[0147] Here, the distance between the upper lip and the lower lip of the first person in the first video frame can be determined based on the distance between the first key point and the second key point corresponding to the first video frame, thereby determining the degree of mouth closure in the first video frame. It is understood that a smaller distance between the upper lip and the lower lip indicates a greater degree of mouth closure, and a larger distance between the upper lip and the lower lip indicates a lesser degree of mouth closure.

[0148] In some embodiments, the number of first key points is one and the number of second key points is also one, and the degree of mouth closure in the first video frame can be determined based on the distance between a first key point and a second key point corresponding to the first video frame. The degree of mouth closure in the first video frame is inversely proportional to the distance between a first key point and a second key point corresponding to the first video frame. For example, the reciprocal of the distance between a first key point and a second key point corresponding to the first video frame can be determined as the degree of mouth closure in the first video frame.

[0149] In some embodiments, there are multiple first key points and multiple second key points, and there is a one-to-one correspondence between the multiple first key points and the multiple second key points. The degree of mouth closure in the first video frame can be determined based on the distances between the multiple first key points and the corresponding second key points in the first video frame. For example, the degree of mouth closure in the first video frame can be determined based on the average value of the distances between the multiple first key points and the corresponding second key points in the first video frame. For another example, the degree of mouth closure in the first video frame can be determined based on the minimum value of the distances between the multiple first key points and the corresponding second key points in the first video frame.

[0150] Step S322: Determine a first reference video frame in which the mouth has the greatest degree of closure from the first video.

[0151] In some implementations, a first video frame in which the mouth in the first video is most closed may be determined as the first reference video frame.

[0152] In some implementations, a first video frame may be selected from a plurality of first video frames in which the mouth closure degree is the greatest in the first video, as the first reference video frame.

[0153] In the above embodiment, for each first video frame, the degree of closure of the mouth in the first video frame is determined based on the first key point and the second key point corresponding to the first video frame. Thus, based on the degree of closure of the mouth in each first video frame, a first reference video frame with the largest degree of mouth closure (i.e., the first person's mouth movement amplitude is smaller) can be determined from the first video, thereby reducing the interference of the mouth movement in the first reference video frame on the identification of identity information. In this way, in the process of extracting the first identity feature of the first person from the first reference video frame, the identification of the person's identity information can be better focused, thereby improving the accuracy of the extracted first identity feature of the first person.

[0154] The present disclosure provides a model training method, which can be executed by a processor of a computer device. FIG4 is a schematic diagram of an implementation flow of a model training method provided by the present disclosure. As shown in FIG4 , the method may include the following steps S401 to S404:

[0155] Step S401, obtaining first voice training data and first head training data of a second person; wherein the first head training data is obtained by masking the mouth area of ​​the second person in the training video, and the first voice training data is obtained based on the training audio.

[0156] Here, the training audio and training video are predetermined audio samples and video samples, respectively, used for model training. The training video may include at least one video frame containing a second person. The training audio may include any suitable speech content. During implementation, those skilled in the art may use any suitable training audio and training video according to actual circumstances, and the embodiments of this disclosure are not limited thereto. For example, the training audio and training video may be pre-collected in a real scene, or may be synthesized based on a set text, context, etc.

[0157] The first speech training data may be obtained based on the training audio. In some embodiments, the first speech training data may include the training audio. In some embodiments, the first speech training data may include the second speech feature extracted from the training audio.

[0158] The first head training data may be obtained by masking the mouth area of ​​the second person in the training video. In some embodiments, the first head training data may include a sixth video obtained by masking the mouth area of ​​the second person in the training video. In some embodiments, the first head training data may include second head features extracted from the sixth video. During implementation, the mouth area of ​​the second person in the training video may be masked to obtain the sixth video, referring to the method of masking the mouth area of ​​the first person in the first video in the aforementioned embodiment.

[0159] Step S402: input the first voice training data and the first head training data into the model to be trained to obtain a fifth video containing the second person.

[0160] Step S403: Determine a target loss using the fifth video, and adjust parameters of the model to be trained based on the target loss.

[0161] Step S404: When the speaking action of the second character in the fifth video matches the training audio, it is determined that the training of the model to be trained is completed.

[0162] Here, the target loss can characterize the matching of the speaking action of the second task in the fifth video with the training audio. During implementation, those skilled in the art can determine the target loss based on the fifth video in an appropriate manner according to actual circumstances, and the embodiments of the present disclosure are not limited to this. For example, the first loss can be determined based on the fifth video and a reference video corresponding to the training audio. The reference video can be a pre-set video label corresponding to the training audio. The speaking action of the second character in the reference video matches the training audio.

[0163] If the target loss characterizes the speaking action of the second task in the fifth video and the training audio matches, the parameters of the model to be trained can be discontinued, and the training of the model to be trained is determined to be complete. If the target loss characterizes the speaking action of the second task in the fifth video and the training audio does not match, the parameters of the model to be trained can be continued, and the training of the model to be trained is determined to be incomplete.

[0164] In some embodiments, if the target loss does not meet the first preset condition, it can be determined that the speaking action of the second task in the fifth video does not match the training audio. If the target loss meets the first preset condition or the number of parameter adjustments to the training model reaches a set threshold, it can be determined that the speaking action of the second task in the fifth video matches the training audio. The first preset condition may include, but is not limited to, at least one of the following: the target loss is less than a set loss threshold, the change in the target loss converges, etc. During implementation, the first preset condition can be set according to actual conditions, and the embodiments of the present disclosure are not limited to this.

[0165] The method of adjusting the parameters of the training model based on the target loss can be determined according to the actual situation, and may include but is not limited to at least one of the gradient descent method, Newton momentum method, etc., which is not limited here.

[0166] In the embodiment of the present disclosure, during the training of the model to be trained, on the one hand, since the first head training data is obtained by masking the mouth area of ​​the second person in the training video, the first head training data is not affected by the mouth area and is more reasonable and accurate, and the first voice training data is obtained based on the training audio and can better reflect the voice features in the training audio; on the other hand, since the target loss is determined based on the fifth video, and the training of the model to be trained is determined to be completed when the speaking action of the second person in the fifth video matches the training audio, the parameter adjustment of the model to be trained based on the target loss can improve the ability of the model to be trained to generate the fifth video based on the first head training data and the first voice training data. In this way, the head appearance posture of the character in the video generated by the trained model to be trained can be more accurate, and the head appearance posture can better match the driving audio, so that the overall speaking action of the character is more reasonable and realistic, thereby improving the quality of the generated video.

[0167] In some embodiments, the model to be trained includes a rendering generation network, the first speech training data includes the second speech feature, and the first head training data includes the second head feature. The above step S402 may include the following step S411:

[0168] Step S411: Using the rendering generation network, based on the second head features and the second voice features, generate a fifth video containing the second person.

[0169] Here, a rendering generation network can be used to generate a fifth video based on the second head features and the second voice features, referring to the method of generating the second video based on the first head features and the first voice features in the aforementioned embodiment. The rendering generation network can include but is not limited to an encoder-decoder network, a decoder network, or a generative adversarial network.

[0170] In some embodiments, the model to be trained includes a rendering generation network, and at least one of a first feature extraction network and a second feature extraction network.

[0171] In the case where the model to be trained includes the first feature extraction network but does not include the second feature extraction network, the first head training data includes a sixth video, which is obtained by masking the mouth area of ​​the second person in the training video, and the first voice training data includes the second voice feature. Step S402 above may include the following steps S421a to S421b:

[0172] Step S421a: Using the first feature extraction network, extract the second head feature of the second person from the sixth video.

[0173] Step S421b: Using the rendering generation network, based on the second head features and the second voice features, generate a fifth video containing the second person.

[0174] Here, the first feature extraction network can be used to extract the second head feature of the second person from the sixth video, referring to the method of extracting the first head feature of the first person from the third video in the aforementioned embodiment. The first feature extraction network may include but is not limited to at least one of CNN, RNN, Transformer encoder, etc. For example, the first feature extraction network may use a VGG network (such as VGG16, VGG19, etc.) as the base network.

[0175] In the case where the model to be trained includes the second feature extraction network but does not include the first feature extraction network, the first head training data includes the second head features, and the first voice training data includes the training audio. The above step S402 may include the following steps S422a to S422b:

[0176] Step S422a: Use the second feature extraction network to extract features from the training audio to obtain second speech features.

[0177] Step S422b: Using the rendering generation network, based on the second head features and the second voice features, generate a fifth video containing the second person.

[0178] Here, a second feature extraction network can be used to extract features from the training audio, referring to the method for extracting features from the first audio in the aforementioned embodiment, to obtain second speech features. The second feature extraction network can include, but is not limited to, at least one of a CNN, an RNN, a Transformer encoder, and the like.

[0179] In the case where the model to be trained includes a first feature extraction network and a second feature extraction network, the first head training data includes the sixth video, and the first voice training data includes the training audio. The above step S402 may include the following steps S423a to S423c:

[0180] Step S423a: Using the first feature extraction network, extract the second head feature of the second person from the sixth video.

[0181] Step S423b: Use the second feature extraction network to extract features from the training audio to obtain second speech features.

[0182] Step S423c: using the rendering generation network, based on the second head features and the second voice features, generate a fifth video containing the second person.

[0183] It should be noted that both the first feature extraction network and the second feature extraction network can be pre-trained or untrained, and this is not limited here. During parameter adjustment of the training model, the network parameters of all networks in the training model can be adjusted, or only the network parameters of the rendering generation network can be adjusted, or the network parameters of the rendering generation network and either the first feature extraction network or the second feature extraction network can be adjusted.

[0184] In some embodiments, the model to be trained includes a rendering generation network and a first feature extraction network. The step S403 described above of adjusting the parameters of the model to be trained based on the target loss includes the following step S431:

[0185] Step S431: Adjust parameters of the rendering generation network and the first feature extraction network based on the target loss.

[0186] In this way, since the training data required to train the first feature extraction network is usually not too much, the parameters of the first feature extraction network and the rendering generation network can be adjusted together to improve the efficiency of model training and the ability of the trained model to generate videos.

[0187] In some embodiments, when the model to be trained includes a second feature extraction network, the second feature extraction network can be pre-trained to obtain a trained second feature extraction network. Because training the second feature extraction network typically requires a large amount of training data, pre-training the second feature extraction network separately can enable the second feature extraction network to better learn the ability to represent speech features and improve the training efficiency of the second feature extraction network, thereby improving the overall efficiency and effectiveness of model training.

[0188] In some embodiments, the first feature extraction network includes a first encoder and a mapping subnetwork, and the second head features include second non-mouth region features and second head motion features. The process of extracting the second head features of the second person from the sixth video using the first feature extraction network may include the following steps S441 to S442:

[0189] Step S441: Use the first encoder to encode the sixth video to obtain the second non-mouth area feature.

[0190] Step S442: Map the second non-mouth area feature to a preset motion feature space using the mapping subnetwork to obtain the second head motion feature.

[0191] Here, steps S411 to S412 correspond to steps S111 to S112 in the aforementioned embodiment, respectively. When implementing, reference may be made to the implementation of the aforementioned steps S111 to S112.

[0192] In the above embodiment, the first encoder and the mapping sub-network are used to model the visual features of the non-mouth area of ​​the second character and the head movement of the second character respectively, which can simplify the network structure of the first encoder and the mapping sub-network, and enable the second non-mouth area features extracted by the trained first encoder to better characterize the visual features of the non-mouth area of ​​the second character, and enable the second head movement features extracted by the trained mapping sub-network to better characterize the head movement of the second character, thereby further improving the rationality and realism of the overall speaking action of the second character in the fifth video generated by the trained model, and improving the quality of the generated fifth video.

[0193] In some embodiments, the process of generating a fifth video containing the second person based on the second head feature and the second voice feature using the rendering generation network may include the following steps S451 to S452:

[0194] Step S451: Fuse the second head feature and the second voice feature to obtain a second fused feature.

[0195] Here, the second head feature and the second voice feature may be fused with reference to the method of fusing the first head feature and the first voice feature in the aforementioned embodiment to obtain a second fused feature.

[0196] Step S452: Generate the fifth video based on the second fusion feature by using the rendering generation network.

[0197] Here, the rendering generation network can be used to generate the fifth video based on the second fusion feature with reference to the method of generating the second video based on the first fusion feature in the aforementioned embodiment.

[0198] In the embodiment of the present disclosure, the second head feature and the second voice feature are first fused to obtain a second fused feature; then, a rendering generation network is used to generate a more accurate fifth video based on the second fused feature, so that the speaking action of the second person in the fifth video can better match the second audio.

[0199] In some embodiments, the model to be trained includes a rendering generation network and a second feature extraction network, the second feature extraction network includes a second encoder and a third encoder, and the second speech feature includes a second content feature and a second emotion feature.

[0200] The process of extracting features from the training audio using the second feature extraction network to obtain the second speech features may include the following steps S461 to S462:

[0201] Step S461: Use the second encoder to encode the training audio to obtain the second content feature.

[0202] Step S462: Utilize the third encoder to encode the training audio to obtain the second emotion feature.

[0203] Here, the second encoder and the third encoder in the second feature extraction network can be used to encode the training audio respectively to obtain a second content feature representing the speech content and a second emotion feature representing the emotion contained in the speech.

[0204] During implementation, the second encoder and the third encoder may be pre-trained in any suitable manner.

[0205] In some implementations, a cross-reconstruction decoupling network may be used to train the second encoder and the third encoder to obtain trained second encoder and third encoder.

[0206] The above step S451 may include the following step S463:

[0207] Step S463: Fusing the second header feature and the second content feature to obtain a second fused feature.

[0208] Here, the second header feature and the second content feature may be fused to obtain a second fused feature by referring to the method of fusing the first header feature and the first content feature in the aforementioned embodiment.

[0209] The above step S452 may include the following step S464:

[0210] Step S464: Generate the fifth video using the rendering generation network based on the second fusion feature and the second emotion feature.

[0211] Here, the rendering generation network can be used to generate a fifth video based on the second fusion feature and the second emotion feature, referring to the method of generating the second video in the aforementioned embodiment.

[0212] In the above embodiment, the second encoder and the third encoder are used to extract the second content feature and the second emotion feature from the training audio respectively; the second head feature and the second content feature are fused to obtain the second fusion feature, and the fifth video is generated based on the second fusion feature and the second emotion feature. In this way, the content feature and the emotion feature in the training audio can be decoupled, so that while the second content feature drives the generation of the mouth movement, the second character's facial expression is adaptively adjusted in combination with the second emotion feature, so that the head appearance and posture of the second character in the generated fifth video can be better matched with the speech content and emotion in the training audio. In this way, the rationality and realism of the overall speaking action of the second character in the fifth video generated by the trained model can be further improved, thereby improving the quality of the generated fifth video.

[0213] In some embodiments, the rendering generation network includes a fourth encoder and a first decoder; the above step S464 may include the following steps S471 to S472:

[0214] Step S471: Use the fourth encoder to encode the second fusion feature to obtain the encoded second fusion feature.

[0215] Step S472: Using the first decoder, decode the second emotion feature and the encoded second fusion feature to obtain the fifth video.

[0216] During implementation, steps S471 to S472 correspond to steps S221 to S222 in the aforementioned embodiment, respectively. During implementation, reference may be made to the implementation of the aforementioned steps S221 to S222.

[0217] In some embodiments, the rendering generation network may employ a U-type network (Unet), the fourth encoder may be an encoder in the Unet network, and the first decoder may be a decoder in the Unet network. The input of each decoder in the Unet network may include the second emotion feature, the encoded second fusion feature, and the feature output by the encoder corresponding to the decoder in the Unet network.

[0218] In some embodiments, the model further includes a face detection network. The method further includes the following steps S481 to S482:

[0219] Step S481: Use the face detection network to detect the second person's face in the training video, obtain the facial key points corresponding to at least one second video frame in the training video, and for each second video frame, determine the mouth area corresponding to the second video frame based on the facial key points corresponding to the second video frame.

[0220] Here, the face detection network can be pre-trained in an appropriate manner according to actual conditions.

[0221] During implementation, a face detection network can be used to detect the face of the first person in the first video, and determine the mouth area corresponding to at least one first video frame in the first video, by referring to the method in the aforementioned embodiment. The faces in the training video can be detected to determine the mouth area corresponding to at least one second video frame in the training video.

[0222] Step S482: Mask the mouth area corresponding to each second video frame to obtain the sixth video.

[0223] Here, the mask processing may be performed on the mouth region corresponding to each first video frame in reference to the manner of performing the mask processing on the mouth region corresponding to each first video frame in the aforementioned embodiment.

[0224] In the above embodiment, a face detection network is used to detect the face of the second person in the training video to obtain facial key points corresponding to at least one second video frame in the training video. For each second video frame, the mouth region corresponding to the second video frame is determined based on the facial key points corresponding to the second video frame. The mouth region corresponding to each second video frame is masked to obtain the sixth video. In this way, the mouth region of the second person in the training video can be quickly and accurately masked to obtain the sixth video.

[0225] In some embodiments, the model further includes a third feature extraction network; and the method further includes the following steps S491 to S492:

[0226] Step S491: Determine a second reference video frame from the training video.

[0227] Step S492: Using the third feature extraction network, extract the second identity feature of the second person from the second reference video frame.

[0228] Here, the third feature extraction network can be determined by those skilled in the art based on actual conditions, and the embodiments of the present disclosure do not limit this. In implementation, the third feature extraction network may include, but is not limited to, at least one of a CNN, an RNN, a Transformer encoder, and the like. For example, the third feature extraction network may use a VGG network (such as VGG16, VGG19, etc.) as the base network.

[0229] The process of generating the fifth video containing the second person based on the second head feature and the second voice feature by using the rendering generation network may include the following step S493:

[0230] Step S493: Using the rendering generation network, a fifth video containing the second person is generated based on the second head feature, the second identity feature, and the second voice feature.

[0231] Here, step S491 and step S493 correspond to step S311 and step S312 in the aforementioned embodiment, respectively. When implementing, reference may be made to the implementation of the aforementioned step S311 and step S312.

[0232] In some embodiments, the parameter adjustment of the model to be trained based on the target loss described in the above step S403 may include: adjusting the network parameters of the first feature extraction network, the network parameters of the third feature extraction network, and the network parameters of the rendering generation network based on the target loss.

[0233] Here, the method of adjusting the network parameters of the third feature extraction network can be determined according to actual conditions, and can include but is not limited to at least one of the gradient descent method, Newton momentum method, etc., and the embodiment of the present disclosure is not limited.

[0234] In the above embodiment, a second reference video frame is determined from the training video, and the second identity features of the second person are extracted from the second reference video frame; then, based on the second head features, the second identity features, and the second voice features, a fifth video containing the second person is generated. In this way, the second identity features of the second person are comprehensively considered during the generation of the fifth video, thereby better maintaining the overall visual features of the second person in the fifth video and the consistency of the second person's identity across all video frames. This, in turn, makes the trained model more suitable for generating videos of different speaking people.

[0235] The present disclosure provides a model training method, which can be executed by a processor of a computer device. FIG5 is a schematic diagram of an implementation flow of a model training method provided by the present disclosure. As shown in FIG5 , the method may include the following steps S501 to S508:

[0236] Step S501, obtaining first voice training data and first head training data of a second person; wherein the first head training data is obtained by masking the mouth area of ​​the second person in the training video, and the first voice training data is obtained based on the training audio.

[0237] Step S502: input the first voice training data and the first head training data into the model to be trained to obtain a fifth video containing the second person.

[0238] Here, the above steps S501 to S502 correspond to steps S401 to S402 in the aforementioned embodiment, respectively. When implementing, reference may be made to the implementation of the aforementioned steps S401 to S402.

[0239] Step S503: Determine a first loss based on the fifth video and a reference video corresponding to the training audio.

[0240] Here, the reference video may be a pre-set video tag corresponding to the training audio, and the speaking action of the second person in the reference video matches the training audio.

[0241] The first loss represents the loss between the fifth video and the reference video. During implementation, any suitable loss function may be used to determine the first loss, and the embodiment of the present disclosure is not limited thereto.

[0242] It is understandable that the loss functions used in step S503 and other steps below may include but are not limited to at least one of an absolute value loss function, a minimum square error loss function, a cosine loss function, a cross entropy loss function, and the like.

[0243] In some embodiments, the following formula 1-1 can be used to adopt the L1 reconstruction loss function to determine the first loss L recon :

[0244] Among them, L gi is the third video frame i in the generated fifth video, L ri is the i-th fourth video frame in the reference video, and K is the total number of video frames in the fifth video or the reference video.

[0245] Step S504 : determining a mouth region corresponding to at least one second video frame in the training video, and determining a mouth region corresponding to at least one third video frame in the fifth video.

[0246] Here, we can refer to steps S301 to S302 in the aforementioned embodiment to detect the face in the first video, determine the mouth area corresponding to at least one first video frame in the first video, determine the mouth area corresponding to at least one second video frame in the training video, and determine the mouth area corresponding to at least one third video frame in the fifth video.

[0247] Step S505 : determining a second loss based on the mouth region corresponding to each of the second video frames and the mouth region corresponding to each of the third video frames.

[0248] Here, the second loss represents the loss between the mouth regions corresponding to the third video frames and the mouth regions corresponding to the second video frames. In implementation, any suitable loss function may be used to determine the second loss, and the present disclosure is not limited thereto.

[0249] In some embodiments, the following formula 1-2 can be used to adopt the L1 reconstruction loss function to determine the second loss L m-recon :

[0250] Among them, M gi is the mouth region in the i-th third video frame in the generated fifth video, M ri is the mouth region in the i-th second video frame in the training video.

[0251] Step S506: Determine a target loss based on the first loss and the second loss.

[0252] It can be understood that the target loss can be in direct proportion to the first loss and the second loss. In this way, the first loss between the fifth video generated by the trained model and the reference video, as well as the second loss between the mouth areas corresponding to each third video frame in the fifth video and the mouth areas corresponding to each second video frame in the training video can be minimized, thereby improving the overall consistency between the fifth video generated by the model and the reference video, as well as the consistency of the mouth areas in the fifth video and the training video.

[0253] In some implementations, the sum of the first loss and the second loss may be determined as the target loss.

[0254] In some implementations, different weights may be set for the first loss and the second loss, respectively, and the first loss and the second loss are weightedly summed to obtain the target loss.

[0255] Step S507: Adjust parameters of the model to be trained based on the target loss.

[0256] Step S508: When the speaking action of the second character in the fifth video matches the training audio, it is determined that the training of the model to be trained is completed.

[0257] Here, the above steps S507 to S508 correspond to steps S403 to S404 in the above embodiment respectively, and the implementation of the above steps S403 to S404 may be referred to during implementation.

[0258] In the embodiment of the present disclosure, in the process of determining the target loss, the loss between the fifth video and the reference video corresponding to the training audio, as well as the loss between the mouth areas corresponding to each third video frame in the fifth video and the mouth areas corresponding to each second video frame in the training video are comprehensively considered. Therefore, during the training process, the overall consistency between the fifth video generated by the model and the reference video, as well as the consistency of the mouth areas in the fifth video and the training video are constrained, thereby improving the synchronization between the mouth movement of the second character in the fifth video generated by the trained model and the training audio, that is, the lip synchronization of the character speaking in the video generated by the trained model can be improved.

[0259] In some embodiments, the above method may further include the following steps S511 to S513:

[0260] Step S511 : extracting mutually matching video segments and audio segments from the fifth video and the training audio respectively.

[0261] Step S512: Encode the video clip and the audio clip respectively to obtain a first encoding feature and a second encoding feature.

[0262] Here, matching video and audio clips refer to those corresponding to the same time interval. For example, a video clip can be first determined from the fifth video. Then, based on the time interval corresponding to the video clip, an audio clip corresponding to the time interval can be determined from the training audio, thereby obtaining a pair of matching video and audio clips.

[0263] In some embodiments, a pre-trained feature alignment network (such as a SyncNet network) can be used to extract matching video and audio segments from the fifth video and training audio, respectively, and encode the extracted video and audio segments to obtain first and second encoding features. During implementation, the feature alignment network can include a video encoder and an audio encoder constructed based on a VGG network. The video encoder can be used to encode the video segment to obtain the first encoding feature. The audio encoder can be used to encode the audio segment corresponding to the time interval of the video segment to obtain the second encoding feature.

[0264] Step S513: Obtain a third loss based on the first encoding feature and the second encoding feature.

[0265] Here, the third loss represents the loss between the first coding feature and the second coding feature. During implementation, any suitable loss function may be used to determine the third loss, and the present disclosure is not limited thereto.

[0266] In some embodiments, a similarity between the first coding feature and the second coding feature is determined, and based on the similarity, a third loss between the first coding feature and the second coding feature is determined.

[0267] For example, the similarity P between the first coding feature and the second coding feature can be determined by the following formula 1-3: sync :

[0268] Among them, v and s are the first coding feature and the second coding feature respectively, and the similarity P sync is a value between 0 and 1, and ε is a very small value used to prevent the denominator in the formula from being 0. For example, ε can be 10 -6 .

[0269] Based on the similarity P between the first coding feature and the second coding feature sync , the third loss E between the first coding feature and the second coding feature can be determined by the following formula 1-4: sync :

[0270] in, is the similarity between the first encoding sub-feature corresponding to the i-th video frame in the first encoding feature and the second encoding sub-feature corresponding to the i-th audio frame in the second encoding feature.

[0271] The above step S506 may include the following step S514:

[0272] Step S514: determining the target loss based on the first loss, the second loss, and the third loss.

[0273] It can be understood that the target loss and the third loss can be in direct proportion to each other. In this way, the third loss between the fifth video generated by the trained model and the training audio can be minimized, thereby improving the lip synchronization of the characters speaking in the video generated by the trained model.

[0274] In some implementations, the sum of the first loss, the second loss, and the third loss may be determined as the target loss.

[0275] In some implementations, different weights may be set for the first loss, the second loss, and the third loss, respectively, and the first loss, the second loss, and the third loss are weighted and summed to obtain the target loss.

[0276] In the above embodiment, in the process of determining the target loss, the first loss between the fifth video and the reference video corresponding to the training audio, the second loss between the mouth areas corresponding to each third video frame in the fifth video and the mouth areas corresponding to each second video frame in the training video, and the third loss characterizing the synchronization between the audio segment in the training audio and the video segment in the fifth video are comprehensively considered. Therefore, during the training process, the overall consistency between the fifth video generated by the model and the reference video, the consistency of the mouth area between the fifth video and the training video, and the lip synchronization between the fifth video and the training audio are constrained, thereby further improving the lip synchronization of the characters speaking in the video generated by the trained model.

[0277] In some embodiments, the above method may further include the following steps S521 to S522:

[0278] Step S521 : Using a first discriminator, determine the authenticity of at least one fourth video frame in the reference video and the authenticity of at least one third video frame in the fifth video.

[0279] Here, the first discriminator may be pre-trained, or updated and trained during the training process of the first feature extraction network and the rendering generation network, which is not limited in the embodiment of the present disclosure.

[0280] The first discriminator can be used to discriminate the overall quality of the fourth video frame and the third video frame to determine the authenticity of the corresponding video frames.

[0281] Step S522 : determining a fourth loss based on the authenticity of each of the fourth video frames and the authenticity of each of the third video frames.

[0282] Here, the fourth loss represents the authenticity of each fourth video frame in the reference video and the authenticity of each third video frame in the fifth video. In implementation, any suitable loss function can be used to determine the fourth loss, and the embodiment of the present disclosure is not limited thereto.

[0283] In some embodiments, the fourth loss L can be determined by the following formula 1-5: disc :

[0284] Among them, L g is a video frame set including each third video frame in the generated fifth video, L ris a set of video frames containing each fourth video frame in the reference video, x2 is a fourth video frame in the reference video, x2 is a third video frame in the fifth video, D1(-) is the first discriminator, D1(x1) is the authenticity of the fourth video frame x1, D1(x2) is the authenticity of the third video frame x2, and E[-] is the mean function.

[0285] The above step S514 may include the following step S523:

[0286] Step S523: Determine the target loss based on the first loss, the second loss, the third loss, and the fourth loss.

[0287] It can be understood that the target loss and the fourth loss can be inversely proportional to each other. In this way, the fourth loss corresponding to the fifth video generated by the trained model can be maximized, that is, the authenticity of each fourth video frame in the reference video and the authenticity of each third video frame in the fifth video can be maximized, thereby improving the overall quality of each video frame in the video generated by the trained model.

[0288] In some embodiments, the target loss may be obtained by subtracting the fourth loss from the sum of the first loss, the second loss, and the third loss.

[0289] In some embodiments, different weights can be assigned to the first, second, third, and fourth losses, and the target loss can be obtained by weighted summing the first, second, third, and fourth losses. In implementation, the weights corresponding to the first, second, and third losses can be positive numbers, while the weight of the fourth loss can be negative.

[0290] In the above embodiment, the fourth loss characterizing the authenticity of each fourth video frame in the reference video and the authenticity of each third video frame in the fifth video is taken into account in the process of determining the target loss, so that the overall quality of each video frame in the video generated by the model can be constrained during the training process, thereby further improving the overall quality of the video generated by the trained model.

[0291] In some embodiments, the above method may further include the following steps S531 to S533:

[0292] Step S531 : for each pair of adjacent fourth video frames in the reference video, using a second discriminator, determining a true degree of proximity between the pair of adjacent fourth video frames.

[0293] Step S532: For each pair of adjacent third video frames in the fifth video, use a second discriminator to determine the true proximity between the pair of adjacent third video frames.

[0294] Here, the second discriminator may be pre-trained, or updated and trained during the training process of the first feature extraction network and the rendering generation network, which is not limited in the embodiment of the present disclosure.

[0295] The true proximity between a pair of adjacent video frames can represent the degree of true proximity between the pair of video frames. The second discriminator can be used to discriminate the true proximity between two adjacent video frames to obtain the true proximity between the two video frames.

[0296] Step S533 : determining a fifth loss based on the true proximity between each pair of adjacent fourth video frames and the true proximity between each pair of adjacent third video frames.

[0297] Here, the fifth loss represents the degree to which each pair of adjacent fourth video frames in the reference video is truly adjacent and the degree to which each pair of adjacent third video frames in the fifth video is truly adjacent. In implementation, any suitable loss function may be used to determine the fifth loss, and the present disclosure is not limited thereto.

[0298] In some embodiments, the fifth loss L can be determined by the following formula 1-6: disc_c :

[0299] Among them, L gc is a set of video frame pairs containing each pair of adjacent third video frames in the generated fifth video, L rc is a set of video frame pairs containing each pair of fourth video frames in the reference video, x c1 is a pair of fourth video frames in the reference video, x c2 is a pair of third video frames in the fifth video, D2(-) is the second discriminator, D2(x c1 ) is a pair of fourth video frames x c1 The true proximity of D2(x c2 ) is a pair of third video frames x c2 The true proximity of , E[-] is the mean function.

[0300] The above step S523 may include the following step S534:

[0301] Step S534: determining the target loss based on the first loss, the second loss, the third loss, the fourth loss, and the fifth loss.

[0302] It can be understood that the target loss and the fifth loss can be inversely proportional to each other. In this way, the fifth loss corresponding to the fifth video generated by the trained model can be maximized, that is, the true proximity between each adjacent fourth video frame in the reference video and the true proximity between each adjacent third video frame in the fifth video can be maximized, thereby improving the inter-frame consistency between each video frame in the video generated by the trained model.

[0303] In some embodiments, the target loss may be obtained by subtracting the fourth loss and the fifth loss from the sum of the first loss, the second loss, and the third loss.

[0304] In some embodiments, different weights may be assigned to the first, second, third, fourth, and fifth losses, and the target loss may be obtained by weighted summing the first, second, third, fourth, and fifth losses. In implementation, the weights corresponding to the first, second, and third losses may be positive numbers, while the weights corresponding to the fourth and fifth losses may be negative numbers.

[0305] For example, the first loss L can be calculated as shown in the following formula 1-7: recon , the second loss L m-recon 、Third loss E sync , the fourth loss L disc and the fifth loss L disc_c Perform weighted summation to obtain the target loss L total : L total =λ1*L recon +λ2*L m-recon +λ3*E sync +λ4*L disc +λ5*L disc_c (1-7);

[0306] Among them, λ1, λ2, λ3, λ4, and λ5 are the first loss L recon , the second loss L m-recon 、Third loss E sync , the fourth loss L disc and the fifth loss L disc_c The weights of λ1, λ2, and λ3 are all greater than 0, and λ4 and λ5 are all less than 0.

[0307] In the above embodiment, in the process of determining the target loss, the fifth loss that characterizes the true proximity between each pair of adjacent fourth video frames in the reference video and the true proximity between each pair of adjacent third video frames in the fifth video is taken into account. This allows the inter-frame consistency between each video frame in the video generated by the model to be constrained during the training process, thereby enabling the video generated by the trained model to have better temporal stability and reduce inter-frame fluctuations, thereby further improving the overall quality of the video generated by the trained model.

[0308] It should be noted that, in some embodiments, those skilled in the art may determine the target loss based on at least one of the first loss, the second loss, the third loss, the fourth loss and the fifth loss according to actual application scenarios, and the embodiments of the present disclosure are not limited to this.

[0309] The following describes the application of the video generation method and model training method provided by the embodiments of the present disclosure in actual scenarios, taking the voice-driven speaker synthesis task as an example.

[0310] Given one or more target facial images or videos of a target person and a speech data source (such as a pre-set first audio source) to drive speech, the speech-driven speaker synthesis task uses the speech data source to drive the target person's facial movements (including lip movements and expressions) to produce a realistic video of the target person speaking. This speech-driven speaker synthesis task has a wide range of applications, such as dubbing videos in different languages, telepresence in video conferencing, character generation in video games, video transmission with limited bandwidth, enhancing speech understanding while protecting privacy, or assistive devices for the hearing-impaired. Furthermore, speech-driven speaker synthesis can promote the development of security countermeasures by providing a large amount of training simulation data. Another highly promising application scenario is the digital human field, where the transition from offline to online and then to presence is a key path for the development of digital humans. From identity-based digital humans focused on animation and rendering to service-oriented digital humans that integrate language understanding, expression, learning, and interaction capabilities, and from online services to on-site experiences, digital humans will enter a period of large-scale application, accelerating their transformation into real-world productivity and creating greater value for the consumer market and enterprise services. Two-dimensional digital humans driven by speech can play a key role in expression and interaction.

[0311] During the implementation of the present disclosure, the inventors discovered that the voice-driven speaker video generation solution and model training solution in the related art have at least the following problems:

[0312] 1) The modeling of the person's external posture outside the mouth area is not considered, and head movements and facial expressions that match the speech content are neglected. In other words, the modeling of the person's head posture outside the mouth area is neglected, resulting in the target person's head movement and expression in the generated video being unreasonable and unrealistic. The external posture of the target person's head (corresponding to the first or second person mentioned above) is very complex, including but not limited to its internal body features, external camera position, head movement, and facial expression. This complexity comes not only from modeling the facial area, but also from modeling head movement and background.

[0313] 2) Insufficient utilization of visual information. The process for processing visual information in the reference video (corresponding to the first or fifth video mentioned above) is limited to multi-scale visual representation, and high-level abstract representations are not fully decoupled. For example, the pose and movement information of the characters are not fully considered from their identity information.

[0314] 3) The output of the video generation model used is usually a sequence of video frames. There is no explicit inter-frame consistency constraint during the model training process, which can easily cause subtle artifacts in the generated video and subtle changes in the target person in the generated video, making it impossible to maintain the consistency of the target person's identity.

[0315] In view of this, the embodiments of the present disclosure provide a video generation method and a model training method, which can improve the rationality and realism of the overall speaking movements of the characters in the generated video, thereby improving the quality of the generated video.

[0316] FIG6A is a schematic diagram illustrating a video generation method according to an embodiment of the present disclosure. As shown in FIG6A , the video generation method can be implemented using a face detection network 10, a first feature extraction network 20, a second feature extraction network 30, a third feature extraction network 40, and a rendering generation network 50 in a pre-trained model. The rendering generation network 50 includes a fourth encoder and a first decoder. The video generation method according to an embodiment of the present disclosure will be described below with reference to FIG6A .

[0317] The video generation method provided by the embodiment of the present disclosure includes the following steps S601 to S609:

[0318] Step S601, obtaining a first audio A and a first video V1 including a first person;

[0319] Step S602: Detect the face in the first video V1 using the pre-trained face detection network 10 to obtain the facial key points corresponding to at least one first video frame in the first video V1, and determine the mouth area M corresponding to each first video frame based on the facial key points corresponding to each first video frame. i, and a third video V3 consisting of the first video frames in which the mouth regions are masked;

[0320] Step S603: Determine the first reference video frame I from the first video V1. ref ;

[0321] Wherein, for each first video frame in the first video V1, based on the distance between the first key point representing the upper lip and the second key point representing the lower lip corresponding to the first video frame, the first reference video frame I can be determined from the first video V1. ref For example, the first video frame having the smallest distance between the first key point and the second key point can be determined as the first reference video frame I. ref .

[0322] Step S604: Encode the third video V3 using the first encoder in the first feature extraction network 20 to obtain a first non-mouth region feature F v , and uses the mapping sub-network in the first feature extraction network 20 to transform the first non-mouth region feature F v Mapping to the preset motion feature space to obtain the first head motion feature H;

[0323] Step S605: Using the second feature extraction network 30, extract features from the first audio A to obtain a first content feature F c and the first emotional feature F e ;

[0324] Step S606: using the third feature extraction network 40, extracting the first reference video frame I ref Extract the first identity feature F of the first person I ;

[0325] Step S607: the first non-mouth region feature F v , first head action feature H, first identity feature F I and the first content feature F c Perform frame-level fusion;

[0326] Step S608: using the rendering generation network 50, based on the frame-level fused features and the first emotion feature F e , rendering to generate the fourth video V4;

[0327] Among them, the frame-level fused features can be used as the input of the fourth encoder, and the output of the fourth encoder and the first emotion feature F e The first visual feature F0 and the first visual feature F0 can be used as the input of the first decoder, and the output of the first decoder is the fourth video V4.

[0328] Step S609: Poisson fusion is performed on the face area and the background area in at least one fourth video frame in the fourth video V4 to obtain a second video V2, wherein the speaking action of the first person in the second video V2 matches the first audio A.

[0329] It should be noted that the first feature extraction network 20 and the rendering generation network 50 in the above model can be trained using any of the model training methods in the aforementioned embodiments. The target loss used in model training is determined based on the aforementioned first, second, third, fourth, and fifth losses, using the method shown in Formulas 1-7.

[0330] The face detection network 10 and the second feature extraction network 30 in the above model may have been trained before the first feature extraction network 20, the third feature extraction network 40 and the rendering generation network 50 are trained.

[0331] The following describes the process of training the second feature extraction network.

[0332] The second feature extraction network includes a content encoder (corresponding to the aforementioned third encoder) for encoding the speech content in the audio and an emotion encoder (corresponding to the aforementioned fourth encoder) for encoding the emotional components in the audio. Using the trained content encoder and emotion encoder, the speech content and emotional information in the first audio can be decoupled and encoded, respectively, to obtain decoupled content encoding (corresponding to the aforementioned first content feature) and emotion encoding (corresponding to the aforementioned first emotion feature).

[0333] In order to achieve emotional control of audio-based face synthesis, two independent latent audio spaces can be extracted to independently decouple the emotion and content components from the audio. The two independent latent audio spaces include: a duration-independent space, based on which the emotional components in the audio can be encoded to obtain an emotion encoding with unknown content; and a time-dependent space, based on which the speech content of the audio can be encoded to obtain a content encoding with unknown emotion. During implementation, aligned audio samples can be first constructed, and then a cross-reconstruction decoupling network can be used to train a content encoder and an emotion encoder based on the aligned audio sample data to decouple the speech content and emotional information in the first audio.

[0334] During implementation, an audio dataset with various characters speaking the same corpus in different emotional states can be used to train this cross-reconstruction decoupling network to obtain the trained content encoder and emotion encoder. The audio dataset may contain audio samples of different characters speaking the same speech content in different emotional states. Since speeches with the same content but different emotions may have different speaking speeds, a time alignment algorithm can be used to align speeches of uneven length in each audio sample. During implementation, Mel Frequency Cepstrum Coefficients (MFCC) can be used as audio representation, and a Dynamic Time Warping (DTW) algorithm can be used to warp the MFCC feature vector by stretching or shrinking along the time dimension, thereby achieving time length alignment of the audio samples. For example, given two MFCC sequences S with the same content but different lengths a and S b , the DTW algorithm can calculate a set of index coordinate pairs {(i, j), ...} through dynamic programming, forcing S a [i] and S b [j] Similarity, given sequence S a and S b The best match between them can be achieved by taking the following formula 2-1 as the goal, that is, minimizing the sum of the distance costs between the aligned MFCC features: min∑ (i,j)∈P d(S a [i],S b [j]) (2-1);

[0335] Where d is the distance loss and P is the alignment path, which is a set of index coordinate pairs calculated by dynamic programming.

[0336] FIG6B is a schematic diagram of an implementation of a method for training a content encoder and an emotion encoder using a cross-reconstruction decoupling network according to an embodiment of the present disclosure. As shown in FIG6B , the cross-reconstruction decoupling network may include a content encoder E for modeling content information. c , emotion encoder E for modeling emotional information e , a decoder D for decoding the content code and the emotion code into audio, and a classifier C for classifying the emotion code e Given 4 aligned audio samples X i,m ={x i,m}、X j,n ={x j,n}、X j,m ={x j,m}、X i,n ={x i,n}, where X i,mThe content information is i, the emotion information is m, and X j,n The content information is j, the emotional information is n, and X j,m The content information is j, the emotion information is m, and X i,n The content information in is i and the emotion information is n. First, we can use the aligned audio sample X i,m and X j,n As the input of the cross-reconstruction disentanglement network, the content encoder E c and emotion encoder E e Decouple the audio samples X separately i,m Content encoding E c (x i,m ) and emotion coding E e (x i,m ), and audio sample X j,n Content encoding E c (x j,n ) and emotion coding E e (x j,n ); Then, the decoder D can be used to pass the audio sample X i,m Content-Encoding E c (x i,m ) and audio sample X j,n Emotional coding E e (x j,n ) to reconstruct the audio segment X' i,n ={x' i,n}, passing audio sample X j,n Content-Encoding E c (x j,n ) and audio sample X i,m Emotional coding E e (x i,m ) to reconstruct the audio segment X' j,m ={x' j,m}; Finally, we can i,n ={x i,n} and X j,m ={x j,m} to supervise the training of the cross-reconstruction decoupling network. The loss used for supervised training of the cross-reconstruction decoupling network may include: cross-reconstruction loss L cross , self-reconstruction loss L self , classification loss L class and content loss L con .

[0337] For example, the cross reconstruction loss L can be determined as shown in the following formula 2-2: cross : L cross =||D(E c(x i,m ),E e (x j,n ))-x i,n ||2+||D(E c (x j,n ),E e (x i,m ))-x j,m ||2 (2-2).

[0338] In addition, the self-reconstruction loss L can be determined as shown in the following formula 2-3: self : L self =||D(E c (x i,m ),E e (x i,m ))-x i,m ||2+||D(E c (x j,n ),E e (x j,n ))-x j,n ||2 (2-3);

[0339] To drive the emotion encoder E e To accurately classify the emotion category of each audio sample, classifier C can be used e To construct the classification loss. The classification loss L can be determined by the following formula 2-4: class :

[0340] Where N represents the number of emotion categories, p k refers to the audio samples belonging to emotion category k, q k Refers to the classifier C e The predicted category probability distribution of the audio sample.

[0341] For audio content, samples of the same audio corpus should have similar audio content encodings, so content loss is added to train the content encoder E c The content loss L can be determined using the following formula 2-5: con : L con =∑ k∈i,j ||E c (x k,m )-E c (x k,n )||2 (2-5);

[0342] Combining the above four losses, the total training loss of the cross-reconstruction decoupling network can be determined using the following formula 2-6: L dis =Lcross +L self +λ cla *E class +λ con *L con (2-6);

[0343] Among them, λ cla and λ con are the weights of classification loss and content loss, respectively.

[0344] In the embodiments of the present disclosure, on the one hand, by separately modeling the mouth area, head movement and non-mouth area of ​​the character in the first video or the training video, and explicitly using visual information from different perception angles, lip synchronization can be effectively improved, and reasonable head movement and expressions can be generated; on the other hand, by decoupling the posture and movement information and identity information of the character in the first video or the training video, the identity representation and posture and movement of the specific character can be fully maintained, and the model is trained in combination with inter-frame consistency constraints, which can not only reduce the unreasonable jitter of the character in the generated video, but also maintain the identity representation of the specific character; on the other hand, by training the audio information through the cross-reconstruction decoupling network, the content code and emotion code in the audio can be decoupled and encoded, so that the mouth movement can be driven by the audio content, and the facial expression of the character can be adaptively adjusted in combination with the emotion code; on the other hand, the generated fourth video is post-processed through Poisson fusion, which can transition the face area with the body and the background, thereby reducing the obvious artifact intervals in the video frame. It can be seen that according to the embodiments of the present disclosure, a 2D digital human speech synthesis model with high lip synchronization rate and high fidelity of the target person can be obtained.

[0345] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0346] Based on the foregoing embodiments, the embodiments of the present disclosure provide a video generation device, which includes the various units included and the various parts included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0347] FIG7A is a schematic diagram of the composition structure of a video generating device provided by an embodiment of the present disclosure. As shown in FIG7A , the video generating device 710 includes: a first acquisition part 711 and a first generation part 712, wherein:

[0348] The first acquisition section 711 is configured to acquire a first voice feature and a first head feature of a first person; wherein the first head feature is obtained by masking the mouth area of ​​the first person in the first video, and the first voice feature is obtained based on the first audio;

[0349] The first generating part 712 is configured to generate a second video containing the first person based on the first head feature and the first voice feature; the speaking action of the first person in the second video matches the first audio.

[0350] In some embodiments, the first head feature includes a first non-mouth area feature and a first head motion feature; the first non-mouth area feature is obtained by encoding a third video, and the third video is obtained by masking the mouth area of ​​the first person in the first video; the first head motion feature is obtained by mapping the first non-mouth area feature to a preset motion feature space.

[0351] In some embodiments, the first acquisition part is further configured to: fuse the first head feature and the first voice feature to obtain a first fusion feature; and generate the second video based on the first fusion feature.

[0352] In some embodiments, the first speech feature includes a first content feature and a first emotion feature; the first acquisition part is also used to: fuse the first head feature and the first content feature to obtain a first fusion feature; and generate the second video based on the first fusion feature and the first emotion feature.

[0353] In some embodiments, the first acquisition part is further configured to: encode the first fusion feature to obtain the encoded first fusion feature; and decode the first emotion feature and the encoded first fusion feature to obtain the second video.

[0354] In some embodiments, the first head feature is obtained based on a third video, and the device further includes: a first detection part, configured to detect the face of the first person in the first video, and obtain facial key points corresponding to at least one first video frame in the first video; a second determination part, configured to determine, for each of the first video frames, the mouth area corresponding to the first video frame based on the facial key points corresponding to the first video frame; and a first mask part, configured to perform mask processing on the mouth area corresponding to each of the first video frames to obtain the third video.

[0355] In some embodiments, the device also includes: a third determination part, configured to determine a first reference video frame from the first video; the first generation part is also configured to: generate a second video containing the first person based on the first head feature, the first identity feature of the first person and the first voice feature; wherein the first identity feature is extracted from the first reference video frame.

[0356] In some embodiments, the facial key points include a first key point representing the upper lip and a second key point representing the lower lip; the third determination part is further configured to: for each of the first video frames, determine the degree of closure of the mouth in the first video frame based on the first key point and the second key point corresponding to the first video frame; determine the first reference video frame with the largest degree of mouth closure from the first video.

[0357] In some embodiments, the first generating part is further configured to: generate a fourth video containing the first person based on the first head feature and the first voice feature; perform image fusion on the face area and the background area in at least one fourth video frame in the fourth video to obtain the second video.

[0358] The embodiment of the present disclosure provides a model training device, which includes the various units included and the various parts included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; during the implementation process, the processor can be a CPU, MPU, DSP or FPGA, etc.

[0359] FIG7B is a schematic diagram of the structure of a model training device provided by an embodiment of the present disclosure. As shown in FIG7B , the model training device 720 includes: a second acquisition part 721, a second generation part 722, an adjustment part 723, and a fourth determination part 724, wherein:

[0360] The second acquisition part 721 is configured to acquire first voice training data and first head training data of a second person; wherein the first head training data is obtained by masking the mouth area of ​​the second person in the training video, and the first voice training data is obtained based on the training audio;

[0361] The second generating part 722 is configured to input the first voice training data and the first head training data into the to-be-trained model to obtain a fifth video containing the second person;

[0362] An adjustment part 723 is configured to determine a target loss using the fifth video and adjust parameters of the to-be-trained model based on the target loss;

[0363] The fourth determining part 724 is configured to determine that the training of the to-be-trained model is completed when the speaking action of the second character in the fifth video matches the training audio.

[0364] In some embodiments, the model to be trained includes a rendering generation network, the first voice training data includes second voice features, and the first head training data includes second head features; the second generation part is also configured to: use the rendering generation network to generate a fifth video containing the second person based on the second head features and the second voice features.

[0365] In some embodiments, the model to be trained includes a rendering generation network, and at least one of a first feature extraction network and a second feature extraction network;

[0366] In a case where the model to be trained includes the first feature extraction network but does not include the second feature extraction network, the first head training data includes a sixth video, the sixth video is obtained by masking the mouth area of ​​the second person in the training video, the first voice training data includes a second voice feature, and the second generating part is further configured to: extract a second head feature of the second person from the sixth video using the first feature extraction network; and generate a fifth video including the second person based on the second head feature and the second voice feature using the rendering generation network;

[0367] In a case where the model to be trained includes the second feature extraction network but does not include the first feature extraction network, the first head training data includes second head features, and the first voice training data includes the training audio, the second generating portion is further configured to: utilize the second feature extraction network to perform feature extraction on the training audio to obtain second voice features; and utilize the rendering generation network to generate a fifth video including the second person based on the second head features and the second voice features;

[0368] In the case where the model to be trained includes the first feature extraction network and the second feature extraction network, the first head training data includes the sixth video, the first voice training data includes the training audio, and the second generation part is further configured to: use the first feature extraction network to extract the second head features of the second person from the sixth video; use the second feature extraction network to perform feature extraction on the training audio to obtain second voice features; and use the rendering generation network to generate a fifth video containing the second person based on the second head features and the second voice features.

[0369] In some embodiments, the model to be trained includes the rendering generation network and the first feature extraction network; the adjustment part is further configured to: adjust parameters of the rendering generation network and the first feature extraction network based on the target loss.

[0370] In some embodiments, the first feature extraction network includes a first encoder and a mapping subnetwork, and the second head feature includes a second non-mouth area feature and a second head motion feature; the second generation part is further configured to: use the first encoder to encode the sixth video to obtain the second non-mouth area feature; use the mapping subnetwork to map the second non-mouth area feature to a preset motion feature space to obtain the second head motion feature.

[0371] In some embodiments, the second generating part is further configured to: fuse the second head feature and the second voice feature to obtain a second fused feature; and generate the fifth video based on the second fused feature by using the rendering generation network.

[0372] In some embodiments, the model to be trained includes the rendering generation network and the second feature extraction network, the second feature extraction network includes a second encoder and a third encoder, and the second speech features include second content features and second emotion features; the second generation part is also configured to: use the second encoder to encode the training audio to obtain the second content features; use the third encoder to encode the training audio to obtain the second emotion features; fuse the second head features and the second content features to obtain the second fusion features; use the rendering generation network to generate the fifth video based on the second fusion features and the second emotion features.

[0373] In some embodiments, the rendering generation network includes a fourth encoder and a first decoder; the second generation part is further configured to: use the fourth encoder to encode the second fusion feature to obtain the encoded second fusion feature; use the first decoder to decode the second emotion feature and the encoded second fusion feature to obtain the fifth video.

[0374] In some embodiments, the model also includes a face detection network; the device also includes: a second detection part, which is used to use the face detection network to detect the second person's face in the training video, obtain the facial key points corresponding to at least one second video frame in the training video, and for each second video frame, based on the facial key points corresponding to the second video frame, determine the mouth area corresponding to the second video frame; a second mask part, which is configured to perform mask processing on the mouth area corresponding to each second video frame to obtain the sixth video.

[0375] In some embodiments, the model also includes a third feature extraction network; the device also includes: a fifth determination part, configured to determine a second reference video frame from the training video; a first extraction part, configured to use the third feature extraction network to extract the second identity feature of the second person from the second reference video frame; the second generation part is also configured to: use the rendering generation network to generate a fifth video containing the second person based on the second head feature, the second identity feature and the second voice feature.

[0376] In some embodiments, the adjustment part is further configured to: determine a first loss based on the fifth video and the reference video corresponding to the training audio; determine the mouth areas corresponding to at least one second video frame in the training video, and determine the mouth areas corresponding to at least one third video frame in the fifth video; determine a second loss based on the mouth areas corresponding to each of the second video frames, and the mouth areas corresponding to each of the third video frames; determine a target loss based on the first loss and the second loss.

[0377] In some embodiments, the adjustment part is further configured to: extract mutually matching video clips and audio clips from the fifth video and the training audio, respectively; encode the video clips and the audio clips, respectively, to obtain first encoding features and second encoding features; obtain a third loss based on the first encoding features and the second encoding features; and determine the target loss based on the first loss, the second loss and the third loss.

[0378] In some embodiments, the adjustment part is further configured to: use a first discriminator to determine the authenticity of at least one fourth video frame in the reference video and the authenticity of at least one third video frame in the fifth video; determine a fourth loss based on the authenticity of each of the fourth video frames and the authenticity of each of the third video frames; determine the target loss based on the first loss, the second loss, the third loss and the fourth loss.

[0379] In some embodiments, the adjustment part is further configured to: for each pair of adjacent fourth video frames in the reference video, use a second discriminator to determine the true proximity between the pair of adjacent fourth video frames; for each pair of adjacent third video frames in the fifth video, use a second discriminator to determine the true proximity between the pair of adjacent third video frames; determine a fifth loss based on the true proximity between each pair of adjacent fourth video frames and the true proximity between each pair of adjacent third video frames; determine the target loss based on the first loss, the second loss, the third loss, the fourth loss and the fifth loss.

[0380] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or parts included in the device provided by the embodiment of the present disclosure can be configured to perform the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0381] In the embodiments of the present disclosure and other embodiments, "part" can be part of a circuit, part of a processor, part of a program or software, etc., and of course it can also be a unit, a module, or a non-modular one.

[0382] It should be noted that, in the embodiments of the present disclosure, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0383] An embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0384] The present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above method. The computer-readable storage medium may be transient or non-transient.

[0385] An embodiment of the present disclosure provides a computer program, including computer-readable codes. When the computer-readable codes are executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0386] An embodiment of the present disclosure provides a computer program product, including a computer program, which implements some or all of the steps in the above method when executed by a processor.

[0387] The present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, implements some or all of the steps in the above method. In some embodiments, the computer program product can be implemented by hardware, software, or a combination thereof. In some embodiments, the computer program product is embodied as a computer storage medium, and in other embodiments, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0388] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between the embodiments, and reference can be made to the similarities or similarities between them. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above-mentioned method embodiments and have similar beneficial effects as the method embodiments. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product disclosed herein, please refer to the description of the method embodiments disclosed herein for understanding.

[0389] It should be noted that FIG8 is a schematic diagram of a hardware entity of a computer device in an embodiment of the present disclosure. As shown in FIG8 , the hardware entity of the computer device 800 includes: a processor 801, a communication interface 802, and a memory 803, wherein:

[0390] Processor 801 generally controls the overall operation of computer device 800 .

[0391] The communication interface 802 enables the computer device to communicate with other terminals or servers through a network.

[0392] The memory 803 is configured to store instructions and applications executable by the processor 801. It can also cache data to be processed or processed by the processor 801 and various components of the computer device 800 (e.g., image data, audio data, voice communication data, and video communication data). This can be implemented using flash memory (FLASH) or random access memory (RAM). Data can be transmitted between the processor 801, the communication interface 802, and the memory 803 via a bus 804.

[0393] It should be understood that “one embodiment” or “an embodiment” mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, “in one embodiment” or “in an embodiment” appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution. The execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure are for description purposes only and do not represent the advantages and disadvantages of the embodiments.

[0394] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0395] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are schematic. For example, the division of the units is a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0396] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, the functional units in the various embodiments of the present disclosure may all be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0397] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0398] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0399] The above is only an embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or replacements that can be easily conceived by any technician familiar with this technical field within the technical scope disclosed in this disclosure should be covered by the protection scope of the present disclosure. Industrial Applicability

[0400] The present disclosure discloses a video generation and model training method, apparatus, device, storage medium, computer program, and computer program product. The video generation method includes: obtaining a first voice feature and a first head feature of a first person; wherein the first head feature is obtained by masking the mouth area of ​​the first person in the first video, and the first voice feature is obtained based on a first audio; generating a second video containing the first person based on the first head feature and the first voice feature; and matching the first person's speaking movements in the second video with the first audio. In the present disclosure, the first voice feature and the first head feature of the first person are obtained, and based on the first head feature and the first voice feature, a second video containing the first person is generated, and the first person's speaking movements in the second video are matched with the first audio. Thus, since the first head feature is obtained by masking the mouth area of ​​the first person in the first video, the first head feature is not affected by the mouth area and is more reasonable and accurate. Since the first voice feature is obtained based on the first audio, the head appearance and posture of the first person in the second video generated based on the first head feature and the first voice feature are more accurate and more closely match the first audio, thereby making the person's overall speaking movements more reasonable and realistic. In this way, the rationality and realism of the overall speaking action of the first person in the generated second video can be improved, thereby improving the quality of the generated second video.

Claims

1. A video generation method, the method include: Acquire a first voice feature and a first head feature of a first person; wherein the first head feature is obtained by masking the mouth area of ​​the first person in the first video, and the first voice feature is obtained based on the first audio; Based on the first head feature and the first voice feature, a second video including the first person is generated; the speaking action of the first person in the second video is matched with the first audio.

2. The method according to claim 1, in, The first head feature includes a first non-mouth area feature and a first head action feature; The first non-mouth area feature is obtained by encoding a third video, and the third video is obtained by masking a mouth area of ​​the first person in the first video; The first head motion feature is obtained by mapping the first non-mouth area feature to a preset motion feature space.

3. The method according to claim 1, in, The step of generating a second video including the first person based on the first head feature and the first voice feature includes: fusing the first head feature and the first voice feature to obtain a first fused feature; Based on the first fusion features, the second video is generated.

4. The method according to claim 3, in, The first speech feature includes a first content feature and a first emotion feature; The fusing the first head feature and the first voice feature to obtain a first fused feature includes: fusing the first head feature and the first content feature to obtain a first fused feature; The generating the second video based on the first fusion feature includes: The second video is generated based on the first fusion feature and the first emotion feature.

5. The method according to claim 4, in, The step of generating the second video based on the first fusion feature and the first emotion feature includes: Performing encoding processing on the first fused feature to obtain the encoded first fused feature; The first emotion feature and the encoded first fusion feature are decoded to obtain the second video.

6. The method according to any one of claims 1 to 5, in, The first head feature is obtained based on the third video, and the method further includes: Detecting a face of a first person in the first video to obtain facial key points corresponding to at least one first video frame in the first video; For each of the first video frames, determining a mouth area corresponding to the first video frame based on facial key points corresponding to the first video frame; Mask processing is performed on the mouth area corresponding to each of the first video frames to obtain the third video.

7. The method according to claim 6, in, The method further comprises: Determining a first reference video frame from the first video; The step of generating a second video including the first person based on the first head feature and the first voice feature includes: Based on the first head feature, the first identity feature of the first person and the first voice feature, a second video including the first person is generated; wherein the first identity feature is extracted from the first reference video frame.

8. The method according to claim 7, in, The facial key points include a first key point representing the upper lip and a second key point representing the lower lip; The determining a first reference video frame from the first video includes: For each of the first video frames, determining a degree of closure of a mouth in the first video frame based on a first key point and a second key point corresponding to the first video frame; A first reference video frame in which the mouth is most closed is determined from the first video.

9. The method according to any one of claims 1 to 5, in, The step of generating a second video including the first person based on the first head feature and the first voice feature includes: Based on the first head feature and the first voice feature, generating a fourth video including the first person; Image fusion is performed on the face area and the background area in at least one fourth video frame in the fourth video to obtain the second video.

10. A model training method, include: Acquire first voice training data and first head training data of a second person; wherein the first head training data is obtained by masking the mouth area of ​​the second person in the training video, and the first voice training data is obtained based on the training audio; Inputting the first voice training data and the first head training data into the model to be trained to obtain a fifth video including the second person; Determine a target loss using the fifth video, and adjust parameters of the model to be trained based on the target loss; When the speaking action of the second character in the fifth video matches the training audio, it is determined that the training of the model to be trained is completed.

11. The method according to claim 10, in, The model to be trained includes a rendering generation network, the first speech training data includes a second speech feature, and the first head training data includes a second head feature; The first voice training data and the first head training data are input into the to-be-trained model, and a fifth visual image including the second person is obtained. Frequency, including: The rendering generation network is used to generate a fifth video containing the second person based on the second head feature and the second voice feature.

12. The method according to claim 10, in, The model to be trained includes a rendering generation network, and at least one of a first feature extraction network and a second feature extraction network; In the case where the model to be trained includes the first feature extraction network but does not include the second feature extraction network, the first head training data includes a sixth video, the sixth video is obtained by masking the mouth area of ​​the second person in the training video, the first voice training data includes a second voice feature, and the first voice training data and the first head training data are input into the model to be trained to obtain a fifth video including the second person, including: using the first feature extraction network to extract the second head feature of the second person from the sixth video; using the rendering generation network to generate a fifth video including the second person based on the second head feature and the second voice feature; In the case where the model to be trained includes the second feature extraction network but does not include the first feature extraction network, the first head training data includes the second head feature, and the first voice training data includes the training audio, and the step of inputting the first voice training data and the first head training data into the model to be trained to obtain a fifth video including the second person includes: using the second feature extraction network to perform feature extraction on the training audio to obtain a second voice feature; and using the rendering generation network to generate a fifth video including the second person based on the second head feature and the second voice feature. In the case where the model to be trained includes the first feature extraction network and the second feature extraction network, the first head training data includes the sixth video, and the first voice training data includes the training audio, and the first voice training data and the first head training data are input into the model to be trained to obtain a fifth video containing the second person, including: using the first feature extraction network to extract the second head feature of the second person from the sixth video; using the second feature extraction network to extract features from the training audio to obtain second voice features; and using the rendering generation network to generate a fifth video containing the second person based on the second head feature and the second voice feature.

13. The method according to claim 12, in, The model to be trained includes the rendering generation network and the first feature extraction network; The step of adjusting parameters of the model to be trained based on the target loss includes: Parameters of the rendering generation network and the first feature extraction network are adjusted based on the target loss.

14. The method according to claim 12, in, The first feature extraction network includes a first encoder and a mapping sub-network, and the second head feature includes a second non-mouth area feature and a second head action feature; The step of extracting the second head feature of the second person from the sixth video by using the first feature extraction network includes: Using the first encoder, encoding the sixth video to obtain the second non-mouth area feature; The second non-mouth area feature is mapped to a preset motion feature space by using the mapping subnetwork to obtain the second head motion feature.

15. The method according to claim 12, in, The step of using the rendering generation network to generate a fifth video containing the second person based on the second head feature and the second voice feature includes: fusing the second head feature and the second voice feature to obtain a second fused feature; The fifth video is generated by utilizing the rendering generation network based on the second fusion feature.

16. The method according to claim 15, in, The model to be trained includes the rendering generation network and the second feature extraction network, the second feature extraction network includes a second encoder and a third encoder, and the second speech feature includes a second content feature and a second emotion feature; The step of extracting features from the training audio using the second feature extraction network to obtain the second speech feature includes: encoding the training audio using the second encoder to obtain the second content feature; encoding the training audio using the third encoder to obtain the second emotion feature; The fusing the second head feature and the second voice feature to obtain a second fused feature includes: fusing the second head feature and the second content feature to obtain a second fused feature; The generating the fifth video by using the rendering generation network based on the second fusion feature includes: generating the fifth video by using the rendering generation network based on the second fusion feature and the second emotion feature.

17. The method according to claim 16, in, The rendering generation network includes a fourth encoder and a first decoder; The step of generating the fifth video by using the rendering generation network based on the second fusion feature and the second emotion feature includes: Using the fourth encoder, encoding the second fused feature to obtain the encoded second fused feature; The first decoder is used to decode the second emotion feature and the encoded second fusion feature to obtain the fifth video.

18. The method according to any one of claims 12 to 17, in, The model also includes a face detection network; the method also includes: Detecting the face of a second person in the training video using the face detection network to obtain facial key points corresponding to at least one second video frame in the training video, and determining a mouth area corresponding to each second video frame based on the facial key points corresponding to the second video frame; Mask processing is performed on the mouth area corresponding to each of the second video frames to obtain the sixth video.

19. The method according to claim 18, in, The model also includes a third feature extraction network; the method also includes: Determining a second reference video frame from the training video; extracting a second identity feature of the second person from the second reference video frame using the third feature extraction network; The method further comprises: using the rendering generation network to generate a fifth visual image including the second person based on the second head feature and the second voice feature. The method comprises: using the rendering generation network to generate a fifth video containing the second person based on the second head feature, the second identity feature and the second voice feature.

20. The method according to any one of claims 10 to 17, in, The step of determining the target loss by using the fifth video comprises: Determining a first loss based on the fifth video and a reference video corresponding to the training audio; Determine a mouth region corresponding to at least one second video frame in the training video, and determine a mouth region corresponding to at least one third video frame in the fifth video; Determine a second loss based on a mouth area corresponding to each of the second video frames and a mouth area corresponding to each of the third video frames; A target loss is determined based on the first loss and the second loss.

21. The method according to claim 20, in, The determining the target loss by using the fifth video further includes: Extracting mutually matching video segments and audio segments from the fifth video and the training audio respectively; Encoding the video clip and the audio clip respectively to obtain a first encoding feature and a second encoding feature; Obtaining a third loss based on the first encoding feature and the second encoding feature; The determining a target loss based on the first loss and the second loss includes: The target loss is determined based on the first loss, the second loss, and the third loss.

22. The method according to claim 21, in, The determining the target loss by using the fifth video further includes: Determining, using a first discriminator, the authenticity of at least one fourth video frame in the reference video and the authenticity of at least one third video frame in the fifth video; determining a fourth loss based on the authenticity of each of the fourth video frames and the authenticity of each of the third video frames; The determining the target loss based on the first loss, the second loss and the third loss includes: The target loss is determined based on the first loss, the second loss, the third loss, and the fourth loss.

23. The method according to claim 22, in, The determining the target loss by using the fifth video further includes: For each pair of adjacent fourth video frames in the reference video, using a second discriminator, determining a true degree of proximity between the pair of adjacent fourth video frames; For each pair of adjacent third video frames in the fifth video, using a second discriminator, determining a true degree of proximity between the pair of adjacent third video frames; determining a fifth loss based on the true proximity between each pair of adjacent fourth video frames and the true proximity between each pair of adjacent third video frames; The determining the target loss based on the first loss, the second loss, the third loss, and the fourth loss includes: The target loss is determined based on the first loss, the second loss, the third loss, the fourth loss, and the fifth loss.

24. A video generating device, the device include: A first acquisition part is configured to acquire a first voice feature and a first head feature of a first person; wherein the first head feature is obtained by masking a mouth area of ​​the first person in the first video, and the first voice feature is obtained based on the first audio; The first generating part is configured to generate a second video containing the first person based on the first head feature and the first voice feature; the speaking action of the first person in the second video matches the first audio.

25. A model training device, the device include: The second acquisition part is configured to acquire first voice training data and first head training data of a second person; wherein the first head training data is obtained by masking the mouth area of ​​the second person in the training video, and the first voice training data is obtained based on the training audio; A second generating part is configured to input the first voice training data and the first head training data into a model to be trained to obtain a fifth video containing the second person; an adjustment part, configured to determine a target loss using the fifth video, and adjust parameters of the model to be trained based on the target loss; The fourth determination part is configured to determine that the training of the model to be trained is completed when the speaking action of the second character in the fifth video matches the training audio.

26. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, in, When the processor executes the program, the steps of the method described in any one of claims 1 to 9 or any one of claims 10 to 23 are implemented.

27. A computer-readable storage medium having a computer program stored thereon, in, When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 9 or any one of claims 10 to 23 are implemented.

28. A computer program comprising computer readable code, in, When the computer readable code is run in a computer device, a processor in the computer device executes the steps for implementing the method of any one of claims 1 to 9 or any one of claims 10 to 23.

29. A computer program product comprising a computer program, in, When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9 or any one of claims 10 to 23.

Citation Information

Patent Citations

  • Video generation method and device and electronic equipment

    CN112927712A

  • Action driving method and device of target object, equipment and storage medium

    CN113554737A

  • Training method and device of speaking video generation model, electronic equipment and storage medium

    CN115620371A

  • Image generation and model training method and device, equipment and storage medium

    CN115690238A

  • Video generation and model training method and device, equipment and storage medium

    CN116385604A

Cited By

  • Real-time digital human generation system and method based on low-calculation-amount voice driving

    CN119991893A

  • Data transmission processing method and system for digital human-generated video

    CN120050452A

  • Video generation model training method and device, equipment, storage medium, program product and video generation method

    CN120614502A

  • Video generation method, video live broadcast method and training method of video generation model

    CN121644922A

  • Voice-driven face video synthesis method and system based on 3d Gaussian

    CN121788572A