Digital population broadcast video generation method and device, equipment, storage medium and program product
The facial expression coefficients are generated by the sound expression model and the facial expression model and the lip shape of the video frame are solved, and the problem of poor lip shape consistency in the existing technology is improved, and the lip shape consistency of digital population videos is reduced. Sensitivity to character identities.
Patent Information
- Application Number
- CN202411088621.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-12
AI Technical Summary
In the existing 2D digital population broadcasting technology, offline-trained lip synthesis model lacks the training data set of synthesized characters, resulting in poor consistency between the generated lip typing content and the actual speaking lip typing.
Using the sound-producing expression model and expression-producing facial model, by obtaining oral audio and oral source video, generate face expression coefficients and modify the lip shape of the video frame, reducing the sensitivity of oral video to the character identity and improving the lip shape consistency.
The consistency between the generated oral video lip shaped content and the actual speaking lip shaped , reduces sensitivity to character identities , and saves the computing power and time cost of online fine-tuning training.
Smart Images

Figure CN118945420A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of digital technology, and in particular to a method, device, equipment, storage medium and program product for generating digital human broadcast video. Background Art
[0002] 2D digital people are digital people that use a deep replication technology to replicate the voice and appearance of real-life videos. In the later stage, by inputting any text content, real-life 2D digital people can be used to broadcast video content, thus replacing real people. It has a wide range of application value in live broadcasting, broadcasting, customer service and other scenarios.
[0003] Existing 2D digital voice-over technology mainly includes sound and image replication technology: 1) Sound replication method: first learn and train the sound in the captured video online to obtain a sound model, and then use text-to-audio conversion (TTS) technology to convert the input text content into audio content or files. 2) Image replication method: Use an end-to-end GAN generation method, such as Wave2Lip, to train an audio lip shape generation model offline, and use the lip shape generation model and the sound model to modify the face in the original captured video to generate a voice-over video with matching audio content and lip shape. The disadvantage is that the offline lip shape synthesis model trained in the existing image replication lacks a training data set for the synthesized character, resulting in poor consistency between the final synthesized lip shape content and the actual speaking lip shape. Summary of the invention
[0004] The present disclosure is proposed in view of the above problems. The present disclosure provides a method, device, equipment, storage medium and program product for generating a digital voice broadcast video.
[0005] According to one aspect of the present disclosure, a method for generating a digital human broadcast video is provided, comprising:
[0006] Get the spoken audio and spoken source video;
[0007] Generate a facial expression coefficient for each audio frame in the spoken audio using a sound-generated expression model;
[0008] By utilizing the expression-generated facial model, based on the facial expression coefficient of each audio frame in the spoken audio, the lip shape of the facial image of each corresponding video frame in the spoken source video is modified to generate spoken images corresponding to each of the facial images, and a spoken video is generated based on each of the spoken images.
[0009] In addition, according to a method for generating a digital oral broadcast video in one aspect of the present disclosure, the step of obtaining oral broadcast audio includes:
[0010] Get the spoken text;
[0011] The spoken audio is generated according to the spoken text using a Vincent audio model, wherein the Vincent audio model is trained based on an audio recording of a replicated character and a text corresponding word for word to the audio recording.
[0012] In addition, according to a method for generating a digital oral broadcast video in one aspect of the present disclosure, the step of obtaining the oral broadcast source video includes:
[0013] Get the video of the person being reproduced;
[0014] Generating a plurality of spoken action video clips according to the reproduced character video;
[0015] Combine several spoken action video segments among the multiple spoken action video segments and inter-frame transition videos between adjacent spoken action video segments to form a spoken action source video.
[0016] In addition, according to a method for generating a digital oral broadcast video in one aspect of the present disclosure, several oral broadcast action video segments among the plurality of oral broadcast action video segments are extracted based on the duration of the oral broadcast video;
[0017] or,
[0018] The inter-frame transition video between adjacent oral action video clips is obtained based on the action frame difference;
[0019] or,
[0020] The step of generating a plurality of spoken action video clips according to the reproduced character video includes:
[0021] Obtaining human body posture data, human face posture data and human face key point data of each video frame of the reproduced person video;
[0022] Extracting action video clips of the reproduced character video according to the human body posture data to obtain a plurality of first action video clips;
[0023] According to the face posture data, cutting the first action video segments in which the face posture angle is greater than a preset angle value to obtain a plurality of second action video segments;
[0024] According to the human body posture data, the facial posture data and the facial key point data, the first and last similar frames of each of the second action video clips are searched, and the middle video between the first and last similar frames is cropped to obtain the spoken action video clip.
[0025] In addition, according to a method for generating a digital oral video according to one aspect of the present disclosure, the audio-generated expression model includes an audio coding network and an audio semantic feature to expression mapping network, the audio coding network is used to encode the audio features of the audio frame of the oral audio, and the audio semantic feature to expression mapping network is used to map the audio features to human facial expressions to obtain human facial expression coefficients;
[0026] and / or,
[0027] Training the sound-generated expression model includes:
[0028] Acquire a first training sample, where the first training sample includes a first training image and a first training audio;
[0029] Using the first training sample, based on a target loss function, training the voice-generated expression model;
[0030] Among them, the target loss function includes facial expression coefficient loss and mouth facial key point loss, the facial expression coefficient loss and the mouth facial key point loss are determined based on the target facial expression coefficient, the target facial expression coefficient is the facial expression coefficient generated after the face reconstruction network model reconstructs the target training image face, and the target training image is the lip-synthesized image of the first training image under the first training audio.
[0031] And / or, the expression-generated facial model is a network model generated by a moving face video;
[0032] Training the expression facial model includes:
[0033] Acquire a second training sample, where the second training sample includes a second training image and a second training audio;
[0034] Reconstructing the face of the second training image using a face reconstruction network model to obtain a first face expression coefficient, a first face posture coefficient, and a first face shape base coefficient, and obtaining a first face key point according to the first face expression coefficient, the first face posture coefficient, and the first face shape base coefficient;
[0035] Using the sound-generated expression model, generating a second facial expression coefficient according to the second training audio, and obtaining a second facial key point according to the second facial expression coefficient, the corresponding first facial posture coefficient and the corresponding first facial shape basis coefficient;
[0036] Training the expression facial model based on the first facial key points, the second facial key points and the second training image;
[0037] In addition, according to a method for generating a digital oral broadcast video in one aspect of the present disclosure, the method of modifying the mouth shape of the face image of each corresponding video frame in the oral broadcast source video based on the face expression coefficient of each audio frame in the oral broadcast audio to generate an oral broadcast image corresponding to each face image includes:
[0038] Based on the facial posture expression coefficients of each audio frame in the spoken audio and the facial posture expression coefficients of the facial images of each corresponding video frame in the spoken source video, the lip shape of the facial images of each corresponding video frame in the spoken source video is modified, wherein the facial posture expression coefficients of the audio frame include the facial expression coefficients of the audio frame and the posture coefficients of the facial images of the corresponding video frame.
[0039] According to another aspect of the present disclosure, a digital human broadcast video generation device is provided, comprising:
[0040] The acquisition module is used to obtain the spoken audio and the spoken source video;
[0041] A first generating module is used to generate a facial expression coefficient of each audio frame in the spoken audio by using a sound-generated expression model;
[0042] The second generation module is used to use the expression facial model to modify the mouth shape of the facial image of each video frame in the oral source video based on the facial expression coefficient of each audio frame in the oral audio, so as to generate oral images corresponding to each of the facial images, and generate oral videos according to each of the oral images.
[0043] According to another aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the above methods.
[0044] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of any of the above methods are implemented.
[0045] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.
[0046] As described in detail below, according to a method, device, equipment, storage medium and program product for generating digital oral broadcast video according to an embodiment of the present disclosure, the facial expression coefficients of each audio frame in the oral broadcast audio are generated by using a sound-generated expression model; the mouth shape of the facial image of each corresponding video frame in the oral broadcast source video is modified based on the facial expression coefficients of each audio frame in the oral broadcast audio by using an expression-generated facial model to generate oral broadcast images corresponding to each facial image, and generate oral broadcast videos based on each oral broadcast image. The sensitivity of the oral broadcast video to the identity of the person is reduced, so that the lip shape content of the synthesized oral broadcast video is more consistent with the actual speaking lip shape.
[0047] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the technology as claimed. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.
[0049] Figure 1 FIG. 4 is a flowchart illustrating a method for generating a digital voice-over video according to an embodiment of the present disclosure.
[0050] Figure 2 FIG. 4 is a flow chart illustrating a method for generating a digital voice broadcast video according to an embodiment of the present disclosure.
[0051] Figure 3 It is a flowchart that further illustrates the acquisition of spoken audio in the method for generating digital spoken video in an embodiment of the present disclosure.
[0052] Figure 4 It is a flowchart that further illustrates the method for generating digital oral video according to an embodiment of the present disclosure for obtaining oral source video.
[0053] Figure 5 It is a flowchart that further illustrates the generation of oral speech action video clips in the digital oral speech video generation method of the embodiment of the present disclosure.
[0054] Figure 6 is a schematic diagram illustrating a facial posture according to an embodiment of the present disclosure;
[0055] Figure 7 is a schematic diagram illustrating a human body posture according to an embodiment of the present disclosure;
[0056] Figure 8is a schematic diagram illustrating key points of a human face according to an embodiment of the present disclosure;
[0057] Fig. 9 is a schematic diagram illustrating the training of a sound-generated expression model according to an embodiment of the present disclosure;
[0058] Fig.10 is another schematic diagram illustrating the training of the sound-generated expression model according to an embodiment of the present disclosure;
[0059] Fig.11 is a schematic diagram illustrating the training of an expression facial model according to an embodiment of the present disclosure;
[0060] Fig.12 is another schematic diagram illustrating the training of an expression facial model according to an embodiment of the present disclosure;
[0061] Fig.13 is a block diagram illustrating a digital voice-over video generating apparatus according to an embodiment of the present disclosure.
[0062] Fig.14 is a schematic diagram illustrating a computer program product according to an embodiment of the present disclosure.
[0063] Fig.15 is a hardware block diagram illustrating an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0064] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.
[0065] See also Figure 1 and Figure 2 , the digital population broadcast video generation method comprises:
[0066] S101, obtaining spoken audio and spoken source video.
[0067] In step S101, the spoken audio can be obtained by using a related spoken audio acquisition method. The spoken audio can be directly obtained, for example, the user records the spoken audio, and the system obtains the spoken audio recorded by the user. The spoken recording can also be obtained indirectly, for example, the spoken text is obtained, and the spoken audio is generated according to the spoken text, so as to achieve the purpose of obtaining the spoken audio; specifically, the user can record the spoken text, and the system generates the spoken audio according to the spoken text entered by the user; wherein, the related audio generation technology can be used to generate the spoken video according to the spoken text, for example, using the Wensheng audio model, based on the text to audio conversion (Text To Speech, referred to as TTS) technology, the spoken audio is generated according to the spoken text, and the spoken audio can be stored as an audio file, such as an audio file in wav format, so as to facilitate the subsequent steps to call the audio file.
[0068] The Vincent audio model can be trained online, see Figure 2 , the sound in the replication stage can be collected, and the Vincent audio model can be trained online based on the collected sound, so as to use the Vincent audio model to synthesize the spoken audio.
[0069] In one embodiment, see Figure 2 and Figure 3 , get the spoken audio, including:
[0070] S301, obtaining the spoken text;
[0071] S302, using a Vincent audio model to generate spoken audio according to the spoken text, wherein the Vincent audio model is trained based on an audio recording of a reproduced character and a text corresponding word for word to the audio recording.
[0072] In this embodiment, the Vincent audio model can be a Vincent audio model based on deep learning, for example, the Tacotron model, which directly converts text into a spectrogram and then generates audio through a vocoder, or it can be other Vincent audio modules, such as the WaveNet model (a generative model for generating high-quality natural audio), which uses a series of causal convolutional layers to generate audio samples, which can capture the long-term dependencies of audio and generate natural audio.
[0073] In order to allow users to choose which voice to use, the Vincent audio model can be trained online to replicate the voice of the person being matched. For example, in a certain application scenario, the user enters the audio recording of the person being replicated and the text corresponding to the audio recording word for word, and the system obtains the entered audio recording of the person being replicated and the text corresponding to the audio recording word for word, and trains the Vincent audio basic model based on the audio recording of the person being replicated and the text corresponding to the audio recording word for word, so as to obtain the trained Vincent audio model.
[0074] In step S101, the oral broadcast source video may be obtained by using a related oral broadcast source video acquisition method. The oral broadcast source video may be directly acquired, for example, a user records an oral broadcast source video, and the system acquires the recorded oral broadcast source video. The oral broadcast source video may be acquired indirectly. For example, a user records a video of a person to be reproduced, and the system acquires the video of the person to be reproduced, and generates the oral broadcast source video according to the video of the person to be reproduced.
[0075] In a specific embodiment, see Figure 4 , get the oral source video, including:
[0076] S401, obtaining a video of a person to be reproduced;
[0077] In step S401, a relevant acquisition method may be used to acquire the video of the person being reproduced. For example, the recorded video of the person being reproduced may be directly acquired. The video of the person being reproduced may be acquired by shooting in advance, or may be acquired by other means, such as downloading from a platform.
[0078] When the video of the person being reproduced is taken by the user, in order to obtain a more suitable video of the person being reproduced, the real person can be used to shoot the video, and the shooting time of each video can be set according to the requirements, and the video can be shot according to the set shooting time. For example, if the shooting time is set to 5 minutes, a video of about 5 minutes can be shot. Among them, in order to have a better shooting effect and obtain a suitable video of the person being reproduced, the shooting process can be shot according to the following requirements:
[0079] a. Do several sets of common hand and body movements. After each movement, keep silent for a while (e.g. 5 seconds) before doing the next movement.
[0080] b. There are no requirements for mouth movements. You can either remain silent or recite the lines.
[0081] c. Make some simple head movements or expressions;
[0082] d. Collect data in standing and sitting positions according to the broadcasting requirements;
[0083] e. According to the scene synthesis requirements, video capture can be performed in a green screen scene.
[0084] S402, generating a plurality of spoken action video clips according to the reproduced character video;
[0085] In step S402, multiple spoken action video clips are generated based on the video of the person being reproduced, so as to combine several spoken action video clips in the multiple spoken action video clips and the inter-frame transition videos between adjacent spoken action video clips to form a spoken source video, and the formed spoken source video can better reflect its spoken feature. Among them, each spoken action video clip is a video clip corresponding to a spoken action, and the inter-frame transition video can be generated between two spoken action video clips using the action frame interpolation method, so that the two spoken actions are continuous.
[0086] In order to obtain a more suitable reproduced oral video, in one embodiment, see Figure 5 , step S402 includes:
[0087] S501, obtaining human body posture data, human face posture data and human face key point data of each video frame of the reproduced person video.
[0088] In step S501, human body posture recognition, face posture recognition and face key point recognition are performed on the video frame respectively to obtain human body posture data, face posture data and face key point data of the video frame image. Figure 6 , the face posture data can specifically be the face rotation angle information euler (pitch angle Pitch, yaw angle Yaw, roll angle Roll); see Figure 7 , human posture recognition is also limb recognition, and the human posture data can be specifically limb joint point information HKP (x, y, score); see Figure 8 , the facial key point data may specifically be the facial key point coordinates FKP (x, y). In this embodiment, a relevant human posture recognition model may be used to recognize the human posture state to obtain human posture data, such as the OpenPose posture recognition model and the PoseNet posture recognition model; a relevant face posture recognition model may be used to recognize the face posture to obtain face posture data, such as the Dlib face posture recognition model and the FacePoseNet face posture recognition model; a relevant face key point recognition model may be used to recognize the face key points to obtain face key point data, such as the MTCNN face key point recognition model and the Facial Landmark Detection with Deep Learning face key point recognition model based on deep learning.
[0089] S502: extracting action video clips of the reproduced character video according to the human body posture data to obtain a plurality of first action video clips.
[0090] In step S502, the action clips of the reproduced character video are extracted using the human body posture data to obtain n1 first action video clips, so that each first action video clip can correspond to an action. Exemplarily, the extraction rules of the action video clips of the reproduced character video are: hand face key points (5 to 10), score greater than the score setting value (for example, 0.8), and the cumulative normalized Euclidean distance of the hand face key point coordinates in the window frame (for example, 30 adjacent frames) is less than 5. It should be understood that the first, second, and third in the disclosed embodiment are for the purpose of distinguishing the first action video clip, the second action video clip, and the third action video clip, and they themselves have no special meaning.
[0091] S503, according to the facial posture data, cutting the first action video segments in which the facial posture angle is greater than the preset angle value to obtain a plurality of second action video segments.
[0092] In step S503, according to the face posture data, n1 first action video clips are cut, and the video clips with face posture angles greater than the angle setting value are cut to obtain n2 second action video clips. It can be known that n2 is greater than n1, and the angle preset value can be set according to the needs. Among them, the angle setting value can be the total angle threshold of each face posture angle, and accordingly, the sum of the absolute values of each face posture angle is compared with the setting value. For example, if the setting value is 30, it is considered that cutting is required when |pitch|+|yaw|+|roll|≥30; the setting value can be the angle threshold of each face posture angle, for example, including the first setting value of pitch, the second setting value of yaw and the third setting value of roll. It can be considered that cutting is required when one of the face postures is greater than the setting value of the corresponding face posture, or it can be considered that cutting is required when any two face postures are greater than the setting value of the corresponding face posture, or it can be considered that cutting is required when all face postures are greater than the setting value of the corresponding face posture. It can be set specifically according to actual needs. Among them, in order to eliminate some action video clips that are too short and affect the subsequent oral video effect, the second action video clips with a duration less than a set duration (for example, 5 seconds) can be discarded to obtain n3 second action video clips.
[0093] S504, searching for the first and last similar frames of each second action video segment according to the human body posture data, the face posture data and the face key point data, and cutting the middle video between the first and last similar frames to obtain the spoken action video segment.
[0094] In step S504, according to the human body posture data, the face posture data and the face key point data, the second action video clip is traversed to find the first and last similar frames in the second action video clip, and the middle video between the first and last similar frames is cropped to obtain n4 oral action video clips. Among them, the oral action video clips with a duration less than the set duration (for example, 5 seconds) can be discarded to obtain m oral action video clips, where m is less than or equal to n3, and are recorded as: C{c1, c2, cm}.
[0095] When the similarity between the first and last frames is greater than a preset similarity threshold, the first and last frames are determined to be first and last similar frames. Exemplarily, the method for calculating the first and last frame similarity is as follows:
[0096]
[0097] The similarity between the first and last frames consists of three parts: the similarity of the hand shape Facial profile similarity Facial pose similarity |euler-euler′|. HKP i Represents the i-th limb joint information of the first frame, HKP′ i Represents the i-th limb joint information of the last frame, i=5 to i=10 correspond to the hand joints; FKP i Represents the coordinates of the i-th facial key point in the first frame, FKP′ i represents the coordinates of the i-th facial key point in the last frame, i=1 to i=17 correspond to the facial key points; euler represents the face rotation angle of the first frame, and euler′ represents the face rotation angle of the last frame.
[0098] An interpolation method (such as the deep interpolation method Super Solomo) can be used between the spoken action video clips to generate inter-frame transition videos between each pair of spoken action video clips, a total of m*m videos, recorded as I{i11, ..i1m, …, im1, …, imm}, so as to facilitate the subsequent acquisition of inter-frame transition videos between adjacent spoken action video clips.
[0099] S403, combining some spoken action video segments among the plurality of spoken action video segments and inter-frame transition videos between adjacent spoken action video segments to form a spoken action source video.
[0100] In step S403, several oral action video clips among the multiple oral action video clips can be extracted based on the duration of the oral video, that is, multiple oral action video clips can be extracted according to the duration T of the oral audio, and the extracted oral action video clips and the inter-frame transition videos of the extracted adjacent oral action video clips are combined as the oral source video. Specifically, multiple oral action video clips can be randomly extracted with replacement, and the extraction is terminated when the total duration of the clip is greater than T. According to the numbers of the two adjacent videos extracted, the corresponding inter-frame transition video is obtained, such as: cx, ixy, cy, iyz, cz. Among them, x, y, z are the video clip numbers, and these videos are spliced to form the oral source video.
[0101] In step 403, the inter-frame transition video between the two spoken action video clips can be generated based on a related inter-frame transition video generation method. For example, the inter-frame transition video between the two spoken action video clips is generated based on the difference between the action frames.
[0102] S102, using the sound-generated expression model, generating facial expression coefficients for each audio frame in the spoken audio.
[0103] In step S102, the audio features (e.g., Mel spectrum mel features) of the audio frame of the spoken audio (corresponding to the current frame image timestamp) can be extracted, and the audio features are sent to the audio-generated expression model Audio2Exp to predict the facial expression coefficient b of the audio frame of the spoken audio. 1 .
[0104] The audio-generated expression model can select the corresponding model according to the needs, which can be a model of audio encoding network (AudioEncoder) + audio semantic feature to expression mapping network (Mapping), Audio2Face model, etc. Among them, the Audio2Face model is a deep learning-based model developed by NVIDIA, which can generate high-fidelity facial expression animations from audio signals.
[0105] In some embodiments, the audio-generated expression model includes an audio coding network and an audio semantic feature to expression mapping network. The audio coding network is used to encode the audio features of the audio frames of the spoken audio, and the audio semantic feature to expression mapping network is used to map the audio features to facial expressions to obtain facial expression coefficients.
[0106] The voice-generated expression model can be pre-trained, see Fig. 9 , training the voice-generated expression model, including:
[0107] Step 901, obtaining a first training sample, where the first training sample includes a first training image and a first training audio;
[0108] Step 902, using the first training sample, based on the target loss function, training the sound-generated expression model; wherein the target loss function includes a facial expression coefficient loss and a mouth facial key point loss, the facial expression coefficient loss and the mouth facial key point loss are determined based on a target facial expression coefficient e′, the target facial expression coefficient e′ is a facial expression coefficient generated by the face reconstruction network model after reconstructing the target training image I′, and the target training image I′ is the first training image I 0 The lip-synthesized image under the first training audio a. The facial expression coefficient loss can be calculated based on the target facial expression coefficient e′ and the facial expression coefficient e output by the model, the mouth facial key point loss can be calculated based on the target facial key point P′ and the facial key point corresponding to the output of the sound-generated expression model, and the facial key point corresponding to the facial expression coefficient output by the sound-generated expression model can be calculated based on the facial expression coefficient output by the sound-generated expression model, the facial posture coefficient and the shape basis coefficient generated by the face reconstruction network model after the target training image I′ face reconstruction, which can be calculated specifically by using relevant calculation methods, and will not be described repeatedly in this embodiment.
[0109] For example, see Fig.10 ,The steps of training the voice-generated expression model include:
[0110] Prepare the required image and audio training set as the first training sample, where I 0 is the first training image, a is the first training audio;
[0111] Build an audio encoding network (AudioEncoder), which performs semantic encoding on audio features, and build an audio semantic feature to expression mapping network (Mapping), which is a fully connected structure network;
[0112] Using the end-to-end generation technique, in the first training image I 0 The lip generation network Wave2Lip is trained in advance under the dataset. The lip generation network is trained for the first training image I 0 The character identity has a good generation effect and is used as a teacher model. The face reconstruction model that has been trained in the industry (such as DeepFace3D) is prepared in advance and named Re.
[0113] Define the target loss function Loss, which is composed of the facial expression coefficient loss Lexp and the mouth facial key point loss Llks, and has:
[0114] Loss = 2L ecp +L lks
[0115]
[0116] Among them, T is the duration, e t is the facial expression coefficient output by the sound-generated expression model at time t, e′ t is the target facial expression coefficient at time t, P t is the facial key point corresponding to the facial expression coefficient output by the sound-generated expression model at time t, P′ t is the target face key point at time t.
[0117] Freeze the model coefficients of the above-mentioned lip generation network Wave2Lip model and the face reconstruction model, use the above-mentioned first training image, and train the Audio2ExpNet network of the sound-generated expression model according to the target loss function Loss, where e 0 is the first training image I 0 Facial expression coefficient of face reconstruction, target training image I′ is the first training image I 0 In the lip-synthesized image under the corresponding audio, the target facial expression coefficient e′ is the facial expression coefficient reconstructed from the image I′ face, and Pose and ID are the facial posture coefficient and shape basis coefficient reconstructed from the image I′ face.
[0118] In one embodiment, in order to improve the stability of the lip shape generated in the video mode, the facial expression coefficient and the facial posture coefficient are de-jittered using OneEurope filtering to obtain p′ 0 , b′ 0 , b′ 1 .
[0119] S103, using the expression-generated facial model, based on the facial expression coefficients of each audio frame in the spoken audio, modify the lip shape of the facial image of each corresponding video frame in the spoken source video to generate spoken images corresponding to each facial image, and generate spoken videos based on each spoken image.
[0120] In the disclosed embodiment, the corresponding video frame refers to the correspondence between the timestamp of the video frame in the spoken source video and the timestamp of the audio frame in the spoken audio.
[0121] In some embodiments, the expression facial model Exp2Face adopts a motion face video generation network model, specifically, NVIDIA's motion face video generation network (Flow & Image Generator) model, high-quality dynamic face video model DVP (Dynamic Video Portraits), etc., which can modify the lip shape model of the facial image of each corresponding video frame in the spoken audio source video based on the facial expression coefficient of each audio frame in the spoken audio.
[0122] In one embodiment, based on the facial expression coefficient of each audio frame in the spoken audio, the mouth shape of the facial image of each corresponding video frame in the spoken source video is modified to generate a spoken image corresponding to each facial image, including:
[0123] Based on the facial posture expression coefficients of each audio frame in the spoken audio and the facial posture expression coefficients of the facial images of each corresponding video frame in the spoken source video, the lip shape of the facial images of each corresponding video frame in the spoken source video is modified, wherein the facial posture expression coefficients of the audio frame include the facial expression coefficients of the audio frame and the posture coefficients of the facial images of the corresponding video frame.
[0124] In this embodiment, the facial posture expression coefficient of the video frame of the oral source video can be obtained, and the facial posture expression coefficient includes the facial expression coefficient and the facial posture coefficient. Specifically, the image frames of the oral source video can be traversed, and the face area image F is obtained for the current frame image I, and the face area image F is reconstructed using a face reconstruction model (such as a DeepFace3D model) Re to obtain the facial expression coefficient b of the face area image frame (original face) of the oral source video. 0 (64 sets of coefficients) and face posture coefficient p 0 (3 groups of coefficients), that is, obtaining the facial posture expression coefficients of the video frame of the spoken source video.
[0125] Specifically, the facial posture expression coefficient of the audio frame, the facial posture expression coefficient of the video frame of the corresponding timestamp, and the facial image of the video frame of the corresponding timestamp can be input into the expression generation facial network model to generate the oral face image. In order to improve the oral face image generation effect, the face image F of the oral source video is scaled to 256x256 size to obtain the scaled face image F1, and the facial posture expression coefficient [p′ 0 ,b′ 0 ], facial expression coefficient of the audio frame [p′ 0 , b′ 1 ]The three inputs are sent to the expression facial network model Exp2Face to generate a spoken face image F2, and a digital voice-over video is generated based on the spoken face image F2.
[0126] See also Figure 2In order to further improve the matching degree between the oral face image and the audio, the local enhancement model Real-ESRGAN is used to super-resolution the oral face image F2 to obtain the oral face image F3 of the same size as the face area image F. The facial enhancement model GFPGAN is used to enhance the teeth area of the oral face image F3 to obtain the final oral face image F4. The current image I face area image F1 is updated to the oral face image F4. After the face area images of each image frame are updated, they are combined with the audio to generate a digital oral video, and the oral video is output.
[0127] The facial expression model can be pre-trained. For example, see Fig.11 and Fig.12 , taking the use of NVIDIA's motion face video generation network model as the skeleton to build an expression face model as an example.
[0128] Training the expression facial model includes:
[0129] S1101, obtaining a second training sample, where the second training sample includes a second training image and a second training audio;
[0130] In step S1101, a second training image and a second training audio are used as second training samples. The second training image can be obtained by using a face image set.
[0131] S1102, using a face reconstruction network model, reconstructing the face of the second training image to obtain a first face expression coefficient, a first face posture coefficient, and a first face shape base coefficient, and obtaining a first face key point according to the first face expression coefficient, the first face posture coefficient, and the first face shape base coefficient;
[0132] The second training image is reconstructed using the face reconstruction network model Re to obtain a reconstruction result, including a first face shape basis coefficient ID, a first face posture coefficient pose and a first face expression coefficient exp, and the reconstruction result is converted into face key points as the original face key points.
[0133] S1103, using the sound-generated expression model to generate a second facial expression coefficient according to the second training audio, and obtaining a second facial key point according to the second facial expression coefficient, the corresponding first facial posture coefficient and the corresponding first facial shape basis coefficient;
[0134] The audio is subjected to the above-mentioned audio-generated expression model (Audio2Exp) to obtain the second facial expression coefficient, and the first facial posture coefficient pose and the first facial shape base coefficient ID obtained by combining the original image reconstruction are used to obtain the second facial key points as the driving facial key points. Among them, the corresponding first facial posture coefficient and the corresponding first facial shape base coefficient refer to the timestamp correspondence of the audio frame of the second training audio corresponding to the second facial expression coefficient.
[0135] S1104, training an expression facial model based on the first facial key points, the second facial key points and the second training image.
[0136] The loss function and training techniques provided in the motion face video generation network model training method can be used to train the model using the first face key points and the second face key points.
[0137] refer to Fig.13 The exemplary embodiment of the present disclosure further provides an electronic device, including:
[0138] The acquisition module 1301 is used to acquire the spoken audio and the spoken source video;
[0139] The first generating module 1302 is used to generate the facial expression coefficient of each audio frame in the spoken audio by using the sound-generated expression model;
[0140] The second generation module 1303 is used to use the expression facial model to modify the mouth shape of the face image of each corresponding video frame in the spoken audio source video based on the facial expression coefficient of each audio frame in the spoken audio, so as to generate spoken images corresponding to each face image, and generate spoken videos according to each spoken image.
[0141] The acquisition module 1301 is used to obtain the spoken audio and the spoken source video, and is specifically used to: obtain the spoken text; use the Vincent audio model to generate the spoken audio according to the spoken text, wherein the Vincent audio model is trained based on the audio recording of the reproduced character and the text corresponding word for word to the audio recording.
[0142] The acquisition module 1301 is used to obtain the spoken-word source video and is specifically used for: obtaining the video of the reproduced character; generating multiple spoken-word action video clips based on the video of the reproduced character; combining several spoken-word action video clips among the multiple spoken-word action video clips and the inter-frame transition videos between adjacent spoken-word action video clips to form the spoken-word source video.
[0143] In one embodiment, some spoken action video clips among the plurality of spoken action video clips are extracted based on the duration of the spoken video.
[0144] In one embodiment, the inter-frame transition video between adjacent spoken action video segments is obtained based on the difference between action frames.
[0145] The acquisition module 1301 is used to generate multiple spoken action video clips based on the reproduced character video, and is specifically used to: obtain the human body posture data, facial posture data and facial key point data of each video frame of the reproduced character video; extract the action video clips of the reproduced character video based on the human body posture data to obtain multiple first action video clips; cut the first action video clips whose facial posture angles are greater than the preset angle value based on the facial posture data to obtain multiple second action video clips; search for the first and last similar frames of each second action video clip based on the human body posture data, facial posture data and facial key point data, and crop the intermediate video between the first and last similar frames to obtain the spoken action video clip.
[0146] In one embodiment, the audio-generated expression model includes an audio coding network and an audio semantic feature to expression mapping network, the audio coding network is used to encode the audio features of the audio frame of the spoken audio, and the audio semantic feature to expression mapping network is used to map the audio features to facial expressions to obtain facial expression coefficients;
[0147] In one embodiment, the device also includes a first training model for training the sound-generated expression model; when the first training model is used to train the sound-generated expression model, it is specifically used to: obtain a first training sample, the first training sample includes a first training image and a first training audio; use the first training sample to train the sound-generated expression model based on a target loss function; wherein the target loss function includes a facial expression coefficient loss and a mouth facial key point loss, the facial expression coefficient loss and the mouth facial key point loss are determined based on a target facial expression coefficient, the target facial expression coefficient is the facial expression coefficient generated by the face reconstruction network model after reconstructing the face of the target training image, and the target training image is a lip-synthesized image of the first training image under the first training audio.
[0148] In one embodiment, the expression-generated facial model is a network model generated by a moving face video; the device also includes a second training model for training the expression-generated facial model.
[0149] The second training model, when used to train the expression-generated facial model, is specifically used to: obtain a second training sample, the second training sample includes a second training image and a second training audio; use a face reconstruction network model to reconstruct the face of the second training image to obtain a first face expression coefficient, a first face posture coefficient and a first face shape base coefficient, and obtain a first face key point based on the first face expression coefficient, the first face posture coefficient and the first face shape base coefficient; use an audio-generated expression model to generate a second face expression coefficient based on the second training audio, and obtain a second face key point based on the second face expression coefficient, the corresponding first face posture coefficient and the corresponding first face shape base coefficient; train the expression-generated facial model based on the first face key point, the second face key point and the second training image;
[0150] In one embodiment, the second generation module 1303 is used to modify the mouth shape of the face image of each corresponding video frame in the oral broadcast source video based on the facial expression coefficient of each audio frame in the oral broadcast audio to generate the oral broadcast image corresponding to each face image, specifically for:
[0151] Based on the facial posture expression coefficients of each audio frame in the spoken audio and the facial posture expression coefficients of the facial images of each corresponding video frame in the spoken source video, the lip shape of the facial images of each corresponding video frame in the spoken source video is modified, wherein the facial posture expression coefficients of the audio frame include the facial expression coefficients of the audio frame and the posture coefficients of the facial images of the corresponding video frame.
[0152] An exemplary embodiment of the present disclosure further provides an electronic device, including: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to the embodiment of the present disclosure.
[0153] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.
[0154] refer to Fig.14 The exemplary embodiment of the present disclosure further provides a computer program product 1400, comprising a computer program 1401, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiment of the present disclosure.
[0155] refer to Fig.15, a block diagram of an electronic device 1500 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. Such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0156] The electronic device 1500 includes a computing unit 1501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1502 or a computer program loaded from a storage unit 1508 into a random access memory (RAM) 1503. In the RAM 1503, various programs and data required for the operation of the device can also be stored. The computing unit 1501, the ROM 1502, and the RAM 1503 are connected to each other via a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.
[0157] Multiple components in the electronic device 1500 are connected to the I / O interface 1505, including: an input unit 1506, an output unit 1507, a storage unit 1508, and a communication unit 1509. The input unit 1506 can be any type of device that can input information to the electronic device 1500, and the input unit 1506 can receive input digital or character information, and generate key signal input related to the user settings and / or function control of the electronic device. The output unit 1507 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1508 can include but is not limited to a disk, an optical disk. The communication unit 1509 allows the electronic device 1500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0158] The computing unit 1501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1501 performs the various methods and processes described above. For example, in some embodiments, the method of the embodiment of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1500 via the ROM 1502 and / or the communication unit 1509. In some embodiments, the computing unit 1501 may be configured to perform the method of the embodiment of the present disclosure in any other appropriate manner (e.g., by means of firmware).
[0159] In the above, a method, device, equipment, storage medium and program product for generating a digital human voice broadcast video according to an embodiment of the present disclosure are described with reference to the accompanying drawings. According to a method, device, equipment, storage medium and program product for generating a digital human voice broadcast video according to an embodiment of the present disclosure, a facial expression coefficient of each audio frame in the oral broadcast audio is generated by using an audio-generated expression model; the mouth shape of the face image of each corresponding video frame in the oral broadcast source video is modified based on the facial expression coefficient of each audio frame in the oral broadcast audio by using an expression-generated facial model to generate an oral broadcast image corresponding to each face image, and an oral broadcast video is generated based on each oral broadcast image. A two-stage synthesis method of audio to facial expression coefficient and facial expression coefficient to mouth shape (face) is used, and the mouth shape synthesis based on the facial expression coefficient reduces the sensitivity of the oral broadcast video to the identity of the character, so that the mouth shape content of the synthesized oral broadcast video is more consistent with the actual speaking mouth shape. At the same time, the computing power and time cost of online fine-tuning training can also be saved.
[0160] In one embodiment of the present disclosure, based on the facial posture expression coefficients of each audio frame in the spoken audio and the facial posture expression coefficients of the facial images of each corresponding video frame in the spoken source video, the lip shape of the facial images of each corresponding video frame in the spoken source video is modified, so that the relationship (for example, the difference) between the facial posture expression coefficients of each audio frame in the spoken audio and the facial expression coefficients of the facial images of each corresponding video frame in the spoken source video can affect the lip shape of the facial image of the video frame, thereby removing the restriction that the spoken source video must be in a closed state.
[0161] In one embodiment of the present disclosure, based on the human body posture data, the action video clips of the reproduced character video are extracted to obtain multiple first action video clips; based on the facial posture data, the first action video clips with facial posture angles greater than the preset angle value are cut to obtain multiple second action video clips; based on the human body posture data, the facial posture data and the facial key point data, the first and last similar frames of each second action video clip are searched, and the middle video between the first and last similar frames is cropped to obtain the spoken action video clip, and the spoken video clip is extracted based on the spoken action video clip to obtain the spoken source video. The problem of single body action in the produced video content is improved by classifying and extracting the action clips of the reproduced character video, and then randomly extracting and combining, and interpolating the action images between frames.
[0162] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.
[0163] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.
[0164] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.
[0165] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.
[0166] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.
[0167] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0168] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A method for generating digital human broadcast video, It is characterized in that include: Get the spoken audio and the spoken source video; Generate a facial expression coefficient for each audio frame in the spoken audio using a sound-generated expression model; By utilizing the expression-generated facial model, based on the facial expression coefficient of each audio frame in the spoken audio, the lip shape of the facial image of each corresponding video frame in the spoken source video is modified to generate spoken images corresponding to each of the facial images, and a spoken video is generated based on each of the spoken images.
2. The method according to claim 1, It is characterized in that The step of obtaining the oral audio includes: Get the spoken text; The spoken audio is generated according to the spoken text using a Vincent audio model, wherein the Vincent audio model is trained based on an audio recording of a replicated character and a text corresponding word for word to the audio recording.
3. The method according to claim 1, It is characterized in that The obtaining of the oral source video includes: Get the video of the person being reproduced; Generating a plurality of spoken action video clips according to the reproduced character video; Combine several spoken action video segments among the multiple spoken action video segments and inter-frame transition videos between adjacent spoken action video segments to form a spoken action source video.
4. The method according to claim 3, It is characterized in that A plurality of spoken action video segments among the plurality of spoken action video segments are extracted based on the duration of the spoken video; or, The inter-frame transition video between adjacent oral action video clips is obtained based on the action frame difference; or, The step of generating a plurality of spoken action video clips according to the reproduced character video includes: Obtaining human body posture data, human face posture data and human face key point data of each video frame of the reproduced person video; Extracting action video clips of the reproduced character video according to the human body posture data to obtain a plurality of first action video clips; According to the face posture data, cutting the first action video segments in which the face posture angle is greater than a preset angle value to obtain a plurality of second action video segments; According to the human body posture data, the facial posture data and the facial key point data, the first and last similar frames of each of the second action video clips are searched, and the middle video between the first and last similar frames is cropped to obtain the spoken action video clip.
5. The method according to claim 1, It is characterized in that The audio-generated expression model includes an audio coding network and an audio semantic feature to expression mapping network, wherein the audio features of the audio frame of the spoken audio are encoded by the audio coding network, and the audio semantic feature to expression mapping network is used to map the audio features to facial expressions to obtain the facial expression coefficients; and / or, Training the sound-generated expression model includes: Acquire a first training sample, where the first training sample includes a first training image and a first training audio; Using the first training sample, based on a target loss function, training the voice-generated expression model; The target loss function includes a facial expression coefficient loss and a mouth facial key point loss, the facial expression coefficient loss and the mouth facial key point loss are determined based on a target facial expression coefficient, the target facial expression coefficient is a facial expression coefficient generated after a face reconstruction network model reconstructs a target training image face, and the target training image is a lip-synthesized image of the first training image under the first training audio; and / or, The expression facial model is a network model generated by moving face video; Training the expression facial model includes: Acquire a second training sample, where the second training sample includes a second training image and a second training audio; Reconstructing the face of the second training image using a face reconstruction network model to obtain a first face expression coefficient, a first face posture coefficient, and a first face shape base coefficient, and obtaining a first face key point according to the first face expression coefficient, the first face posture coefficient, and the first face shape base coefficient; Using the sound-generated expression model, generating a second facial expression coefficient according to the second training audio, and obtaining a second facial key point according to the second facial expression coefficient, the corresponding first facial posture coefficient and the corresponding first facial shape basis coefficient; The expression facial model is trained based on the first facial key points, the second facial key points and the second training image.
6. The method according to claim 1, It is characterized in that The method of modifying the mouth shape of the face image of each corresponding video frame in the oral broadcast source video based on the face expression coefficient of each audio frame in the oral broadcast audio to generate the oral broadcast image corresponding to each face image includes: Based on the facial posture expression coefficients of each audio frame in the spoken audio and the facial posture expression coefficients of the facial images of each corresponding video frame in the spoken source video, the lip shape of the facial images of each corresponding video frame in the spoken source video is modified, wherein the facial posture expression coefficients of the audio frame include the facial expression coefficients of the audio frame and the posture coefficients of the facial images of the corresponding video frame.
7. A digital voice broadcast video generation device, It is characterized in that include: The acquisition module is used to obtain the spoken audio and the spoken source video; A first generating module is used to generate a facial expression coefficient of each audio frame in the spoken audio by using a sound-generated expression model; The second generation module is used to use the expression facial model to modify the mouth shape of the facial image of each video frame in the oral source video based on the facial expression coefficient of each audio frame in the oral audio, so as to generate oral images corresponding to each of the facial images, and generate oral videos according to each of the oral images.
8. An electronic device comprising a memory, a processor and a computer program stored in the memory, It is characterized in that The processor executes the computer program to implement the steps of any one of the methods of claim 1 to claim 6.
9. A computer-readable storage medium having a computer program / instructions stored thereon, It is characterized in that When the computer program / instruction is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.
10. A computer program product comprising a computer program / instructions, It is characterized in that When the computer program / instruction is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.
Citation Information
Cited By
Voice-driven video synthesis method and device, and storage medium
CN121037652A
Digital human interaction method and system based on multi-mode sensing intelligent action switching
CN121050590A
