Emotion synchronization 2D digital human model training method and device

Abstract

The invention provides an emotion-synchronized 2D digital human model training method and device, and relates to the technical field of 2D digital humans. According to the method, an audio and video data set containing various expressions and emotions is generated, a LivePortrait model, a sound cloning technology and a hallo2 model are utilized, accurate alignment of a digital population type and an input audio is achieved, and the emotion-synchronized 2D digital human model is obtained. And meanwhile, the facial expression consistent with the audio emotion is kept. Meanwhile, an emotion classifier is used in a model training stage, so that the reasoning speed is increased; an expression loss function and a mouth loss function are designed, the model training process is optimized, and it is ensured that the expression and mouth of the digital human can accurately reflect the emotion of the input audio. The method aims at improving the expression naturalness and reality sense of a 2D digital person in voice communication and ensuring that the expression of the digital person is synchronous with the emotion in audio.

Description

Technical Field

[0001] The present invention relates to the technical field of 2D digital human, and in particular to a method and device for training a 2D digital human model with emotional synchronization. Background Art

[0002] With the development of computer graphics and artificial intelligence technology, 2D digital human technology has become an important application in the fields of virtual assistants, online education, entertainment interaction, etc. 2D digital humans provide users with a more natural and friendly interactive experience by simulating real people's facial expressions and lip shapes. However, existing 2D digital human technology has certain limitations in terms of expression synchronization.

[0003] For example, when the input audio is mostly happy, the expression displayed by the digital human will be happy accordingly. However, since human emotions are often changeable, some excitement, sadness, etc. may appear in the audio from time to time. That is, the existing technology often cannot accurately capture the emotional nuances in the entire audio, which leads to a mismatch between the digital human's expression and the audio emotion, thus affecting the user experience. In addition, the existing 2D digital human model has difficulty in achieving natural expression and lip transition when dealing with silent or weak sound periods.

[0004] In view of this, this application is filed. Summary of the invention

[0005] Traditional 2D digital human models mainly rely on reference frame technology, which drives the expression changes of digital humans through a series of pre-defined expression frames. Although this technology has achieved certain results in synchronizing lip movements with audio, it is unable to cope with complex emotional expressions. Specifically, reference frame technology often causes the expression changes of digital humans to be too rigid and lack natural transitions, and it is difficult to adjust the expression in real time according to the emotional changes in the audio.

[0006] The present invention provides an emotion-synchronized 2D digital human model training method and device, which can at least partially improve the above-mentioned problems.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for training an emotion-synchronized 2D digital human model, comprising:

[0009] Obtaining original training videos and original audio, and preprocessing the original training videos and original audio to generate a plurality of training videos with different emotions and different expressions, wherein the facial expression and audio emotion of each training video are consistent;

[0010] Pre-training the 2D digital human model according to the training video to generate a training image, wherein the pre-training process includes: extracting emotions and performing feedback processing on the audio with emotions in the training video, and performing decoding processing based on the extracted audio embedding vector and face embedding vector;

[0011] Calculating a loss function between the training image and the real image to generate a loss value, wherein the real image is a video frame of the training video, and the loss function includes an expression loss function and a lip shape loss function;

[0012] When it is determined that the loss value reaches a preset value, the training is terminated to obtain a trained 2D digital human model.

[0013] The present invention also provides an emotion-synchronized 2D digital human model training device, which comprises:

[0014] A training video generating unit is used to obtain an original training video and an original audio, and pre-process the original training video and the original audio to generate a plurality of training videos with different emotions and different expressions, wherein the facial expression and the audio emotion of each training video are consistent;

[0015] A pre-training unit, configured to perform pre-training processing on the 2D digital human model according to the training video to generate a training image, wherein the pre-training processing includes: performing emotion extraction and feedback processing on the audio with emotions in the training video, and performing decoding processing based on the extracted audio embedding vector and face embedding vector;

[0016] A loss function unit, used to calculate a loss function between the training image and a real image to generate a loss value, wherein the real image is a video frame of the training video, and the loss function includes an expression loss function and a lip shape loss function;

[0017] The model generation unit is used to end the training and obtain a trained 2D digital human model when it is determined that the loss value reaches a preset value.

[0018] In summary, the emotion-synchronized 2D digital human model training method aims to improve the naturalness and realism of the 2D digital human's expression in voice communication, and ensure that the digital human's expression is synchronized with the emotion in the audio. This method generates audio and video datasets containing a variety of expressions and emotions, and uses the LivePortrait model and sound cloning technology, as well as the hallo2 model, to achieve accurate alignment of the digital human's mouth shape with the input audio, while maintaining facial expressions consistent with the audio emotions. At the same time, the 2D digital human model structure is improved to enhance the extraction and expression of emotional information. In the model training stage, the emotion classifier participates in the calculation, but not in the inference stage, thereby improving the inference speed. In addition, the expression loss function and the lip shape loss function are designed to optimize the model training process to ensure that the digital human's expression and lip shape can accurately reflect the emotion of the input audio.

[0019] The emotion-synchronized 2D digital human model training method allows the digital human to more naturally display expressions and lip shapes that match the audio emotions when speaking, and even when there is no sound, the digital human can naturally close its mouth and maintain natural expressions and mouth movements. This method is applicable to multiple fields such as virtual assistants, online education, and entertainment interaction, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the emotion-synchronized 2D digital human model training method provided by the first embodiment of the present invention;

[0021] Figure 2 It is a module schematic diagram of the emotion-synchronized 2D digital human model training method provided by the first embodiment of the present invention;

[0022] Figure 3 It is a module schematic diagram of the emotion-synchronized 2D digital human model training device provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0024] refer to Figure 1 , Figure 2 As shown, the first embodiment of the present invention discloses a method for training a 2D digital human model with emotional synchronization, which can be executed by a 2D digital human model training device with emotional synchronization (hereinafter referred to as a training device), and in particular, by one or more processors in the training device to implement the following method:

[0025] S1, obtaining an original training video and an original audio, and preprocessing the original training video and the original audio to generate a plurality of training videos with different emotions and different expressions, wherein the facial expression and the audio emotion of each training video are consistent;

[0026] Specifically, step S2 includes: obtaining original audio, and based on the original audio, performing cloning processing using a preset sound cloning technology to obtain a plurality of audio data with different emotions;

[0027] Obtain an original training video, and use the facial expression driving model LivePortrait to drive and process the original training video to obtain video data of multiple different expressions;

[0028] The lip-driven model hallo2 is controlled to align each video data with different expressions and each audio data with different emotions respectively, so as to align the lip shape of the video data and the audio data without changing the expression of the video data, and generate multiple training videos.

[0029] Preferably, the voice cloning technology is a CosyVoice model.

[0030] In this embodiment, LivePortrait is a facial expression driven model, and LivePortrait model can be used to drive videos with various expressions; at the same time, the voice cloning technology can be used to clone audio with various emotions, wherein the voice cloning technology adopts CosyVoice model, which can convert text into audio with different timbres and emotions. Hallo2 is a lip-sync driven model, the function of which is to align the lip shape of the original video with the input audio, but does not change the expression of the original video.

[0031] Therefore, before training the model, it is necessary to control the emotion of the input audio and the expression of the character in the original video to align; finally, the generated result can be obtained with the lip shape aligned and the expression of the character consistent with the emotion contained in the input audio. In actual operation, using the same original video, you can synthesize many training videos based on different emotional input audio and different expression-driven videos. Repeating this step, you can get lip-sync training materials with consistent expressions and audio emotions.

[0032] S2, performing pre-training processing on the 2D digital human model according to the training video to generate a training image, wherein the pre-training processing includes: performing emotion extraction and feedback processing on the audio with emotions in the training video, and performing decoding processing based on the extracted audio embedding vector and face embedding vector;

[0033] Preferably, the 2D digital human model comprises an audio encoder, an emotion classifier, an image encoder and an image decoder.

[0034] Specifically, step S2 includes: obtaining a video frame to be driven and an audio with emotions from the training video as input of the 2D digital human model;

[0035] Using the audio encoder to extract and process the audio with emotion to obtain an audio embedding vector audioembedding;

[0036] Input the audio embedding vector audio embedding into the emotion classifier, extract the emotion information of the audio embedding vector audio embedding, wherein the emotion information includes five categories: happiness, sadness, surprise, fear and anger, and use five sequence numbers [1,0,0,0,0], [0,1,0,0,0], [0,0,1,0,0], [0,0,0,1,0], [0,0,0,0,1] to distinguish them;

[0037] The emotional information gradient of the audio embedding vector audio embedding extracted by the loss function of the emotion classifier is fed back to the audio encoder to obtain the audio embedding vector audioembedding with emotional features.

[0038] The loss function formula of the sentiment classifier is: H(p,q)=-∑p(x)logq(x), where q is the probability distribution predicted by the sentiment classifier, p is the true probability distribution, q(x) is the probability distribution predicted by the x-th sentiment classifier, and p(x) is the x-th true probability distribution.

[0039] Inputting the video frame to be driven into the image encoder for extraction processing to obtain a face embedding vector face embedding;

[0040] The image encoder is used to decode the face embedding vector face embedding and the audio embedding vector audio embedding with emotional features to generate a training image.

[0041] In this embodiment, the input of the 2D digital human model consists of video frames (images) to be driven and audio with emotions; wherein the entire model structure includes an audio encoder, an emotion classifier, an image encoder and an image decoder.

[0042] In this embodiment, the audio encoder is responsible for extracting the audio embedding vector audio embedding. The emotion classifier is mainly used to enhance the audio encoder's extraction of emotional information in the audio. It only participates in the calculation in the model training stage, and does not participate in the calculation in the actual use stage, thereby improving the reasoning speed.

[0043] Specifically, in order to enhance the audio encoder's extraction of emotional information, a sentiment classifier is added behind it, and the value of the sentiment classifier's loss function H(p,q)=-∑p(x)logq(x) is fed back by gradient, so that the audio encoder learns the emotional features.

[0044] In this embodiment, the main structure of the 2D digital human model of the image encoder and the image decoder is used to extract image embedding and generate a 2D digital human image. In simple terms, it is to control the image decoder to decode the face embedding vector face embedding extracted by the image encoder and the audio embedding vector audio embedding extracted by the audio encoder to generate an image.

[0045] S3, calculating a loss function between the training image and the real image to generate a loss value, wherein the real image is a video frame of the training video, and the loss function includes an expression loss function and a lip shape loss function;

[0046] Specifically, step S3 includes: the mathematical expression of the expression loss function is: Lp=cross_entropy(perceptual(F), f), wherein cross_entropy() is a cross entropy loss function, perceptual is an expression sensor, F is a face region output by an image decoder, and f is a true expression sequence;

[0047] The mathematical expression of the mouth shape loss function is: Lmouth=L1(M, m), where M is the mouth area output by the image decoder, m is the mouth area of ​​the real image, and L1 is the distance function between vectors, which is used to measure the absolute error between two vectors;

[0048] The real image has a lip shape consistent with the input audio, and an expression consistent with the emotion of the input audio.

[0049] In this embodiment, the loss function between the generated image and the real image is calculated, wherein the real image is a video frame in the synthetic training video, and the real image at this time not only has a lip shape consistent with the input audio, but also has an expression consistent with the emotion of the input audio, which is synthesized by step S1. This method designs two loss functions, an expression loss function and a lip shape loss function, which are used to control the learning of the decoder.

[0050] S4, when it is determined that the loss value reaches a preset value, the training is terminated to obtain a trained 2D digital human model.

[0051] Preferably, it also includes:

[0052] Get the current video frame to be processed;

[0053] The current video frame to be processed is used as the input of the trained 2D digital human model, and the lip shape of the current video frame to be processed is adjusted to obtain the lip shape corresponding to the audio of the video frame, and generate a video.

[0054] In this embodiment, the method abandons the reference frame strategy of the traditional 2D model and adopts a reference frame strategy that is more consistent with reasoning. In actual use, each reference frame is the current frame, and the model adjusts the lip shape of the current frame (reference frame) to generate a new lip shape corresponding to the input audio.

[0055] In short, the emotion-synchronized 2D digital human model training method is based on the reasoning process, designs a training scheme that is more in line with practical applications, and adopts a reference frame strategy that is more consistent with reasoning. The reference frame image is changed to the current frame with changed expression and lip shape, that is, the reference frame and the current frame are the same frame at the same moment in the original video, but the expression and lip shape have changed through the processing of step one. Through training, the model can generate more realistic character images based on different audio and the emotions contained in the audio.

[0056] In summary, compared with the prior art, the emotion-synchronized 2D digital human model training method has the following advantages:

[0057] 1. Using voice cloning technology and emotional synchronization training sets, digital humans can show richer emotional expressions. When used in practice, the model is not limited to basic happiness and sadness, but can also handle complex emotional states;

[0058] 2. The use of sentiment classifiers is omitted in the time stage, which reduces the computational complexity and improves the inference speed and efficiency of the model operation;

[0059] 3. Different from the existing schemes using reference frames, this method adopts a new reference frame strategy and uses the current frame as the reference value, so that the model can adapt to more dynamic changes and expressions of different emotions, improving the adaptability and flexibility of the model in practical applications;

[0060] 4. The more natural and realistic expression synchronization technology provided by this method can significantly improve the user experience when interacting with digital humans, especially in application scenarios such as virtual assistants, online education, and entertainment interactions.

[0061] See also Figure 3 The second embodiment of the present invention provides an emotion-synchronized 2D digital human model training device, which includes:

[0062] The training video generating unit 201 is used to obtain the original training video and the original audio, and pre-process the original training video and the original audio to generate a plurality of training videos with different emotions and different expressions, wherein the facial expression and the audio emotion of each training video are consistent;

[0063] A pre-training unit 202 is used to perform pre-training processing on the 2D digital human model according to the training video to generate a training image, wherein the pre-training processing includes: performing emotion extraction and feedback processing on the audio with emotions in the training video, and performing decoding processing based on the extracted audio embedding vector and face embedding vector;

[0064] A loss function unit 203, used to calculate a loss function between the training image and the real image to generate a loss value, wherein the real image is a video frame of the training video, and the loss function includes an expression loss function and a lip shape loss function;

[0065] The model generation unit 204 is used to terminate the training and obtain a trained 2D digital human model when it is determined that the loss value reaches a preset value.

[0066] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for training a 2D digital human model with emotional synchronization, characterized in that: include: Obtaining original training videos and original audio, and preprocessing the original training videos and original audio to generate a plurality of training videos with different emotions and different expressions, wherein the facial expression and audio emotion of each training video are consistent; Pre-training the 2D digital human model according to the training video to generate a training image, wherein the pre-training process includes: extracting emotions and performing feedback processing on the audio with emotions in the training video, and performing decoding processing based on the extracted audio embedding vector and face embedding vector; Calculating a loss function between the training image and the real image to generate a loss value, wherein the real image is a video frame of the training video, and the loss function includes an expression loss function and a lip shape loss function; When it is determined that the loss value reaches a preset value, the training is terminated to obtain a trained 2D digital human model.

2. The emotionally synchronized 2D digital human model training method according to claim 1, characterized in that: The original training video and the original audio are obtained, and the original training video and the original audio are preprocessed to generate a plurality of training videos with different emotions and different expressions, specifically: Acquire original audio, and perform cloning processing based on the original audio using a preset sound cloning technology to obtain multiple audio data with different emotions; Obtain an original training video, and use the facial expression driving model LivePortrait to drive and process the original training video to obtain video data of multiple different expressions; The lip-driven model hallo2 is controlled to align each video data with different expressions and each audio data with different emotions respectively, so as to align the lip shape and audio data of the video data without changing the expression of the video data, and generate multiple training videos.

3. The emotionally synchronized 2D digital human model training method according to claim 2, characterized in that: The voice cloning technology is the CosyVoice model.

4. The emotionally synchronized 2D digital human model training method according to claim 1, characterized in that: The 2D digital human model includes an audio encoder, an emotion classifier, an image encoder and an image decoder.

5. The emotionally synchronized 2D digital human model training method according to claim 4, characterized in that: The emotional audio in the training video is subjected to emotion extraction and feedback processing, specifically: Acquire the video frames to be driven and the audio with emotions from the training video as inputs of the 2D digital human model; Using the audio encoder to extract and process the audio with emotion to obtain an audio embedding vector audioembedding; Input the audio embedding vector audio embedding into the emotion classifier, extract the emotion information of the audio embedding vector audio embedding, wherein the emotion information includes five categories: happiness, sadness, surprise, fear and anger, and use five sequence numbers [1,0,0,0,0], [0,1,0,0,0], [0,0,1,0,0], [0,0,0,1,0], [0,0,0,0,1] to distinguish them; The emotional information gradient of the audio embedding vector audio embedding extracted by the loss function of the emotion classifier is fed back to the audio encoder to obtain the audio embedding vector audio embedding with emotional features.

6. The emotion-synchronized 2D digital human model training method according to claim 5, characterized in that: The loss function formula of the sentiment classifier is: H(p,q)=-∑p(x)logq(x), where q is the probability distribution predicted by the sentiment classifier, p is the true probability distribution, q(x) is the probability distribution predicted by the x-th sentiment classifier, and p(x) is the x-th true probability distribution.

7. The emotion-synchronized 2D digital human model training method according to claim 5, characterized in that: Decoding is performed based on the extracted audio embedding vector and face embedding vector, specifically: Inputting the video frame to be driven into the image encoder for extraction processing to obtain a face embedding vector face embedding; The image encoder is used to decode the face embedding vector face embedding and the audio embedding vector audio embedding with emotional features to generate a training image.

8. The emotionally synchronized 2D digital human model training method according to claim 1, characterized in that: Calculate the loss function between the training image and the real image to generate a loss value, specifically: The mathematical expression of the expression loss function is: Lp=cross_entropy(perceptual(F), f), where cross_entropy() is the cross entropy loss function, perceptual is the expression sensor, F is the face area output by the image decoder, and f is the real expression sequence; The mathematical expression of the mouth shape loss function is: Lmouth=L1(M, m), where M is the mouth area output by the image decoder, m is the mouth area of ​​the real image, and L1 is the distance function between vectors, which is used to measure the absolute error between two vectors; The real image has a lip shape consistent with the input audio, and an expression consistent with the emotion of the input audio.

9. The emotionally synchronized 2D digital human model training method according to claim 1, characterized in that: Also includes: Get the current video frame to be processed; The current video frame to be processed is used as the input of the trained 2D digital human model, and the lip shape of the current video frame to be processed is adjusted to obtain the lip shape corresponding to the audio of the video frame, and generate a video.

10. An emotionally synchronized 2D digital human model training device, characterized in that: include: A training video generating unit is used to obtain an original training video and an original audio, and pre-process the original training video and the original audio to generate a plurality of training videos with different emotions and different expressions, wherein the facial expression and the audio emotion of each training video are consistent; A pre-training unit, configured to perform pre-training processing on the 2D digital human model according to the training video to generate a training image, wherein the pre-training processing includes: performing emotion extraction and feedback processing on the audio with emotions in the training video, and performing decoding processing based on the extracted audio embedding vector and face embedding vector; A loss function unit, used to calculate a loss function between the training image and a real image to generate a loss value, wherein the real image is a video frame of the training video, and the loss function includes an expression loss function and a lip shape loss function; The model generation unit is used to end the training and obtain a trained 2D digital human model when it is determined that the loss value reaches a preset value.

Images