Virtual human expression synthesis method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a virtual human expression synthesis method and device, electronic equipment and a storage medium, and the method comprises the steps: extracting the audio features of a reference audio, and extracting the face features of a face reference image based on a PDFGC encoder; inputting the face features and the audio features into a trained generative model based on a probability diffusion process to obtain a feature sequence output by the generative model; and inputting the feature sequence and the human face reference image into a PDFGC decoder to obtain a video which is output by the decoder, corresponds to the reference audio and contains human face expressions. According to the virtual human expression synthesis method and device, the electronic equipment and the storage medium provided by the invention, richer and more diversified expressions can be generated based on the generation model of the probability diffusion process, and the expressions are more expressive.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for synthesizing virtual human expressions. Background Art

[0002] Virtual human expression synthesis technology is a technology that digitizes human facial expressions and applies them to virtual humans or digital humans. By simulating human facial expressions, this technology enables virtual characters to express rich emotions in virtual environments, thereby enhancing the realism and interactivity of virtual characters.

[0003] In the related art, facial expression control is achieved by injecting facial expression conditions into the lip expert module. The expression conditions are extracted from the reference style video and act on the lip shape, which makes it difficult to achieve high-precision control of the overall expression. In addition, this method requires a specific reference expression driving video, and the reasoning process is complicated. Summary of the invention

[0004] The present invention provides a method, device, electronic device and storage medium for synthesizing virtual human expressions, which are used to solve the defect of poor control accuracy of virtual human overall expressions in the prior art.

[0005] The present invention provides a method for synthesizing virtual human expressions, comprising: Extracting audio features of reference audio, and extracting facial features of reference face images based on PDFGC encoder; Inputting the facial features and the audio features into a trained generative model based on a probability diffusion process to obtain a feature sequence output by the generative model; The feature sequence and the human face reference image are input into a PDFGC decoder, and a video containing human facial expressions corresponding to the reference audio is output by the decoder.

[0006] According to the virtual human expression synthesis method provided by the present invention, the facial features and the audio features are input into a trained generation model based on a probability diffusion process to obtain a feature sequence output by the generation model, including: Acquiring a control signal, wherein the control signal includes an expression control signal and / or a sight control signal; The facial features, the audio features, and the control signal are input into the generation model to obtain a feature sequence output by the generation model.

[0007] According to the virtual human expression synthesis method provided by the present invention, the facial features, the audio features, and the control signal are input into the generation model to obtain the feature sequence output by the generation model, including: Inputting the facial features and the audio features into the feature extraction module of the generation model to obtain the intermediate features output by the feature extraction module; The intermediate features and the control signal are input into the cross-attention module of the generation model to obtain the feature sequence output by the cross-attention module.

[0008] According to the virtual human expression synthesis method provided by the present invention, the step of inputting the intermediate features and the control signal into the cross attention module of the generation model to obtain the feature sequence output by the cross attention module includes: determining a bias characteristic of the control signal; The intermediate features and the bias features are input into the cross-attention module to obtain the conditional feature sequence output by the cross-attention module.

[0009] According to the virtual human expression synthesis method provided by the present invention, the obtaining of the control signal comprises: Based on the PDFGC encoder, extracting each expression feature corresponding to each expression reference image, and performing linear interpolation smoothing processing on each expression feature to obtain an expression control signal; And / or, extracting facial marker points in the gaze reference image, and determining the gaze control signal based on the facial marker points.

[0010] According to the virtual human expression synthesis method provided by the present invention, the determining of the sight line control signal based on the facial markers comprises: Determine the center of gravity of the eye region markers among the facial markers as the sight line center point; The sight line control signal is determined based on the displacement between the eyeball mark point in the face mark point and the sight line center point.

[0011] According to the virtual human expression synthesis method provided by the present invention, the facial features, the audio features, and the control signal are input into the generation model to obtain the feature sequence output by the generation model, including: Inputting the facial features, the audio features, and the control signal into the generation model to obtain a conditional feature sequence output by the generation model; Inputting the facial features and the audio features into the generative model to obtain an unconditional feature sequence output by the generative model; The conditional feature sequence and the unconditional feature sequence are fused to obtain the feature sequence.

[0012] According to the virtual human expression synthesis method provided by the present invention, the training step of the generation model includes: The sample face features and the sample audio features are merged to obtain the original features, and the original features are subjected to noise processing to obtain the noise features of the current time step; Inputting the noise feature and the sample control signal into a denoising module of the initial model to obtain a prediction feature of the current time step output by the denoising module; Based on the difference between the original features and the predicted features at each time step, the initial model is iterated on parameters to obtain the generated model.

[0013] The present invention also provides a virtual human expression synthesis device, comprising: A feature extraction unit, used for extracting audio features of a reference audio, and extracting facial features of a reference facial image based on a PDFGC encoder; A sequence generation unit, used for inputting the facial features and the audio features into a trained generation model based on a probability diffusion process to obtain a feature sequence output by the generation model; The sequence decoding unit is used to input the feature sequence and the human face reference image into a PDFGC decoder to obtain a video containing human face expressions corresponding to the reference audio output by the decoder.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned virtual human expression synthesis methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for synthesizing virtual human expressions as described above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for synthesizing virtual human expressions as described above is implemented.

[0017] The virtual human expression synthesis method, device, electronic device and storage medium provided by the present invention extract the facial features of a more highly realistic and finely controllable facial reference image through a PDFGC encoder; input the facial features and audio features into a trained generative model based on a probability diffusion process to obtain a feature sequence output by the generative model, and finally input the feature sequence and the facial reference image into a PDFGC decoder to obtain a video containing facial expressions corresponding to the reference audio output by the decoder. The generative model based on the probability diffusion process can generate richer and more diverse expressions, and the expressions are more expressive. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the virtual human expression synthesis method provided by the present invention.

[0020] Figure 2 This is one of the flowchart diagrams of step 120 in the virtual human expression synthesis method provided by the present invention.

[0021] Figure 3 This is the second flowchart of step 120 in the virtual human expression synthesis method provided by the present invention.

[0022] Figure 4 This is the third flowchart diagram of step 120 in the virtual human expression synthesis method provided by the present invention.

[0023] Figure 5 It is a schematic diagram of facial landmarks provided by the present invention.

[0024] Figure 6 This is the fourth flowchart of step 120 in the virtual human expression synthesis method provided by the present invention.

[0025] Figure 7 It is a flow chart of the generative model training method provided by the present invention.

[0026] Figure 8 This is the second flow chart of the virtual human expression synthesis method provided by the present invention.

[0027] Fig. 9 It is a structural schematic diagram of a virtual human expression synthesis device provided by the present invention.

[0028] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Thanks to the rapid development of technologies such as deep learning, image processing, and computer vision, virtual human expression synthesis technology can simulate highly realistic virtual facial expressions. This not only improves the expressiveness of virtual characters, but also increases their affinity, making virtual digital characters more natural and real in daily applications.

[0031] Currently, many studies focus on generating emotionless virtual speakers. These methods are mainly driven by audio or reference video. Among them, the reference video drive needs to ensure that the expression and lip shape of the generated virtual person are as consistent as possible with the reference video; while the audio-driven virtual human generation must not only generate accurate lip shape, but also ensure that the expression of the generated virtual person is reasonable with the emotion reflected in the audio.

[0032] In an attempt to solve the problem of controlling emotional expressions, DreamTalk achieves control of expressions by adding a style-aware lip-sync expert module and a style prediction module, but there is still a defect of insufficient control of expression conditions. Related technologies achieve control of expressions by injecting facial expression conditions into the lip-sync expert module. The expression conditions are extracted from the reference style video and act on the lip shape. It is difficult to achieve high-precision control of the most overall expression, and specific reference expression driving videos are required, and the reasoning process is complicated.

[0033] In view of the above problems, an embodiment of the present invention provides a method for synthesizing virtual human expressions. In the method, audio features of reference audio are first extracted, and facial features of a more highly realistic and finely controllable facial reference image are extracted through a PDFGC encoder; the facial features and audio features are input into a trained generative model based on a probability diffusion process to obtain a feature sequence output by the generative model. The generative model based on the probability diffusion process can generate richer expressions with more expressiveness. Finally, the feature sequence and the facial reference image are input into a PDFGC decoder to obtain a video containing facial expressions corresponding to the reference audio output by the decoder.

[0034] The embodiments of the present invention can be applied to scenarios where it is necessary to generate a virtual speaker with rich emotions, such as film and television production, game development, virtual assistants, virtual anchors and virtual customer service, education and training, advertising and marketing, and human-computer interaction. The execution subject of the method can be an electronic device such as a terminal device, a computer, a server, a server cluster, or a specially designed virtual human expression synthesis device, or a virtual human expression synthesis device set in the electronic device, and the virtual human expression synthesis device can be implemented by software, hardware, or a combination of the two.

[0035] Figure 1 FIG. 1 is one of the flow charts of the virtual human expression synthesis method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110: extracting audio features of the reference audio, and extracting facial features of the reference facial image based on the PDFGC encoder.

[0036] Specifically, the reference audio is the audio corresponding to the video containing the virtual human expression to be synthesized, and the reference audio can be a piece of voice data. The reference audio can be obtained through a sound pickup device, which can be a smart phone, a tablet computer, or a smart appliance such as a stereo, a television, and an air conditioner. After the sound pickup device obtains the reference audio through a microphone array, it can also amplify and reduce noise on the reference audio, which is not specifically limited in the embodiment of the present invention.

[0037] The reference audio is analyzed using audio feature extraction algorithms (such as Mel-frequency cepstral coefficients (MFCCs), spectrograms, etc.) to extract audio features that can reflect the audio content and emotions. Audio features usually include the audio's spectrum information, energy distribution, fundamental frequency, etc.

[0038] The face reference image refers to the face image corresponding to the video screen containing the virtual human expression to be synthesized, that is, the face of the virtual human expression is determined based on the face reference image. Here, the face of the virtual human expression can be the same as the face in the face reference image, or it can be a virtual image face generated based on the face in the face reference image.

[0039] The facial features of the face reference image usually include information such as the face outline, facial features, expression texture, etc. The extraction of facial features can be achieved through the pre-trained PDFGC encoder.

[0040] Here, PDFGC (Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis) refers to progressive disentangled representation learning for fine-grained controllable talking head synthesis. Progressive disentangled representation learning is a machine learning method that aims to learn disentangled representations of data in a progressive manner. Disentangled representation means that different features or factors of the data are separated, so that these factors can be controlled and manipulated more flexibly. Fine-grained controllable talking head synthesis refers to a technology that can generate highly realistic and finely controllable talking head synthesis. The pre-trained PDFGC encoder can extract facial features of more highly realistic and finely controllable face reference images.

[0041] Step 120, input the facial features and audio features into the trained generative model based on the probability diffusion process to obtain a feature sequence output by the generative model.

[0042] Specifically, the facial features and audio features extracted in step 110 are fused, which can be achieved by simple concatenation, weighted summation, etc., to form a comprehensive feature vector.

[0043] The fused feature vector is input into a trained generative model based on a probability diffusion process. For example, the generative model can adopt a diffusion architecture. The generative model uses the probability diffusion principle to iteratively update the feature vector and gradually generate a feature sequence that conforms to the target distribution. After multiple iterations, the generative model outputs a feature sequence containing facial expressions and audio information. The feature sequence not only retains the rhythm and emotional information of the audio, but also incorporates the changes in facial expressions.

[0044] Step 130, input the feature sequence and the human face reference image into the PDFGC decoder, and obtain a video containing human face expressions corresponding to the reference audio output by the decoder.

[0045] Specifically, after obtaining the feature sequence, necessary processing, such as smoothing, normalization, etc., may be performed first to ensure that it is suitable as an input of the decoder.

[0046] The processed feature sequence and the face reference image are input into the PDFGC decoder. The decoder uses the principle of the PDFGC algorithm to gradually generate video frames containing facial expressions based on the input feature sequence and face reference image.

[0047] After multiple iterations, the PDFGC decoder outputs a video containing facial expressions. The video is consistent with the reference audio in terms of speech content and rhythm, and the facial expressions change as the audio content changes.

[0048] The method provided by the embodiment of the present invention extracts the facial features of a more highly realistic and finely controllable facial reference image through a PDFGC encoder; inputs the facial features and audio features into a trained generative model based on a probability diffusion process to obtain a feature sequence output by the generative model; the generative model based on the probability diffusion process can generate richer expressions with more expressiveness; finally, the feature sequence and the facial reference image are input into a PDFGC decoder to obtain a video containing facial expressions corresponding to the reference audio output by the decoder.

[0049] In some embodiments, Figure 2 is one of the flow charts of step 120 in the virtual human expression synthesis method provided by the present invention, such as Figure 2 As shown, the facial features and audio features are input into the trained generative model based on the probability diffusion process to obtain the feature sequence output by the generative model, including: Step 121, obtaining a control signal, where the control signal includes an expression control signal and / or a sight control signal; Step 122, inputting the facial features, audio features, and control signals into the trained generative model based on the probability diffusion process to obtain a feature sequence output by the generative model, wherein the control signals include expression control signals and / or sight control signals.

[0050] Specifically, considering that the generated feature sequence not only retains the rhythm and emotional information of the audio, but also incorporates the changes in facial expressions, in order to further improve the expressiveness of facial expressions and the overall control accuracy, and improve the quality of the generated results, the generation model in this embodiment can also generate feature sequences under a given control signal. The control signal here is a signal that controls the generation of virtual human expressions. The control signal can be any form of data, such as text prompts, labels, values, etc., which are used to guide the generation model to generate specific virtual human expressions.

[0051] Considering that the existing technology cannot achieve control over eye movements. The DreamTalk method generates virtual humans by directly learning the speaking style and lip shape through audio, and inputs it into the denoising module as a condition to obtain motion frames. In the process, no constraints and learning are made on fine expressions such as head posture and eye movements. Therefore, it is difficult for the existing technology to achieve fine control over the rendered eye movements. Based on the expression synthesis, this embodiment can also add fine training and control of eye movements, that is, add a sight control signal to achieve a variety of realistic expression control results.

[0052] In this embodiment, the control signal can be an expression control signal, or a sight control signal, or can include both an expression control signal and a sight control signal. The expression control signal can be used to control the expression change of the virtual character, and the sight control signal can be used to control the sight direction or sight change of the virtual character.

[0053] For the generation of feature sequences, facial features, audio features, and control signals can be input into a trained generative model based on a probability diffusion process to obtain a feature sequence output by the generative model.

[0054] In some embodiments, Figure 3 FIG. 2 is a second flow chart of step 120 in the method for synthesizing virtual human expressions provided by the present invention. Figure 3 As shown, step 122 specifically includes: Step 122-1, inputting the facial features and the audio features into the feature extraction module of the generation model to obtain the intermediate features output by the feature extraction module; Step 122-2, input the intermediate features and control signals into the cross-attention module of the generation model to obtain the feature sequence output by the cross-attention module.

[0055] Specifically, in the case where there is a control signal, the related art usually directly injects the expression control conditions through splicing, and the generated expression synthesis result is relatively bland, not vivid enough, and not realistic enough.

[0056] In this embodiment, the generative model adopts a transformer architecture, and firstly, the facial features and audio features are fused and input into the feature extraction module of the generative model. The feature extraction module here can be a transformer unit, and the feature extraction module processes the input facial features and audio features, and outputs an intermediate feature vector containing information of both, i.e., an intermediate feature.

[0057] Figure 4 FIG. 3 is a flow chart of step 120 in the method for synthesizing virtual human expressions provided by the present invention. Figure 4 As shown in the figure, taking the control signal including the expression control signal and the sight control signal as an example, the intermediate features output by the feature extraction module and the control signal are taken as input and passed to the cross attention module. This module is a deep learning model that can process multimodal input and generate conditional feature sequences.

[0058] In the criss-cross attention module, the attention mechanism is used to calculate the correlation between the intermediate features and the control signal. The criss-cross attention module can focus on the intermediate feature parts that are most relevant to the control signal and generate a conditional feature sequence. This feature sequence reflects the comprehensive information of the input data and is guided by the control signal.

[0059] The method provided by the embodiment of the present invention can achieve more precise expression control through the cross-attention module.

[0060] In some embodiments, the intermediate features and the control signal are input into the cross-attention module of the generative model to obtain a conditional feature sequence output by the cross-attention module, specifically including: determining a bias characteristic of a control signal; The intermediate features and bias features are input into the cross-attention module to obtain the conditional feature sequence output by the cross-attention module.

[0061] Specifically, when the embodiment of the present invention controls the expression of the virtual human through the control signal, it does not directly learn the expression feature indicated by the control signal itself, but learns the mapping relationship between the control signal and the expression of the virtual human. Here, the bias feature of the control signal refers to the tendency or preference of the model to generate an expression corresponding to the control signal when the control signal is given. By learning the association or mapping relationship between the control signal and the expression of the virtual human, the generative model can gradually master the ability to generate appropriate expressions under a given control signal.

[0062] In this embodiment, a corresponding conditional feature sequence is generated according to the bias feature and the intermediate feature of the control signal, which can more effectively control the generation of facial expressions and improve the accuracy and naturalness of the generated facial expressions.

[0063] Based on any of the above embodiments, step 121 specifically includes: Step 121-1, based on the PDFGC encoder, extracting each expression feature corresponding to each expression reference image, and performing linear interpolation smoothing processing on each expression feature to obtain an expression control signal; and / or,

[0064] Step 121 - 2 , extracting facial markers from the gaze reference image, and determining a gaze control signal based on the facial markers.

[0065] Specifically, the expression control signal can be obtained from each expression reference image. Here, each expression reference image refers to an image containing the expression range required for generating a video, such as an image of expressions such as happiness, sadness, anger, surprise, etc.

[0066] Each expression reference image is input into the PDFGC encoder, which uses deep learning technology to automatically extract expression-related features from the image. These features may include facial organ features, texture areas, and predefined feature points. The encoder gradually abstracts higher-level expression features through multi-layer convolution, pooling, full connection, and other operations. Finally, the expression feature vector is output, which contains the key information of the expression in the expression reference image.

[0067] Considering that the related art directly uses specific expression conditions for control, it is difficult to achieve smooth and seamless switching of expression rendering. For example, when the expression condition representing happiness is directly switched to the expression condition representing sadness, the expression will suddenly change between the rendered video frames, and it is impossible to obtain a smooth and realistic expression switching result.

[0068] Different from most expression control signals that use discrete values, such as one hot as a control signal, the expression control signal extracted in this embodiment is a continuous signal.

[0069] For the acquisition of expression control signals, smooth transitions between different expressions can be achieved by linear interpolation of each expression feature. In the expression feature space, regression algorithms (such as linear regression, ridge regression, etc.) are used to fit the expression feature vector and output a continuous expression control signal. The expression control signal can be used to control the expression changes of the virtual character.

[0070] The sight control signal may be obtained from a sight reference image. The sight reference image may be an image containing a human face and having clear eye sights, such as straight ahead, sideways, upward, downward, etc.

[0071] Facial landmarks can be extracted from the gaze reference image using methods based on template matching, supervised learning, detectors, or feature descriptors. Figure 5 Schematic diagram of facial landmarks provided by the present invention. Facial landmarks usually include feature points of key parts such as eyes, nose, and mouth.

[0072] For gaze control, special attention is paid to the marker points in the eye area, such as the corners of the eyes, the center of the pupil, etc. The gaze direction is calculated through the facial marker points, and the calculated gaze direction is converted into a gaze control signal, which can be used to control the gaze direction or gaze change of the virtual character.

[0073] The method provided by the embodiment of the present invention achieves various realistic expression control results through expression control signals and / or sight line control signals.

[0074] In some embodiments, determining the gaze control signal based on the facial landmarks specifically includes: The center of gravity of the eye area markers among the facial markers is determined as the sight center point; The gaze control signal is determined based on the displacement between the eyeball marker point and the gaze center point in the face marker point.

[0075] Specifically, the eye area markers usually include key points such as the corners of the eyes and the edge of the pupil. Based on the extracted eye area markers, calculate their center of gravity. The center of gravity can be obtained by calculating the weighted average position of all markers, where the weights can be set according to factors such as the importance or distance of the markers. In order to simplify the calculation, it can be assumed that the eye area markers are evenly distributed and their geometric center is directly calculated as the center of gravity. The calculated center of gravity of the eye area markers is used as the center of sight. This center point can be regarded as a reference point for the direction of eye gaze.

[0076] According to the positions of the extracted eyeball markers (such as the pupil center) and the sight center point, the displacement vector between them is calculated. This displacement vector represents the offset of the eyeball relative to the sight center point.

[0077] The calculated displacement vector is mapped to the sight line control signal, such as converting the direction and magnitude of the displacement vector into specific parameters of the control signal, such as angle, speed, acceleration, etc., so as to obtain the sight line control signal.

[0078] In some embodiments, Figure 6 FIG. 4 is a flow chart of step 120 in the method for synthesizing virtual human expressions provided by the present invention. Figure 6 As shown, step 122 specifically includes: Step 122-3, inputting the facial features, audio features, and control signals into the generation model to obtain a conditional feature sequence output by the generation model; Step 122-4, inputting the facial features and the audio features into the generative model to obtain an unconditional feature sequence output by the generative model; Step 122-5, the conditional feature sequence and the unconditional feature sequence are merged to obtain a feature sequence.

[0079] Specifically, the generative model not only makes predictions based on given conditions (i.e., control signals), but also considers unconditional generation. In this way, the model can make the generated results more in line with the given conditions while maintaining the naturalness of the results.

[0080] The generative model generates a conditional prediction based on facial features, audio features, and control signals, that is, it outputs a conditional feature sequence given a control condition. The generative model also generates an unconditional prediction based on facial features and audio features, that is, it outputs an unconditional feature sequence without any prompts or control conditions.

[0081] It should be noted that the conditional feature sequence and the unconditional feature sequence can be generated simultaneously or sequentially, and the execution order of the two is not limited when they are generated sequentially. That is, step 122-3 and step 122-4 can be executed simultaneously or sequentially, and the execution order of the two is not limited.

[0082] Then, the conditional feature sequence and the unconditional feature sequence are fused to obtain the final feature sequence. The fusion method can be simple splicing, weighted summation, attention mechanism, etc. The purpose of fusion is to combine the advantages of conditional and unconditional feature sequences to generate a comprehensive feature sequence that contains control signal guidance information and maintains naturalness. After fusion, a new feature sequence is obtained, which will be used as the input of the decoder to generate a video containing facial expressions.

[0083] In some embodiments, the conditional feature sequence and the unconditional feature sequence are merged to obtain a feature sequence, that is, step 122-5 specifically includes: Based on the control strength of the control signal, the conditional feature sequence and the unconditional feature sequence are fused to obtain the feature sequence.

[0084] Specifically, the fusion of conditional feature sequences and unconditional feature sequences can be achieved through the control strength of the control signal. The control strength of the control signal is used to characterize the correlation between the generated feature sequence and the control signal. The greater the correlation, the greater the control strength; conversely, the smaller the correlation, the smaller the control strength.

[0085] In some embodiments, the characteristic sequence is calculated using the following formula:

[0086] In the formula, represents the final generated feature sequence; Indicates the corresponding control signal The intensity of control; Indicates that the corresponding control signal is set to 0, that is, there is no control signal; represents the unconditional feature sequence, Represents a conditional feature sequence.

[0087] The method provided by the embodiment of the present invention can obtain a more desired generation result by fusing the conditional feature sequence and the unconditional feature sequence to obtain a feature sequence by controlling the control strength of the control signal, thereby controlling the diversity and correlation of the generated feature sequence and further improving the control accuracy of the virtual human expression synthesis.

[0088] Based on any of the above embodiments, Figure 7 It is a flow chart of the generative model training method provided by the present invention, such as Figure 7 As shown, the training steps of the generative model include: Step 710, fusing the sample face features and the sample audio features to obtain the original features, and performing noise processing on the original features to obtain the noise features of the current time step; Step 720, input the noise feature and the sample control signal into the denoising module of the initial model to obtain the prediction feature of the current time step output by the denoising module; Step 730, based on the difference between the original features and the predicted features at each time step, the parameters of the initial model are iterated to obtain a generated model.

[0089] Specifically, the sample facial features and the sample audio features can be fused, and the fusion method can be simple concatenation, weighted summation, or more complex fusion through a deep learning model to obtain the original features. The extraction of sample facial features and sample audio features can refer to the extraction method of facial features and audio features, which will not be repeated here.

[0090] Generate noise according to a preset noise distribution (such as Gaussian noise). The intensity and type of noise can be adjusted according to actual needs. Add the generated noise to the original feature to get the noise feature of the current time step.

[0091] The noise features and sample control signals of the current time step are used as inputs and sent to the denoising module of the initial model. The denoising module here does not directly learn the expression features in the control signal, but learns the association or mapping relationship between the control signal and the virtual human's expression, and tries to recover the original features or information closer to the original features from the noise features. It should be noted that the sample control signal uses Classifier-free guidance (CFG), that is, the sample control signal is randomly set to 0.

[0092] The denoising module outputs the predicted features of the current time step. The predicted features are closer to the original features than the input noise features, while retaining the expression information in the control signal.

[0093] Based on the difference between the original features and the predicted features at each time step (such as mean square error MSE, cross entropy loss, etc.), the loss function value is calculated. The loss function value reflects the prediction performance of the model at the current time step. Through the back propagation algorithm, the loss value is passed from the output layer to the input layer layer by layer, and the gradient of each parameter is calculated. According to the calculated gradient, the optimization algorithm is used to update the parameters of the model. Iterate continuously until the loss value converges to a preset threshold or reaches a preset number of iterations.

[0094] After multiple iterations, the model parameters are gradually optimized to the optimal state, and the model obtained at this time is the generative model. The generative model can generate high-quality prediction features based on the input noise characteristics and control signals.

[0095] In some embodiments, the loss function of the generative model can be expressed as:

[0096] In the formula, represents the time step, is the original input feature, is the time step right The result of adding noise is is the denoising module, is the sample control signal.

[0097] Based on any of the above embodiments, Figure 8 FIG. 2 is a flow chart of the method for synthesizing virtual human expressions provided by the present invention. Figure 8 As shown, the method includes: S1, extracting audio features of the reference audio, and extracting facial features of the face reference image based on the PDFGC encoder.

[0098] S2, based on the PDFGC encoder, extracting the expression features corresponding to the expression reference images respectively, and performing linear interpolation smoothing processing on the expression features to obtain an expression control signal.

[0099] S3, extracting the face markers in the sight reference image, determining the center of gravity of the eye area markers in the face markers as the sight center; and determining the sight control signal based on the displacement between the eyeball markers in the face markers and the sight center.

[0100] S4, inputting facial features, audio features, expression control signals, and sight control signals into the trained generative model based on the probability diffusion process to obtain a feature sequence output by the generative model. Specifically, it includes: The facial features and audio features are input into the feature extraction module of the generative model to obtain the intermediate features output by the feature extraction module; the intermediate features and the control signal are input into the cross-attention module of the generative model to obtain the feature sequence output by the cross-attention module.

[0101] S5, inputting the feature sequence and the reference face image into the PDFGC decoder, and obtaining a video output by the decoder that corresponds to the reference audio and contains facial expressions.

[0102] The method provided by the embodiment of the present invention injects facial features and audio features into the control signal in the form of attention, so as to achieve fine expression control. When training the expression condition control, no specific expression condition is used, but the average value of the control signal is used as the condition input. The expression conditions learned in this way are in a smooth space. When switching expressions, different expression conditions are linearly interpolated to achieve smooth, seamless, and lifelike switching results of expression synthesis. On the basis of expression synthesis, fine training and control of eye movements are also added to achieve diverse and lifelike expression control results. When controlling expressions, the expressions are not learned directly, but the bias of the expression conditions is learned, so that the generated feature sequence can be more expressive, more vivid and distinctive in expression expression.

[0103] Based on any of the above embodiments, Fig. 9 : is a schematic diagram of the structure of the virtual human expression synthesis device provided by the present invention, such as Fig. 9 As shown, the device comprises: A feature extraction unit 910, configured to extract audio features of a reference audio, and extract facial features of a reference facial image based on a PDFGC encoder; A sequence generation unit 920, configured to input the facial features and the audio features into a trained generation model based on a probability diffusion process to obtain a feature sequence output by the generation model; The sequence decoding unit 930 is used to input the feature sequence and the face reference image into a PDFGC decoder to obtain a video containing facial expressions corresponding to the reference audio output by the decoder.

[0104] Based on any of the above embodiments, the sequence generation unit is specifically used for: Acquiring a control signal, wherein the control signal includes an expression control signal and / or a sight control signal; The facial features, the audio features, and the control signal are input into the generation model to obtain a feature sequence output by the generation model.

[0105] Based on any of the above embodiments, the sequence generation unit is specifically used for: Inputting the facial features and the audio features into the feature extraction module of the generation model to obtain the intermediate features output by the feature extraction module; The intermediate features and the control signal are input into the cross-attention module of the generation model to obtain the feature sequence output by the cross-attention module.

[0106] Based on any of the above embodiments, the sequence generation unit is specifically used for: determining a bias characteristic of the control signal; The intermediate features and the bias features are input into the cross-attention module to obtain the conditional feature sequence output by the cross-attention module.

[0107] Based on any of the above embodiments, the sequence generation unit is specifically used for: Based on the PDFGC encoder, extracting each expression feature corresponding to each expression reference image, and performing linear interpolation smoothing processing on each expression feature to obtain an expression control signal; And / or, extracting facial marker points in the gaze reference image, and determining the gaze control signal based on the facial marker points.

[0108] Based on any of the above embodiments, the sequence generation unit is specifically used for: Determine the center of gravity of the eye region markers among the facial markers as the sight line center point; The sight line control signal is determined based on the displacement between the eyeball mark point in the face mark point and the sight line center point.

[0109] Based on any of the above embodiments, the sequence generation unit is specifically used for: Inputting the facial features, the audio features, and the control signal into the generation model to obtain a conditional feature sequence output by the generation model; Inputting the facial features and the audio features into the generative model to obtain an unconditional feature sequence output by the generative model; The conditional feature sequence and the unconditional feature sequence are fused to obtain the feature sequence.

[0110] Based on any of the above embodiments, it also includes a model training unit, which is specifically used to: The sample face features and the sample audio features are merged to obtain the original features, and the original features are subjected to noise processing to obtain the noise features of the current time step; Inputting the noise feature and the sample control signal into a denoising module of the initial model to obtain a prediction feature of the current time step output by the denoising module; Based on the difference between the original features and the predicted features at each time step, the initial model is iterated on parameters to obtain the generated model.

[0111] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the virtual human expression synthesis method, which includes: extracting audio features of reference audio, and extracting facial features of a facial reference image based on a PDFGC encoder; inputting the facial features and the audio features into a trained generation model based on a probability diffusion process to obtain a feature sequence output by the generation model; inputting the feature sequence and the facial reference image into a PDFGC decoder to obtain a video containing facial expressions corresponding to the reference audio output by the decoder.

[0112] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0113] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the virtual human expression synthesis method provided by the above-mentioned methods, which includes: extracting audio features of reference audio, and extracting facial features of a facial reference image based on a PDFGC encoder; inputting the facial features and the audio features into a trained generative model based on a probability diffusion process to obtain a feature sequence output by the generative model; inputting the feature sequence and the facial reference image into a PDFGC decoder to obtain a video containing facial expressions corresponding to the reference audio output by the decoder.

[0114] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the virtual human expression synthesis method provided by the above-mentioned methods, the method comprising: extracting audio features of reference audio, and extracting facial features of a facial reference image based on a PDFGC encoder; inputting the facial features and the audio features into a trained generative model based on a probability diffusion process to obtain a feature sequence output by the generative model; inputting the feature sequence and the facial reference image into a PDFGC decoder to obtain a video containing facial expressions corresponding to the reference audio output by the decoder.

[0115] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for synthesizing virtual human expressions, characterized in that: include: Extracting audio features of reference audio, and extracting facial features of reference face images based on PDFGC encoder; Inputting the facial features and the audio features into a trained generative model based on a probability diffusion process to obtain a feature sequence output by the generative model; The feature sequence and the human face reference image are input into a PDFGC decoder, and a video containing human facial expressions corresponding to the reference audio is output by the decoder.

2. The method for synthesizing virtual human expressions according to claim 1, characterized in that: The step of inputting the facial features and the audio features into a trained generative model based on a probability diffusion process to obtain a feature sequence output by the generative model comprises: Acquiring a control signal, wherein the control signal includes an expression control signal and / or a sight control signal; The facial features, the audio features, and the control signal are input into the generation model to obtain a feature sequence output by the generation model.

3. The method for synthesizing virtual human expressions according to claim 2, characterized in that: The step of inputting the facial features, the audio features, and the control signal into the generation model to obtain a feature sequence output by the generation model includes: Inputting the facial features and the audio features into the feature extraction module of the generation model to obtain the intermediate features output by the feature extraction module; The intermediate features and the control signal are input into the cross-attention module of the generation model to obtain the feature sequence output by the cross-attention module.

4. The method for synthesizing virtual human expressions according to claim 3, characterized in that: The step of inputting the intermediate features and the control signal into the cross attention module of the generation model to obtain the feature sequence output by the cross attention module comprises: determining a bias characteristic of the control signal; The intermediate features and the bias features are input into the cross-attention module to obtain the conditional feature sequence output by the cross-attention module.

5. The method for synthesizing virtual human expressions according to claim 2, characterized in that: The obtaining of the control signal comprises: Based on the PDFGC encoder, extracting each expression feature corresponding to each expression reference image, and performing linear interpolation smoothing processing on each expression feature to obtain an expression control signal; And / or, extracting facial marker points in the gaze reference image, and determining the gaze control signal based on the facial marker points.

6. The method for synthesizing virtual human expressions according to claim 5, characterized in that: The determining of the sight line control signal based on the facial markers includes: Determine the center of gravity of the eye region markers among the facial markers as the sight line center point; The sight line control signal is determined based on the displacement between the eyeball mark point in the face mark point and the sight line center point.

7. The method for synthesizing virtual human expressions according to claim 2, characterized in that: The step of inputting the facial features, the audio features, and the control signal into the generation model to obtain a feature sequence output by the generation model includes: Inputting the facial features, the audio features, and the control signal into the generation model to obtain a conditional feature sequence output by the generation model; Inputting the facial features and the audio features into the generative model to obtain an unconditional feature sequence output by the generative model; The conditional feature sequence and the unconditional feature sequence are fused to obtain the feature sequence.

8. The method for synthesizing virtual human expressions according to any one of claims 1 to 7, characterized in that: The training steps of the generative model include: The sample face features and the sample audio features are merged to obtain the original features, and the original features are subjected to noise processing to obtain the noise features of the current time step; Inputting the noise feature and the sample control signal into a denoising module of the initial model to obtain a prediction feature of the current time step output by the denoising module; Based on the difference between the original features and the predicted features at each time step, the initial model is iterated on parameters to obtain the generated model.

9. A virtual human expression synthesis device, characterized in that: include: A feature extraction unit, used for extracting audio features of a reference audio, and extracting facial features of a reference facial image based on a PDFGC encoder; A sequence generation unit, used for inputting the facial features and the audio features into a trained generation model based on a probability diffusion process to obtain a feature sequence output by the generation model; The sequence decoding unit is used to input the feature sequence and the human face reference image into a PDFGC decoder to obtain a video containing human face expressions corresponding to the reference audio output by the decoder.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the virtual human expression synthesis method as described in any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the virtual human expression synthesis method as claimed in any one of claims 1 to 8 is implemented.

Images