Audio-driven digital human generation method and device, electronic equipment and storage medium

By extracting audio features and combining manifold mapping algorithms, high-fidelity digital human videos are generated, which solves the problem of facial expressions and voice being out of sync in digital human video synthesis, and improves customer experience in financial scenarios.

CN119071591APending Publication Date: 2024-12-03PING AN TECH (SHENZHEN) CO LTD

Patent Information

Application Number
CN202411199643.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the digital human video synthesis, the facial expressions of the human face are not synchronized with the input driver voice, resulting in low fidelity of the video screen and poor rendering effect, which affects the customer's impression and trust in financial scenarios.

Method used

By obtaining the audio signal input by the user and the target face video, the pitch contour features and the Mel frequency cepspectral coefficient features are extracted, and the feature splicing is performed, and the facial action sequence is converted into a sound-lip synchronization. The manifold mapping algorithm is used to generate the face Move video frames and render them using a pre-trained target face renderer to generate high-fidelity digital human videos.

Benefits of technology

Digital human videos that achieve sound and lips synchronization have high fidelity and good rendering effects, which enhance customer perception and trust in financial scenarios and improve the quality of financial services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071591A_ABST
    Figure CN119071591A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an audio-driven digital human generation method and device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology. The method comprises the following steps: acquiring an audio signal input by a user and a target face video; performing feature extraction on the audio signal to obtain audio features; converting the audio features into a facial action sequence with synchronous sound and lip; processing the facial action sequence by adopting a manifold mapping algorithm to obtain a linear combination of a plurality of facial actions; generating a video frame of the facial motion according to the linear combination of the plurality of facial actions; and using the target face renderer to render the video frame of the face motion to generate a target digital human video. On the basis of the method and the device, the digital human video with synchronous sound and lip can be generated, the method and the device have high fidelity, and in a financial scene, the digital human video with high fidelity is generated, so that the impression of a customer can be improved, the reliability of the customer can be improved, the financial service quality can be improved, and the competitiveness of a financial enterprise can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of financial technology (Fintech), and in particular to an audio-driven digital human generation method and device, an electronic device, and a storage medium. Background Art

[0002] In the field of digital humans, speech-driven speaker video synthesis is a hot topic. It aims to synthesize a target face speaking video based on an input speech. However, the facial expressions of the target face speaking video currently synthesized are not synchronized with the input driving speech, the fidelity of the synthesized video is not high, and the rendering effect is not good. In financial scenarios, low-fidelity digital human videos will affect customer perception and reduce customer trust. Based on this, how to generate high-fidelity digital human videos has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an audio-driven digital human generation method and device, an electronic device and a storage medium, which can generate lip-synchronized digital human videos with high fidelity and good rendering effects. In financial scenarios, generating high-fidelity digital human videos can enhance customer perception, increase customer trust, help improve the quality of financial services, and thus enhance the competitiveness of financial enterprises.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes an audio-driven digital human generation method, the method comprising:

[0005] Obtain the audio signal input by the user and the target face video;

[0006] Extracting features of the audio signal to obtain pitch contour features and Mel-frequency cepstral coefficient features;

[0007] Performing feature concatenation on the pitch contour feature and the Mel-frequency cepstral coefficient feature to obtain the audio feature;

[0008] converting the audio features into a lip-synced facial action sequence;

[0009] Using a manifold mapping algorithm to process the facial action sequence to obtain a linear combination of multiple facial actions;

[0010] generating a video frame of facial motion based on a linear combination of the plurality of facial actions;

[0011] The video frames of the facial motion are rendered using a pre-trained target face renderer to generate a target digital human video, wherein the target face renderer is trained by the target face video.

[0012] In some embodiments, the method is applied to an audio-driven digital human generation model, which includes a pre-trained audio feature-to-facial action module, a facial action post-processing module, and a facial action rendering module, and the audio feature-to-facial action module is connected to the facial action rendering module via the facial action post-processing module.

[0013] In some embodiments, the audio feature to facial action module is used to convert the audio feature into the facial action sequence of lip synchronization, the audio feature to facial action module includes an encoder-decoder model, and the facial action post-processing module is used to process the facial action sequence to obtain a linear combination of the multiple facial actions.

[0014] In some embodiments, the training method of the audio feature to facial action module includes:

[0015] Acquire a data set based on lip reading recognition, wherein the data set includes a plurality of data pairs, wherein the data pairs are mutually related person videos and person audios;

[0016] Extracting audio features corresponding to the character audio from the character audio;

[0017] The human face three-dimensional reconstruction technology is used to transform the human face video to obtain the first human face 3D key point;

[0018] Input the audio features into the encoder-decoder model, and output the second face 3D key points;

[0019] Calculate the distance loss between the first face 3D key point and the second face 3D key point to obtain a first loss function;

[0020] Input the audio feature and the second facial 3D key point into a pre-trained lip synchronization discriminator, output the cross entropy loss of the probability value, and obtain a second loss function;

[0021] The audio feature to facial action module is trained based on the first loss function and the second loss function to obtain the trained audio feature to facial action module.

[0022] In some embodiments, the process of processing the facial action sequence using a manifold mapping algorithm to obtain a linear combination of multiple facial actions includes:

[0023] inputting the facial action sequence into a facial motion set;

[0024] Using a K nearest neighbor algorithm to calculate the first K most similar facial movements from the facial movement set;

[0025] The least square method is used to solve the linear weighted weights of the K most approximate facial actions to obtain a linear combination of the K most approximate facial actions.

[0026] In some embodiments, the facial action rendering module includes a pre-trained target face renderer, and the facial action rendering module is used to generate a video frame of facial motion according to a linear combination of the multiple facial actions, and use the pre-trained target face renderer to render the video frame of facial motion to generate the target digital human video, and the target face renderer is a conditional NeRF renderer.

[0027] In some embodiments, the training method of the target face renderer includes:

[0028] Extracting 3D facial key points and a facial image from the target facial video, wherein the facial image and the 3D facial key points correspond to each other;

[0029] Determine a direction vector and a 3D position according to the face image;

[0030] Determine an implicit function according to the 3D key points of the face, the direction vector and the 3D position;

[0031] Calculating the density of the color and radiation field based on the implicit function;

[0032] Rendering the color and the density of the radiation field into a rendered image using a differentiable volume rendering method;

[0033] Calculating the Euclidean distance between the rendered image and the face image to obtain a third loss function;

[0034] The target face renderer is trained based on the third loss function to obtain the trained target face renderer.

[0035] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present application proposes an audio-driven digital human generation device, the device comprising:

[0036] An acquisition module is used to acquire the audio signal input by the user and the target face video;

[0037] An extraction module, used for extracting features from the audio signal to obtain pitch contour features and Mel frequency cepstral coefficient features;

[0038] A splicing module, used for performing feature splicing on the pitch contour feature and the Mel-frequency cepstral coefficient feature to obtain the audio feature;

[0039] A conversion module, used for converting the audio features into a lip-synchronized facial action sequence;

[0040] A mapping module, used for processing the facial action sequence using a manifold mapping algorithm to obtain a linear combination of multiple facial actions;

[0041] A generating module, configured to generate a video frame of facial motion according to a linear combination of the plurality of facial motions;

[0042] The rendering module is used to render the video frame of the facial motion using a pre-trained target face renderer to generate a target digital human video, wherein the target face renderer is trained by the target face video.

[0043] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0044] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0045] The audio-driven digital human generation method and device, electronic device and storage medium proposed in the present application obtain the audio signal input by the user and the target face video; extract the features of the audio signal to obtain the pitch contour features and the Mel-frequency cepstral coefficient features; perform feature splicing on the pitch contour features and the Mel-frequency cepstral coefficient features to obtain the audio features; convert the audio features into a facial action sequence with lip synchronization; process the facial action sequence using a manifold mapping algorithm to obtain a linear combination of multiple facial actions; generate a video frame of facial motion according to the linear combination of multiple facial actions; use a pre-trained target face renderer to render the video frame of facial motion to generate the target digital human video, wherein the target face renderer is trained by the target face video. By extracting the pitch contour features and the Mel-frequency cepstral coefficient features from the audio signal, using the pitch contour as the audio-to-motion mapping, the audio features are converted into a long facial motion sequence with high temporal consistency and audio lip synchronization accuracy, so as to improve the synchronization between audio and face and improve the quality of facial motion. A method based on manifold space projection is used as post-processing of facial motion sequences to avoid outliers in the predicted facial motion sequences, so as to improve the robustness of facial movements. Finally, a pre-trained target face renderer is used to render the video frames of facial motion to generate a high-fidelity target digital human video. Based on this, the embodiment of the present application can generate a lip-synced digital human video with high fidelity and good rendering effect. In financial scenarios, generating a high-fidelity digital human video can enhance customer perception and trust, help improve the quality of financial services, and thus enhance the competitiveness of financial enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the audio-driven digital human generation method provided in an embodiment of the present application;

[0047] Figure 2 It is an overall schematic diagram of the audio-driven digital human generation model provided in the embodiment of the present application;

[0048] Figure 3 is a schematic diagram of extracting features from an audio signal and converting it into a facial action sequence provided by an embodiment of the present application;

[0049] Figure 4A This is a flow chart of the training method for the audio feature to facial action module;

[0050] Figure 4B This is a flowchart of introducing the lip sync discriminator for training in the audio feature to facial action module;

[0051] Figure 5A yes Figure 1 Flow chart of step S105 in FIG.

[0052] Figure 5B is a processing flow chart of the facial action post-processing module;

[0053] Figure 6 It is a flow chart of the training method of the target face renderer;

[0054] Figure 7 is a structural schematic diagram of an audio-driven digital human generation device provided in an embodiment of the present application;

[0055] Figure 8 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] First, some nouns involved in this application are analyzed:

[0060] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0061] Digital Human / Meta Human: A digital human image close to human image created by digital technology. Digital humans exist in the non-physical world, are created and used by computer means, and are a comprehensive product with multiple human characteristics (appearance characteristics, human performance and interaction capabilities, etc.). Virtual digital humans can be divided according to personality symbols and graphic dimensions, or according to character graphic dimensions. Character image, voice generation module, animation generation module, audio and video synthesis display module, and interaction module constitute the general system framework of virtual digital humans. Digital humans are virtual characters with digital appearance. Breaking physical boundaries to provide anthropomorphic services and experiences is their core value. Hyperrealism, toolization, and strong interaction are development trends.

[0062] Mel-Frequency Cepstral Coefficient (MFCC): It is the coefficient that makes up the Mel-Frequency Cepstral. It is derived from the cepstrum of the audio segment. The difference between the cepstrum and the Mel-Frequency Cepstral is that the frequency band division of the Mel-Frequency Cepstral is equally spaced on the Mel scale, which is more similar to the human auditory system than the linearly spaced frequency bands used in the normal logarithmic cepstrum. Such nonlinear representation can make sound signals better represented in many fields, such as in audio compression.

[0063] K-Nearest Neighbor (KNN) classification algorithm: It is a relatively mature method in theory and one of the simplest machine learning algorithms. The idea of ​​this method is: in the feature space, if most of the k nearest samples (i.e. the closest samples in the feature space) near a sample belong to a certain category, then the sample also belongs to this category.

[0064] Neural Radiance Fields (NeRF) is a computer vision technology used to generate high-quality 3D reconstruction models, which is a unique improvement over traditional and very mature 3D reconstruction technologies.

[0065] In the field of digital humans, speech-driven speaker video synthesis is a hot topic. It aims to synthesize a target face speaking video based on an input speech. However, the facial expressions of the target face speaking video currently synthesized are not synchronized with the input driving speech, the fidelity of the synthesized video is not high, and the rendering effect is not good. In financial scenarios, low-fidelity digital human videos will affect customer perception and reduce customer trust. Based on this, how to generate high-fidelity digital human videos has become a technical problem that needs to be solved urgently.

[0066] Based on this, the embodiment of the present application provides an audio-driven digital human generation method and device, an electronic device and a storage medium, which obtains an audio signal input by a user and a target face video; extracts features of the audio signal to obtain pitch contour features and Mel-frequency cepstral coefficient features; performs feature splicing on the pitch contour features and Mel-frequency cepstral coefficient features to obtain audio features; converts the audio features into a facial action sequence with lip sync; processes the facial action sequence using a manifold mapping algorithm to obtain a linear combination of multiple facial actions; generates a video frame of facial motion according to the linear combination of multiple facial actions; uses a pre-trained target face renderer to render the video frame of facial motion to generate a target digital human video, wherein the target face renderer is trained by the target face video. By extracting pitch contour features and Mel-frequency cepstral coefficient features from the audio signal, using the pitch contour as an audio-to-motion mapping, the audio features are converted into a long facial motion sequence with high temporal consistency and audio lip synchronization accuracy, so as to improve the synchronization between audio and face and improve the quality of facial motion. A method based on manifold space projection is used as post-processing of facial motion sequences to avoid outliers in the predicted facial motion sequences, so as to improve the robustness of facial movements. Finally, a pre-trained target face renderer is used to render the video frames of facial motion to generate a high-fidelity target digital human video. Based on this, the embodiment of the present application can generate a lip-synced digital human video with high fidelity and good rendering effect. In financial scenarios, generating a high-fidelity digital human video can enhance customer perception and trust, help improve the quality of financial services, and thus enhance the competitiveness of financial enterprises.

[0067] The audio-driven digital human generation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the audio-driven digital human generation method in the embodiments of the present application is described.

[0068] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0069] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0070] The audio-driven digital human generation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The audio-driven digital human generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the audio-driven digital human generation method, etc., but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0073] Figure 1 is an optional flow chart of the audio-driven digital human generation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0074] Step S101, obtaining an audio signal input by a user and a target face video;

[0075] Step S102, extracting features from the audio signal to obtain pitch contour features and Mel-frequency cepstral coefficient features;

[0076] Step S103, performing feature concatenation on the pitch contour feature and the Mel-frequency cepstral coefficient feature to obtain an audio feature;

[0077] Step S104, converting the audio features into a lip-synchronized facial action sequence;

[0078] Step S105, using a manifold mapping algorithm to process the facial action sequence to obtain a linear combination of multiple facial actions;

[0079] Step S106, generating a video frame of facial motion according to a linear combination of multiple facial movements;

[0080] Step S107, using a pre-trained target face renderer to render the video frame of the facial motion to generate a target digital human video, wherein the target face renderer is trained by the target face video.

[0081] In step S101 of some embodiments, an audio signal and a target face video input by a user are obtained. The audio signal may be a voice input by the user. The target face video is a real-life video of the target face that the user wants to generate. In a financial scenario, taking the introduction of financial products as an example, by obtaining a pre-recorded voice introducing the financial product and obtaining the target face video of the spokesperson of the financial product, a digital human video introducing the financial product with the facial image of the spokesperson can be synthesized later, thereby improving customer trust.

[0082] In step S102 of some embodiments, feature extraction is performed on the audio signal to obtain pitch contour features and Mel-frequency cepstral coefficient features. Specifically, feature extraction can be performed on the audio signal through a pitch contour extraction module, such as discretization of continuous pitch (fundamental frequency) values, so as to obtain pitch contour features. Among them, the pitch contour feature is an important feature in audio signal analysis, which represents the fundamental frequency or basic frequency of the sound, and is usually related to the pitch of the sound. The pitch contour is used as an audio-to-motion mapping. Pitch can be used as a useful facial movement indicator to improve the quality of facial movements. At the same time, feature extraction can also be performed on the audio signal through an MFCC coefficient extraction module to obtain Mel-frequency cepstral coefficient features. The Mel-frequency cepstral coefficient (MFCC) of an audio signal is a set of 10-20 features, which can describe the overall shape of the spectrum envelope.

[0083] In step S103 of some embodiments, the pitch contour feature and the Mel-frequency cepstral coefficient feature are feature concatenated to obtain the audio feature. The overall audio feature is obtained by feature concatenating the pitch contour feature and the Mel-frequency cepstral coefficient feature.

[0084] In step S104 of some embodiments, the audio features are converted into a lip-synchronized facial action sequence. Since the audio features include pitch contour features and Mel-frequency cepstral coefficient features, the pitch contour features can be used as a useful facial movement indicator. Therefore, the facial action sequence obtained by converting the audio features has high time consistency and audio lip synchronization accuracy, thereby improving the synchronization between audio and face and improving the quality of facial actions.

[0085] In step S105 of some embodiments, a manifold mapping algorithm is used to process the facial action sequence to obtain a linear combination of multiple facial actions. A method based on manifold space projection is used as a post-processing of the facial action sequence to avoid outliers in the predicted action sequence to improve the robustness of the facial action.

[0086] In step S106 of some embodiments, a video frame of facial motion is generated based on a linear combination of multiple facial movements. The linear combination of facial movements is a combination of different facial expressions, including but not limited to a combination of different facial expressions such as eyebrow movement changes, eye corner movement changes, and mouth corner movement changes. The video frame of facial motion is predicted based on the linear combination of multiple facial movements.

[0087] In step S107 of some embodiments, a pre-trained target face renderer is used to render the video frames of facial motion to generate a target digital human video, wherein the target face renderer is trained by the target face video. For each new face instance, a few minutes of video of the face is needed to train a renderer of a specific face, so as to train a controllable high-fidelity renderer for each instance. Based on this, a pre-trained target face renderer is used to render the video frames of facial motion to generate a high-fidelity target digital human video.

[0088] The audio-driven digital human generation method of the embodiment of the present application can generate lip-synchronized digital human videos with high fidelity and good rendering effects. In financial scenarios, generating high-fidelity digital human videos can enhance customer perception, increase customer trust, and help improve the quality of financial services, thereby enhancing the competitiveness of financial enterprises.

[0089] Steps S101 to S107 shown in the embodiment of the present application are to obtain the audio signal input by the user and the target face video; extract the features of the audio signal to obtain the pitch contour features and the Mel-frequency cepstral coefficient features; perform feature splicing on the pitch contour features and the Mel-frequency cepstral coefficient features to obtain the audio features; convert the audio features into a facial action sequence with lip synchronization; process the facial action sequence using a manifold mapping algorithm to obtain a linear combination of multiple facial actions; generate a video frame of facial motion according to the linear combination of multiple facial actions; use a pre-trained target face renderer to render the video frame of facial motion to generate a target digital human video, wherein the target face renderer is trained by the target face video. By extracting the pitch contour features and the Mel-frequency cepstral coefficient features from the audio signal, using the pitch contour as the audio-to-motion mapping, the audio features are converted into a long facial motion sequence with high temporal consistency and audio lip synchronization accuracy, so as to improve the synchronization between audio and face and improve the quality of facial actions. A method based on manifold space projection is used as post-processing of facial motion sequences to avoid outliers in the predicted facial motion sequences, so as to improve the robustness of facial movements. Finally, a pre-trained target face renderer is used to render the video frames of facial motion to generate a high-fidelity target digital human video. Based on this, the embodiment of the present application can generate a lip-synced digital human video with high fidelity and good rendering effect. In financial scenarios, generating a high-fidelity digital human video can enhance customer perception and trust, help improve the quality of financial services, and thus enhance the competitiveness of financial enterprises.

[0090] In some embodiments, Figure 2 As shown, the audio-driven digital human generation method of the embodiment of the present application can be applied to an audio-driven digital human generation model, which includes a pre-trained audio feature conversion module to facial action, a facial action post-processing module and a facial action rendering module, and the audio feature conversion module to facial action is connected to the facial action rendering module through the facial action post-processing module. Among them, the audio feature conversion module to facial action is used to convert audio features into a facial action sequence with lip synchronization; the facial action post-processing module is used to process the facial action sequence to obtain a linear combination of multiple facial actions; the facial action rendering module includes a target face renderer, and the facial action rendering module is used to generate a video frame of facial movement according to a linear combination of multiple facial movements, and use a pre-trained target face renderer to render the video frame of facial movement to generate a target digital human video. Among them, the target face renderer can use a conditional NeRF renderer.

[0091] In some embodiments, Figure 3As shown, before the driving audio is input into the audio feature to facial action module, the driving audio is first input into the pitch contour extraction module and the MFCC coefficient extraction module to extract features of the audio signal, and the pitch contour feature and the Mel-frequency cepstral coefficient feature are extracted respectively. Among them, the pitch contour feature is an important feature in audio signal analysis, which represents the fundamental frequency or basic frequency of the sound, and is usually related to the pitch of the sound. The pitch contour is used as an audio-to-motion mapping. Pitch can be used as a useful facial movement indicator to improve the quality of facial movements. The Mel-frequency cepstral coefficient (MFCC) of an audio signal is a set of 10-20 features, which can describe the overall shape of the spectrum envelope.

[0092] Please refer to FIG. 4 . In some embodiments, the training method of the audio feature to facial action module of the embodiment of the present application includes but is not limited to steps S401 to S407:

[0093] Step S401, obtaining a data set based on lip reading recognition, the data set includes a plurality of data pairs, and the data pairs are mutually related character videos and character audios;

[0094] Step S402, extracting audio features corresponding to the character audio from the character audio;

[0095] Step S403, using a 3D face reconstruction technology to convert the character video to obtain a first 3D face key point;

[0096] Step S404, input the audio features into the encoder-decoder model, and output the second face 3D key points;

[0097] Step S405, calculating the distance loss between the first face 3D key point and the second face 3D key point to obtain a first loss function;

[0098] Step S406, inputting the audio features and the 3D key points of the second face into a pre-trained lip synchronization discriminator, outputting the cross entropy loss of the probability value, and obtaining a second loss function;

[0099] Step S407: training the audio feature to facial action module based on the first loss function and the second loss function to obtain a trained audio feature to facial action module.

[0100] like Figure 4BAs shown, a data set based on lip reading recognition obtains multiple data pairs, which are interrelated character videos and character audios. Feature extraction is performed on the character audio to obtain pitch contour features and Mel-frequency cepstral coefficient features, and then the pitch contour features and Mel-frequency cepstral coefficient features are feature spliced ​​to obtain audio features corresponding to the character audio. Using the face three-dimensional reconstruction technology, the character video in the data set is converted to obtain the first face 3D key point, and the first face 3D key point is used as a training label. The audio feature conversion facial action module includes an encoder-decoder model, and the audio feature is input into the audio feature conversion facial action module to obtain the predicted second face 3D key point. The distance loss between the first face 3D key point and the second face 3D key point is calculated to obtain a first loss function. The audio feature and the second face 3D key point are input into a pre-trained lip sync discriminator, and the cross entropy loss of the probability value is output to obtain a second loss function, namely, the lip sync loss. It should be pointed out that the lip sync discriminator has been pre-trained and does not participate in the training process of the audio feature conversion facial action module. The audio feature-to-facial action module is trained based on the first loss function and the second loss function to obtain a trained audio feature-to-facial action module. Based on this, by introducing pitch information and lip synchronization loss in the audio feature-to-facial action module, the temporal consistency and naturalness of the predicted facial action can be improved.

[0101] See also Figure 5A In some embodiments, step S105 may include but is not limited to steps S501 to S503:

[0102] Step S501, inputting a facial action sequence into a facial motion set;

[0103] Step S502, using a K nearest neighbor algorithm to calculate the first K most similar facial movements from the facial movement set;

[0104] Step S503: using the least square method to solve the linear weighted weights of the K most similar facial actions, to obtain a linear combination of the K most similar facial actions.

[0105] It should be pointed out that simply improving the quality of predicted facial motion is not enough to achieve excellent video quality. Because it requires a facial motion to video module to accurately render character images corresponding to the assigned facial motion. However, such renderers are usually learned from a very small dataset (a few minutes long video), and such facial motion input space is relatively narrow. When faced with facial actions that are not in the training domain (a few minutes long video), the renderer may produce inaccurate facial motion or even collapsed rendering results. Therefore, with the help of manifold mapping, the predicted facial motions are dragged closer to the facial motion set of the training domain used to train the renderer. Figure 5BAs shown, the facial action sequence predicted based on the audio signal is input into the facial action set. The facial action set can use the GT facial action library. The K nearest neighbor algorithm (KNN) is used to calculate the top K most approximate facial actions from the facial action set. The least squares method is used to solve the linear weighted weights of the K most approximate facial actions, and the linear combination of the K most approximate facial actions is obtained as the output.

[0106] See also Figure 6 In some embodiments, the training method of the target face renderer of the embodiment of the present application includes but is not limited to steps S601 to S607:

[0107] Step S601, extracting 3D facial key points and a facial image from a target facial video, wherein the facial image and the 3D facial key points correspond to each other;

[0108] Step S602, determining a direction vector and a 3D position according to the face image;

[0109] Step S603, determining an implicit function according to the 3D key points, direction vectors and 3D positions of the face;

[0110] Step S604, calculating the density of the color and radiation field based on the implicit function;

[0111] Step S605, using a differentiable volume rendering method to render the color and the density of the radiation field into a rendered image;

[0112] Step S606, calculating the Euclidean distance between the rendered image and the face image to obtain a third loss function;

[0113] Step S607: training the target face renderer based on the third loss function to obtain a trained target face renderer.

[0114] The embodiment of the present application adopts conditional NeRF as a facial action rendering module to effectively render video frames based on predicted facial movements. For each new face instance, it is necessary to use a few minutes of video of the face to train a renderer for the specific face. For the training process of the renderer, the face 3D key points and face images are extracted from the target face video, wherein the face image and the face 3D key points correspond to each other, that is, the face 3D key points and the face image are used as training data pairs. In addition to the direction vector d and 3D position x required by traditional NeRF, the embodiment of the present application additionally introduces the face 3D key point l as a condition for the color information and geometric information of the rendering space. The implicit function F is determined based on the face 3D key point l, the direction vector d and the 3D position x. The implicit function F is defined as follows:

[0115] F θ :(x,d,l)→(c,σ)

[0116] Where c is the color and б is the density of the radiation field. The color and the density of the radiation field are calculated based on implicit functions. According to the differentiable rendering method, the color c and the density б of the radiation field are rendered into a rendered image. The Euclidean distance between the rendered image and the face image is calculated to obtain the third loss function, that is, the third loss function is the L2 distance between the rendered image and the real image. The target face renderer is trained based on the third loss function to obtain a trained target face renderer.

[0117] Based on this, the audio-driven digital human generation method of the application embodiment achieves more general and high-fidelity conversational face generation by introducing pitch contour features, adversarial lip synchronization training, face motion manifold mapping, and 3D face controllable high-fidelity renderer NeRF.

[0118] See also Figure 7 The embodiment of the present application further provides an audio-driven digital human generation device, which can implement the above-mentioned audio-driven digital human generation method, and the device includes:

[0119] An acquisition module 710 is used to acquire an audio signal input by a user and a target face video;

[0120] An extraction module 720 is used to extract features from the audio signal to obtain pitch contour features and Mel-frequency cepstral coefficient features;

[0121] A splicing module 730 is used to perform feature splicing on the pitch contour feature and the Mel-frequency cepstral coefficient feature to obtain an audio feature;

[0122] A conversion module 740, for converting audio features into a lip-synced facial action sequence;

[0123] A mapping module 750, for processing the facial action sequence using a manifold mapping algorithm to obtain a linear combination of multiple facial actions;

[0124] A generating module 760, for generating a video frame of facial motion according to a linear combination of multiple facial actions;

[0125] The rendering module 770 is used to render the video frames of facial motion using a pre-trained target face renderer to generate a target digital human video, wherein the target face renderer is trained by the target face video.

[0126] In some embodiments of the present application, the acquisition module 710 acquires an audio signal and a target face video input by a user; the extraction module 720 performs feature extraction on the audio signal to obtain a pitch contour feature and a Mel-frequency cepstral coefficient feature; the splicing module 730 performs feature splicing on the pitch contour feature and the Mel-frequency cepstral coefficient feature to obtain an audio feature; the conversion module 740 converts the audio feature into a lip-synchronized facial action sequence; the mapping module 750 uses a manifold mapping algorithm to process the facial action sequence to obtain a linear combination of multiple facial actions; the generation module 760 generates a video frame of facial motion based on the linear combination of multiple facial actions; the rendering module 770 uses a pre-trained target face renderer to render the video frame of facial motion to generate a target digital human video, wherein the target face renderer is trained by the target face video.

[0127] In some embodiments of the present application, an audio signal and a target face video input by a user are obtained. The audio signal may be a voice input by the user. The target face video is a real-life video of the target face that the user wants to generate. In a financial scenario, taking the introduction of a financial product as an example, by obtaining a pre-recorded voice introducing the financial product and obtaining the target face video of the spokesperson of the financial product, a digital human video introducing the financial product with the facial image of the spokesperson can be synthesized in the future, thereby improving customer trust.

[0128] In some embodiments of the present application, feature extraction is performed on an audio signal to obtain a pitch contour feature and a Mel-frequency cepstral coefficient feature. Specifically, feature extraction can be performed on an audio signal by a pitch contour extraction module, such as discretization of continuous pitch (fundamental frequency) values, thereby obtaining a pitch contour feature. Among them, the pitch contour feature is an important feature in audio signal analysis, which represents the fundamental frequency or basic frequency of the sound, and is usually related to the pitch of the sound. The pitch contour is used as an audio-to-motion mapping. Pitch can be used as a useful facial movement indicator to improve the quality of facial movements. At the same time, feature extraction can also be performed on an audio signal by an MFCC coefficient extraction module to obtain a Mel-frequency cepstral coefficient feature. The Mel-frequency cepstral coefficient (MFCC) of an audio signal is a set of 10-20 features, which can describe the overall shape of the spectrum envelope.

[0129] In some embodiments of the present application, the pitch contour feature and the Mel-frequency cepstral coefficient feature are feature concatenated to obtain the audio feature. The overall audio feature is obtained by feature concatenating the pitch contour feature and the Mel-frequency cepstral coefficient feature.

[0130] In some embodiments of the present application, audio features are converted into lip-synchronized facial action sequences. Since the audio features include pitch contour features and Mel-frequency cepstral coefficient features, the pitch contour features can be used as a useful facial movement indicator. Therefore, the facial action sequence obtained by converting the audio features has high time consistency and audio lip synchronization accuracy, thereby improving the synchronization between audio and face and improving the quality of facial actions.

[0131] In some embodiments of the present application, a manifold mapping algorithm is used to process the facial action sequence to obtain a linear combination of multiple facial actions. A method based on manifold space projection is used as a post-processing of the facial action sequence to avoid outliers in the predicted action sequence to improve the robustness of the facial action.

[0132] In some embodiments of the present application, a video frame of facial motion is generated based on a linear combination of multiple facial movements. The linear combination of facial movements is a combination of different facial expressions, including but not limited to a combination of different facial expressions such as eyebrow movement changes, eye corner movement changes, and mouth corner movement changes. The video frame of facial motion is predicted based on the linear combination of multiple facial movements.

[0133] In some embodiments of the present application, a pre-trained target face renderer is used to render a video frame of facial motion to generate a target digital human video, wherein the target face renderer is trained by the target face video. For each new face instance, a few minutes of video of the face is needed to train a renderer of a specific face, so as to train a controllable high-fidelity renderer for each instance. Based on this, a pre-trained target face renderer is used to render a video frame of facial motion to generate a high-fidelity target digital human video.

[0134] Based on this, in the audio-driven digital human generation device of the embodiment of the present application, the acquisition module 710 acquires the audio signal input by the user and the target face video; the extraction module 720 performs feature extraction on the audio signal to obtain pitch contour features and Mel-frequency cepstral coefficient features; the splicing module 730 performs feature splicing on the pitch contour features and Mel-frequency cepstral coefficient features to obtain audio features; the conversion module 740 converts the audio features into a lip-synchronized facial action sequence; the mapping module 750 uses a manifold mapping algorithm to process the facial action sequence to obtain a linear combination of multiple facial actions; the generation module 760 generates video frames of facial movements based on the linear combination of multiple facial movements; the rendering module 770 uses a pre-trained target face renderer to render the video frames of facial movements to generate a target digital human video, wherein the target face renderer is trained by the target face video. The present application obtains the audio signal and target face video input by the user; extracts the features of the audio signal to obtain the pitch contour features and the Mel-frequency cepstral coefficient features; performs feature splicing on the pitch contour features and the Mel-frequency cepstral coefficient features to obtain the audio features; converts the audio features into a facial action sequence of lip sync; processes the facial action sequence using a manifold mapping algorithm to obtain a linear combination of multiple facial actions; generates a video frame of facial motion according to the linear combination of multiple facial actions; uses a pre-trained target face renderer to render the video frame of facial motion to generate a target digital human video, wherein the target face renderer is trained by the target face video. By extracting the pitch contour features and the Mel-frequency cepstral coefficient features from the audio signal, using the pitch contour as the audio-to-motion mapping, the audio features are converted into a long facial motion sequence with high temporal consistency and audio lip synchronization accuracy, so as to improve the synchronization between audio and face and improve the quality of facial motion. And a method based on manifold space projection is used as a post-processing of the facial motion sequence to avoid outliers in the predicted facial motion sequence to improve the robustness of facial motion. Finally, the pre-trained target face renderer is used to render the video frames of facial motion to generate a high-fidelity target digital human video. Based on this, the embodiment of the present application can generate a lip-synced digital human video with high fidelity and good rendering effect. In financial scenarios, generating a high-fidelity digital human video can improve customer perception and increase customer trust, which helps to improve the quality of financial services, thereby enhancing the competitiveness of financial enterprises.

[0135] The specific implementation of the audio-driven digital human generation device is basically the same as the specific implementation of the above-mentioned audio-driven digital human generation method, and will not be repeated here.

[0136] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned audio-driven digital human generation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0137] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0138] The processor 801 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in the embodiment of the present application.

[0139] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 802 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the audio-driven digital human generation method of the embodiment of this application, that is, by obtaining the audio signal and the target face video input by the user; extracting the features of the audio signal to obtain the pitch contour features and the Mel-frequency cepstral coefficient features; performing feature splicing on the pitch contour features and the Mel-frequency cepstral coefficient features to obtain the audio features; converting the audio features into a facial action sequence of lip synchronization; processing the facial action sequence by a manifold mapping algorithm to obtain a linear combination of multiple facial actions; generating a video frame of facial motion according to the linear combination of multiple facial actions; using a pre-trained target face renderer to render the video frame of facial motion to generate a target digital human video, wherein the target face renderer is trained by the target face video. By extracting pitch contour features and Mel-frequency cepstral coefficient features from the audio signal, using the pitch contour as an audio-to-motion mapping, the audio features are converted into long facial motion sequences with high temporal consistency and audio lip synchronization accuracy to improve the synchronization between audio and face and the quality of facial movements. A method based on manifold space projection is used as post-processing of facial motion sequences to avoid outliers in predicted facial motion sequences to improve the robustness of facial movements. Finally, a pre-trained target face renderer is used to render the video frames of facial motion to generate a high-fidelity target digital human video. Based on this, the embodiment of the present application can generate a lip-synchronized digital human video with high fidelity and good rendering effect. In financial scenarios, generating a high-fidelity digital human video can enhance customer perception and customer trust, help improve the quality of financial services, and thus enhance the competitiveness of financial enterprises.

[0140] The input / output interface 803 is used to implement information input and output.

[0141] The communication interface 804 is used to realize the communication interaction between this device and other devices. Communication can be realized through wired methods (such as USB, network cable, etc.) or wireless methods (such as mobile network, WIFI, Bluetooth, etc.).

[0142] The bus transmits information between various components of the device (eg, the processor 801 , the memory 802 , the input / output interface 803 , and the communication interface 804 ).

[0143] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus.

[0144] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned audio-driven digital human generation method is implemented.

[0145] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The audio-driven digital human generation method, audio-driven digital human generation device, electronic device and storage medium provided in the embodiment of the present application obtain the audio signal input by the user and the target face video; extract the features of the audio signal to obtain the pitch contour features and the Mel-frequency cepstral coefficient features; perform feature splicing on the pitch contour features and the Mel-frequency cepstral coefficient features to obtain the audio features; convert the audio features into a facial action sequence with lip sync; process the facial action sequence using a manifold mapping algorithm to obtain a linear combination of multiple facial actions; generate a video frame of facial motion according to the linear combination of multiple facial actions; use a pre-trained target face renderer to render the video frame of facial motion to generate the target digital human video, wherein the target face renderer is trained by the target face video. By extracting the pitch contour features and the Mel-frequency cepstral coefficient features from the audio signal, using the pitch contour as the audio-to-motion mapping, the audio features are converted into a long facial motion sequence with high temporal consistency and audio lip synchronization accuracy, so as to improve the synchronization between audio and face and improve the quality of facial motion. A method based on manifold space projection is used as post-processing of facial motion sequences to avoid outliers in the predicted facial motion sequences, so as to improve the robustness of facial movements. Finally, a pre-trained target face renderer is used to render the video frames of facial motion to generate a high-fidelity target digital human video. Based on this, the embodiment of the present application can generate a lip-synced digital human video with high fidelity and good rendering effect. In financial scenarios, generating a high-fidelity digital human video can enhance customer perception and trust, help improve the quality of financial services, and thus enhance the competitiveness of financial enterprises.

[0147] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as a computer-readable program, a data structure, a program module or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media generally contain computer-readable programs, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0148] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0149] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0150] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0152] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0153] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0155] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0158] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An audio-driven digital human generation method, It is characterized in that The method comprises: Obtain the audio signal input by the user and the target face video; Extracting features of the audio signal to obtain pitch contour features and Mel-frequency cepstral coefficient features; Performing feature concatenation on the pitch contour feature and the Mel-frequency cepstral coefficient feature to obtain the audio feature; converting the audio features into a lip-synced facial action sequence; Using a manifold mapping algorithm to process the facial action sequence to obtain a linear combination of multiple facial actions; generating a video frame of facial motion based on a linear combination of the plurality of facial actions; The video frames of the facial motion are rendered using a pre-trained target face renderer to generate a target digital human video, wherein the target face renderer is trained by the target face video.

2. The method according to claim 1, It is characterized in that The method is applied to an audio-driven digital human generation model, which includes a pre-trained audio feature conversion facial action module, a facial action post-processing module and a facial action rendering module, wherein the audio feature conversion facial action module is connected to the facial action rendering module via the facial action post-processing module.

3. The method according to claim 2, It is characterized in that The audio feature to facial action module is used to convert the audio feature into the facial action sequence of lip synchronization, the audio feature to facial action module includes an encoder-decoder model, and the facial action post-processing module is used to process the facial action sequence to obtain a linear combination of the multiple facial actions.

4. The method according to claim 3, It is characterized in that The training method of the audio feature to facial action module includes: Acquire a data set based on lip reading recognition, wherein the data set includes a plurality of data pairs, wherein the data pairs are mutually related person videos and person audios; Extracting audio features corresponding to the character audio from the character audio; The human face three-dimensional reconstruction technology is used to transform the human face video to obtain the first human face 3D key point; Input the audio features into the encoder-decoder model, and output the second face 3D key points; Calculate the distance loss between the first face 3D key point and the second face 3D key point to obtain a first loss function; Input the audio feature and the second facial 3D key point into a pre-trained lip synchronization discriminator, output the cross entropy loss of the probability value, and obtain a second loss function; The audio feature to facial action module is trained based on the first loss function and the second loss function to obtain the trained audio feature to facial action module.

5. The method according to claim 1, It is characterized in that The facial action sequence is processed by a manifold mapping algorithm to obtain a linear combination of multiple facial actions, including: inputting the facial action sequence into a facial motion set; Using a K nearest neighbor algorithm to calculate the first K most similar facial movements from the facial movement set; The least square method is used to solve the linear weighted weights of the K most approximate facial actions to obtain a linear combination of the K most approximate facial actions.

6. The method according to claim 2, It is characterized in that The facial action rendering module includes the pre-trained target face renderer, and the facial action rendering module is used to generate a video frame of facial movement according to a linear combination of the multiple facial movements, and use the pre-trained target face renderer to render the video frame of facial movement to generate the target digital human video, and the target face renderer is a conditional NeRF renderer.

7. The method according to claim 1 or 6, It is characterized in that The training method of the target face renderer includes: Extracting 3D facial key points and a facial image from the target facial video, wherein the facial image and the 3D facial key points correspond to each other; Determine a direction vector and a 3D position according to the face image; Determine an implicit function according to the 3D key points of the face, the direction vector and the 3D position; Calculating the density of the color and radiation field based on the implicit function; Rendering the color and the density of the radiation field into a rendered image using a differentiable volume rendering method; Calculating the Euclidean distance between the rendered image and the face image to obtain a third loss function; The target face renderer is trained based on the third loss function to obtain the trained target face renderer.

8. An audio-driven digital human generation device, It is characterized in that The device comprises: An acquisition module is used to acquire the audio signal input by the user and the target face video; An extraction module, used for extracting features from the audio signal to obtain pitch contour features and Mel frequency cepstral coefficient features; A splicing module, used for performing feature splicing on the pitch contour feature and the Mel-frequency cepstral coefficient feature to obtain the audio feature; A conversion module, used for converting the audio features into a lip-synchronized facial action sequence; A mapping module, used for processing the facial action sequence using a manifold mapping algorithm to obtain a linear combination of multiple facial actions; A generating module, configured to generate a video frame of facial motion according to a linear combination of the plurality of facial motions; The rendering module is used to render the video frame of the facial motion using a pre-trained target face renderer to generate a target digital human video, wherein the target face renderer is trained by the target face video.

9. An electronic device, It is characterized in that The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the audio-driven digital human generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the audio-driven digital human generation method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Speaker voice conversion method based on adversarial learning and related equipment

    CN112863529A

  • Digital human head driving device based on voice rhythm

    CN117577096A

Cited By

  • Digital human batch generation method and device, equipment, medium and program product

    CN121509769A