Digital human generation method, device, equipment and medium
By combining audio generation and 3D face reconstruction parameters, a natural digital human pose is generated, which solves the problem of unnatural poses generated based on sound reasoning and improves the user experience.
Patent Information
- Application Number
- CN202111165975.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-30
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-09-30
AI Technical Summary
The digital human pose generated based on sound reasoning is unnatural, resulting in a decline in user experience.
By inputting the target audio into the trained generator to obtain the first face key point that does not contain face pose information, extract 3D face reconstruction parameters from the target face image, and combine these key points and parameters to determine the face pose. The second face key point of the information, generate digital human pictures, and avoid the pose angle problems caused by audio inference.
It improves the naturalness of the digital human posture and improves the user experience.
Smart Images

Figure CN113886640B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and medium for generating a digital human. Background Art
[0002] Digital human is a method that uses information science to simulate the human body at different levels of form and function. With the rapid development of computer technology, the technology for generating digital human is becoming more and more mature. In actual processing, if digital human generation technology is to be applied in business and realize real-time interaction, the digital human generation solution must meet at least two points: good generation effect and high reasoning efficiency; among them, good digital human generation effect is a necessary prerequisite, and high reasoning efficiency is a business need.
[0003] At present, in order to improve the reasoning efficiency of digital humans, a solution for generating digital humans based on sound reasoning has emerged. It mainly uses a sound reasoning model to generate facial key points, and then combines the facial key points with the information extracted from the real original picture through a 3D face reconstruction model to finally generate a digital human.
[0004] However, the facial key points inferred based on sound contain facial posture information, and the facial posture information has an angle problem. Therefore, the posture of the digital human finally generated by the above solution is unnatural. Summary of the invention
[0005] In view of this, the present application provides a method, apparatus, device and medium for generating a digital human, so that the posture of the digital human generated based on sound reasoning can be more natural, thereby improving the user experience.
[0006] In a first aspect, an embodiment of the present application provides a method for generating a digital human, comprising:
[0007] Input the target audio into the trained first generator to obtain a plurality of first facial key points that do not contain facial posture information;
[0008] Extract 3D face reconstruction parameters from the target face image;
[0009] Determining a plurality of second facial key points including facial posture information according to the plurality of first facial key points and the 3D face reconstruction parameters;
[0010] A digital human image is generated based on the second facial key points.
[0011] Optionally, the method further includes:
[0012] Extracting face edge lines from the target face image;
[0013] The step of generating a digital human image according to the second facial key points includes:
[0014] Determine a target face contour line according to the second face key points and the face edge line;
[0015] The target face contour line is input into the trained second generator to obtain a digital human image.
[0016] Optionally, determining the target face contour line according to the second face key points and the face edge line includes:
[0017] Performing a convolution operation on a plurality of the second facial key points to obtain a facial key point feature vector, and performing a convolution operation on the facial edge lines to obtain an edge line feature vector;
[0018] Performing vector summation on the facial key point feature vector and the edge line feature vector to obtain a hidden vector;
[0019] The hidden vector is decoded by using a preset decoder to obtain a target face contour line.
[0020] Optionally, the target audio includes one audio frame or several consecutive audio frames;
[0021] When the target audio includes a plurality of continuous audio frames, the target audio is input into the trained first generator to obtain a plurality of first facial key points that do not include facial posture information, including:
[0022] Input each audio frame included in the target audio into the trained first generator to obtain a plurality of third facial key points that do not contain facial posture information and correspond to each audio frame;
[0023] Determine an average value of the third facial key points corresponding to a plurality of the audio frames;
[0024] The average value is determined as the first facial key point.
[0025] Optionally, the method further includes:
[0026] According to the time sequence of the target audios, the digital human images corresponding to the target audios are combined to generate a digital human video.
[0027] Optionally, the first generator is trained in the following manner:
[0028] Extracting a plurality of picture frames and audio frames corresponding to the picture frames from a video stream;
[0029] For each of the picture frames, extracting a plurality of fourth facial key points that do not contain facial posture information from the picture frame, and extracting audio features from the audio frame corresponding to the picture frame;
[0030] Determine a plurality of the fourth facial key points and the audio features as sample data;
[0031] Model training is performed based on a number of the sample data to obtain the first generator.
[0032] Optionally, the audio frame corresponding to the picture frame includes one audio frame or several consecutive audio frames.
[0033] Optionally, extracting a plurality of fourth facial key points that do not contain facial posture information from the picture frame includes:
[0034] Performing facial key point detection on the picture frame using a preset facial key point detection algorithm to obtain a fifth facial key point containing facial posture information;
[0035] The fifth facial key point is processed using a facial 3D deformation statistical model to obtain the fourth facial key point.
[0036] Optionally, extracting audio features from the audio frame corresponding to the picture frame includes:
[0037] For the audio frame corresponding to the image frequency, using Fourier transform to extract Mel frequency cepstrum coefficients as audio features corresponding to the audio frame;
[0038] or,
[0039] For the audio frame corresponding to the picture frame, the audio features corresponding to the audio frame are extracted using a preset speech recognition model.
[0040] Optionally, performing model training based on a plurality of sample data to obtain the first generator includes:
[0041] Inputting the audio features in each of the sample data into an initial first generator to obtain predicted facial key points corresponding to the audio features;
[0042] Determine a model loss value according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data;
[0043] If the model loss value does not meet the preset model convergence conditions, the model parameters of the first generator are updated based on the model loss value, and the first generator with updated model parameters is iteratively trained until the model loss value meets the model convergence conditions.
[0044] Optionally, determining the model loss value according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data includes:
[0045] Determine first loss value information according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data;
[0046] Determine second loss value information according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data;
[0047] Based on the first loss value information and the second loss value information, a model loss value is determined.
[0048] In a second aspect, an embodiment of the present application provides a digital human generation device, including:
[0049] A first key point determination module, used for inputting the target audio into the trained first generator to obtain a plurality of first facial key points that do not contain facial posture information;
[0050] A parameter extraction module is used to extract 3D face reconstruction parameters from the target face image;
[0051] A second key point determination module, configured to determine a plurality of second face key points containing face posture information based on the plurality of first face key points and the 3D face reconstruction parameters;
[0052] The digital human generation module is used to generate a digital human image based on the second facial key points.
[0053] Optionally, the device further comprises:
[0054] An edge line extraction module, used to extract face edge lines from the target face image;
[0055] The digital human generation module includes:
[0056] A contour line determination submodule, used to determine a target face contour line based on the second face key points and the face edge line;
[0057] The model input submodule is used to input the target face contour line into the trained second generator to obtain a digital human image.
[0058] Optionally, the contour determination submodule is specifically used to:
[0059] A convolution operation is performed on a plurality of the second facial key points to obtain a facial key point feature vector, and a convolution operation is performed on the facial edge line to obtain an edge line feature vector; vector summation is performed on the facial key point feature vector and the edge line feature vector to obtain a hidden vector; and the hidden vector is decoded using a preset decoder to obtain a target facial contour line.
[0060] Optionally, the target audio includes one audio frame or several consecutive audio frames;
[0061] The model input submodule is specifically used to: when the target audio includes several continuous audio frames, input each audio frame included in the target audio into the trained first generator respectively, to obtain several third facial key points that do not contain facial posture information corresponding to each of the audio frames; determine the average value of the third facial key points corresponding to the several audio frames; and determine the average value as the first facial key point.
[0062] Optionally, the digital human generation module is further used to combine the digital human images corresponding to the target audios in a time sequence of the target audios to generate a digital human video.
[0063] Optionally, the device further comprises:
[0064] A model training module, used for training to obtain the first generator;
[0065] The model training module specifically includes:
[0066] A frame extraction submodule, used to extract a number of picture frames and audio frames corresponding to the picture frames from a video stream;
[0067] A key point extraction submodule, for extracting a plurality of fourth face key points that do not contain face posture information from each of the picture frames;
[0068] An audio feature extraction submodule, used to extract audio features from the audio frame corresponding to the picture frame;
[0069] A sample construction submodule, used for determining a plurality of the fourth facial key points and the audio features as sample data;
[0070] The training submodule is used to perform model training based on a number of the sample data to obtain the first generator.
[0071] Optionally, the audio frame corresponding to the picture frame includes one audio frame or several consecutive audio frames.
[0072] Optionally, the key point extraction submodule is specifically used for:
[0073] The image frame is subjected to facial key point detection using a preset facial key point detection algorithm to obtain fifth facial key points containing facial posture information; the fifth facial key points are processed using a facial 3D deformation statistical model to obtain the fourth facial key points.
[0074] Optionally, the audio feature extraction submodule is specifically used to:
[0075] For the audio frame corresponding to the image frequency, using Fourier transform to extract Mel frequency cepstrum coefficients as audio features corresponding to the audio frame;
[0076] or,
[0077] For the audio frame corresponding to the picture frame, the audio features corresponding to the audio frame are extracted using a preset speech recognition model.
[0078] Optionally, the training submodule is specifically used for:
[0079] Inputting the audio features in each of the sample data into an initial first generator to obtain predicted facial key points corresponding to the audio features;
[0080] Determine a model loss value according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data;
[0081] If the model loss value does not meet the preset model convergence conditions, the model parameters of the first generator are updated based on the model loss value, and the first generator with updated model parameters is iteratively trained until the model loss value meets the model convergence conditions.
[0082] Optionally, the training submodule determines the model loss value according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data, including:
[0083] Determine first loss value information according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data;
[0084] Determine second loss value information according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data;
[0085] Based on the first loss value information and the second loss value information, a model loss value is determined.
[0086] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the digital human generation method as described in any one of the first aspects when executing the program stored in the memory.
[0087] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the digital human generation method as described in any one of the first aspects are implemented.
[0088] The technical solution provided in the embodiment of the present application obtains several first face key points that do not contain face posture information by inputting the target audio into the trained first generator, extracts 3D face reconstruction parameters from the target face image, determines several second face key points that contain face posture information based on the several first face key points and the 3D face reconstruction parameters, and generates a digital human image based on the second face key points. Since the face posture information contained in the second face key points ultimately used to generate the digital human image comes from the 3D face reconstruction parameters extracted from the real face image, compared with the prior art, the problem of unnatural posture of the digital human generated due to angle problems of the face posture information inferred based on audio can be avoided. That is, the technical solution proposed in the embodiment of the present application can make the posture of the digital human generated based on audio reasoning more natural and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0090] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0091] Figure 1 is a flowchart of the steps of a method for generating a digital human provided in an embodiment of the present application;
[0092] Figure 2 A flowchart of a method for generating a digital human provided in an optional embodiment of the present application;
[0093] Figure 3 is an example of the target face contour;
[0094] Figure 4 A flowchart of a method for generating a digital human provided in another optional embodiment of the present application;
[0095] Figure 5 This is a structural block diagram of a digital human generation device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0096] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0097] See also Figure 1 , showing a flow chart of the steps of a digital human generation method provided by an embodiment of the present application. Specifically, the digital human generation method provided by the present application can be applied to video generation scenarios, such as the case of generating a virtual image of a video based on a real picture; wherein the virtual image can be a digital human image in a video, and can be specifically used to represent a digital human in a digital human video. Specifically, as an example, the video generation scenario is a live video generation scenario. In this scenario, the digital human generation method provided by the present application is applied to the user audio and the picture of the live broadcast room host (such as a virtual host), so that the live broadcast video of the virtual host can be generated based on the user audio. As another example, the video generation scenario is an online education scenario. In this scenario, the digital human generation method provided by the present application is applied to the lecturer's audio and the lecturer's (such as a virtual lecturer) picture, so that the lecturer's picture can be driven based on the lecturer's audio, thereby generating a video of a virtual lecturer teaching online. Of course, the above application scenarios are only exemplary application scenarios of the digital human generation method provided by the present application, and the embodiments of the present application do not limit the specific application scenarios.
[0098] like Figure 1 As shown, the digital human generation method in the embodiment of the present application may specifically include the following steps:
[0099] Step 110: input the target audio into the trained first generator to obtain a plurality of first facial key points that do not contain facial posture information.
[0100] The target audio may refer to the real audio to be processed, such as the audio recorded by the user, or the audio in the video recorded by the user. Furthermore, in the embodiment of the present application, the target audio may be the audio contained in the final generated digital human video, that is, the audio output by the final generated digital human.
[0101] In one embodiment, the target audio includes an audio frame, such as individual audio frames in the audio or video recorded by the user. When the target audio includes an audio frame, it means that a digital human image is inferred and generated based on the audio frame.
[0102] In another embodiment, the target audio includes several continuous audio frames, such as the audio frames from the tth frame to the t+nth frame in the audio or video recorded by the user, where t is a positive integer and n is an integer not less than 1. When the target audio includes several continuous audio frames, it means that a digital human image is inferred and generated based on the several continuous audio frames.
[0103] The embodiment of the present application does not impose any specific limitation on the number of audio frames contained in the target audio.
[0104] The first generator is a machine learning model that is pre-trained and used to infer a number of facial key points (hereinafter referred to as first facial key points for ease of description) based on audio without facial posture information (i.e., pose information). For example, the first generator can infer 68 first facial key points based on audio. As for how the first generator is trained, we will explain it in detail below. Figure 4 The process shown is explained and will not be described in detail here.
[0105] It should be noted that, in the embodiment of the present application, the first facial key point inferred by the first generator does not contain pose information, which is different from the prior art, in which pose information and expression parameters are inferred based on audio, and then the pose information and expression parameters inferred based on audio are combined with the information extracted from the real original picture through a 3D face reconstruction model to finally generate a digital human. In addition, compared with the prior art, it can avoid the problem of unnatural posture of the final generated digital human due to the angle problem of the pose information inferred based on audio.
[0106] Based on the above description, in step 110, the target audio is input into the trained first generator to obtain a plurality of first facial key points that do not contain facial posture information.
[0107] Among them, when the target audio includes several continuous audio frames, the target audio is input into a trained first generator to obtain several first facial key points that do not contain facial posture information, including: inputting each audio frame included in the target audio into the trained first generator respectively to obtain several third facial key points that do not contain facial posture information corresponding to each audio frame, determining the average value of the third facial key points corresponding to the several audio frames, and determining the average value as the first facial key point.
[0108] For example, the target audio includes the tth, t+1th, and t+2th audio frames in an audio segment. According to the above description, the tth, t+1th, and t+2th audio frames are respectively input into the trained first generator to obtain three sets of first facial key points, which are respectively recorded as (x1, x2, x3, ..., x 68 ), (y1, y2, y3, ..., y 68 ), (z1, z2, z3, ..., z 68 ). According to the above-described processing method, a first face key point set finally obtained based on the three sets of first face key point sets is ((x1+y1+z1) / 3, (x2+y2+z2) / 3, ..., (x 68 +y 68 +z 68 ) / 3).
[0109] It can be seen that the technical solution provided in the embodiment of the present application can infer a group of facial key points based on multiple audio frames, which can improve the stability and accuracy of the inferred facial key points compared to inferring a group of facial key points based on one audio frame. If there is sudden noise in addition to the voice of the person speaking in the above-mentioned t+2th audio frame, then inferring the facial key points based only on the t+2th audio frame will cause the inferred facial key points to be greatly affected by the noise. However, if the facial key points are inferred together with the tth frame and the t+1th audio frame in the above manner, the influence of noise on the inferred facial key points will be weakened, making the inferred facial key points more stable and accurate.
[0110] Step 120: extracting 3D face reconstruction parameters from the target face image.
[0111] The target face image may refer to a real picture to be processed, such as an image or video of the target person captured by an image acquisition device.
[0112] 3D face reconstruction parameters include but are not limited to: face shape information, reflection information (including face angle information and posture information), texture information, lighting information, etc.
[0113] In a specific implementation, a preset 3D face reconstruction parameter extraction algorithm or a 3D face reconstruction parameter extraction model can be used to extract 3D face reconstruction parameters from the target face image. As for the specific 3D face reconstruction parameter extraction algorithm or 3D face reconstruction parameter extraction model, this embodiment of the application will not be elaborated in detail.
[0114] Step 130: Determine a plurality of second facial key points including facial posture information based on the plurality of first facial key points and 3D face reconstruction parameters.
[0115] From the above description, it can be seen that the 3D face reconstruction parameters contain facial posture information. Therefore, by combining several first facial key points with the 3D face reconstruction parameters, several facial key points containing facial posture information can be obtained (for convenience of description, referred to as second facial key points below).
[0116] In a specific implementation, a 3D facial deformation statistical model (such as 3DMM) may be used to process a number of first facial key points and 3D face reconstruction parameters to obtain the above-mentioned second facial key points.
[0117] Step 140: Generate a digital human image based on the second facial key points.
[0118] As for the specific implementation of step 140, please refer to the following Figure 2 The detailed description of the illustrated embodiment will not be given here in detail.
[0119] It can be seen that the technical solution provided in the embodiment of the present application, by inputting the target audio into the trained first generator, obtains several first face key points that do not contain face posture information, extracts 3D face reconstruction parameters from the target face image, and determines several second face key points that contain face posture information based on the several first face key points and the 3D face reconstruction parameters, and generates a digital human image based on the second face key points. Since the face posture information contained in the second face key points ultimately used to generate the digital human image comes from the 3D face reconstruction parameters extracted from the real face image, compared with the prior art, it can avoid the problem of unnatural posture of the digital human generated due to angle problems of the face posture information inferred based on audio. That is, the application of the technical solution proposed in the embodiment of the present application can make the posture of the digital human generated based on audio reasoning more natural and improve the user experience.
[0120] See also Figure 2 , shows a flowchart of a method for generating a digital human provided by an optional embodiment of the present application. Figure 2 As shown, the digital human generation method in the embodiment of the present application may specifically include the following steps:
[0121] Step 210: input the target audio into the trained first generator to obtain a plurality of first facial key points that do not contain facial posture information.
[0122] Step 220: extracting 3D face reconstruction parameters from the target face image.
[0123] For detailed description of step 210 and step 220, please refer to the above Figure 1 The relevant description in the illustrated embodiment will not be repeated here.
[0124] Step 230: extracting facial edge lines from the target facial image.
[0125] Edge refers to a collection of pixels whose grayscale of surrounding pixels has a step change or a roof change, and is one of the most important features of an image.
[0126] In the embodiment of the present application, a preset edge detection algorithm may be used to extract facial edge lines from the target facial image.
[0127] It should be noted that the execution order of step 220 and step 230 is not limited in the embodiment of the present application. For example, in a specific implementation, step 220 may be executed first, and then step 230; or step 230 may be executed first, and then step 220; or step 220 and step 230 may be executed simultaneously.
[0128] Step 240, determining the target face contour line based on the second face key points and the face edge line.
[0129] Step 250: input the target face contour line into the trained second generator to obtain a digital human image.
[0130] The following is a unified description of step 240 and step 250:
[0131] First, it can be seen from the description of step 240 and step 250 that in the embodiment of the present application, generating a digital human image based on the second facial key points includes: determining a target facial contour line based on the second facial key points and the facial edge line, and inputting the target facial contour line into the trained second generator to obtain a digital human image. Figure 3 The figure shows an example of the target face contour line.
[0132] Specifically, determining the target face contour line according to the second face key points and the face edge line in step 240 includes the following steps:
[0133] Step a, performing a convolution operation on a plurality of second facial key points to obtain a facial key point feature vector;
[0134] Step b, performing a convolution operation on the edge line of the face to obtain an edge line feature vector;
[0135] The purpose of the convolution operation is to extract features. For example, in step a, a convolution operation is performed on a plurality of second facial key points to extract the features of the plurality of second facial key points, and a corresponding feature vector is obtained, i.e., the facial key point feature vector. For example, in step b, a convolution operation is performed on the facial edge line to extract the features of the facial edge line, and a corresponding feature vector is obtained, i.e., the edge line feature vector.
[0136] In a specific implementation, a plurality of second facial key points and facial edge lines may be respectively input into a corresponding CNN (Convolutional Neural Network) model to perform a convolution operation thereon to obtain a corresponding feature vector.
[0137] It should be noted that the CNN model used to extract features of several second facial key points and the CNN model used to extract features of facial edge lines may be different, and the embodiments of the present application do not limit this.
[0138] It should also be noted that the execution order of step a and step b is not limited in the embodiment of the present application. For example, in a specific implementation, step a may be executed first, and then step b; or step b may be executed first, and then step a; or step a and step b may be executed simultaneously.
[0139] Step c, performing vector summation on the facial key point feature vectors and the edge line feature vectors to obtain a hidden vector;
[0140] In one embodiment, the hidden vector may be obtained by directly performing vector summation on the facial key point feature vector and the edge line feature vector.
[0141] In another embodiment, based on a preset weight parameter, a vector weighted summation may be performed on the facial key point feature vector and the edge line feature vector to obtain a hidden vector. As for the specific weight parameter, the embodiment of the present application does not limit it.
[0142] Step d: Use a preset decoder to decode the hidden vector to obtain the target face contour.
[0143] The second generator is a pre-trained machine learning model used to infer digital human images based on facial contours, such as a GAN (Generative Adversarial Networks) model.
[0144] Based on this, in an embodiment of the present application, the target face contour line is input into the trained second generator to obtain a digital human image.
[0145] Step 260: Combine the digital human images corresponding to the target audios in the time sequence of the target audios to generate a digital human video.
[0146] In the embodiment of the present application, the digital human generation method provided in the embodiment of the present application can be applied to a segment of audio (such as a segment of audio in wav format with a frame rate of 100) to ultimately generate a digital human video.
[0147] Specifically, in one embodiment, each audio frame in the audio segment can be determined as a target audio, and then the above steps 240 to 250 are performed for each target audio to obtain a digital human image corresponding to each target audio. Finally, according to the time sequence of the target audios, the digital human images corresponding to the target audios are combined to generate a digital human video.
[0148] In another embodiment, starting from the first audio frame of the audio segment, several consecutive audio frames may be determined as target audios in sequence. For example, if the audio segment includes 7 audio frames, the first to third frames may be determined as target audios, the second to fourth frames may be determined as target audios, the third to fifth frames may be determined as target audios, the fourth to sixth frames may be determined as target audios, and the fifth to seventh frames may be determined as target audios. Then, the above steps 240 to 250 are performed for each target audio to obtain a digital human image corresponding to each target audio. Finally, the digital human images corresponding to the target audios are combined according to the time sequence of the target audios to generate a digital human video.
[0149] It can be seen that the technical solution provided in the embodiment of the present application, since the facial posture information contained in the second facial key points ultimately used to generate the digital human image comes from the 3D face reconstruction parameters extracted from the real face image, therefore, compared with the prior art, it can avoid the problem of unnatural posture of the digital human finally generated due to the angle problem of the facial posture information inferred based on audio. That is, the application of the technical solution proposed in the embodiment of the present application can make the posture of the digital human generated based on audio reasoning more natural, thereby improving the user experience.
[0150] See also Figure 4 , shows a flowchart of a method for generating a digital human provided by another optional embodiment of the present application. Specifically, the method for generating a digital human provided by the embodiment of the present application may include the following steps in the model training phase:
[0151] Step 410: extract a number of picture frames and audio frames corresponding to the picture frames from the video stream.
[0152] The video stream may refer to a real video stream to be processed, such as a video stream recorded by a user. In a video stream, each video frame includes an audio frame and a picture frame. For example, if a one-second video stream includes 5 video frames, the video stream includes 5 audio frames and 5 picture frames, that is, the audio frames correspond to the picture frames one by one. Furthermore, in a specific implementation, the picture frames corresponding to several consecutive audio frames may be the same. Therefore, in this case, it can also be regarded that the several consecutive audio frames correspond to one picture frame.
[0153] In summary, in a video stream, each audio frame has a corresponding picture frame.
[0154] Step 420: for each picture frame, extract a plurality of fourth facial key points that do not contain facial posture information from the picture frame, and extract audio features from the audio frame corresponding to the picture frame.
[0155] In one embodiment, the specific implementation of step 420 extracting a number of fourth facial key points that do not contain facial posture information from each picture frame includes: using a preset facial key point detection algorithm to perform facial key point detection on the picture frame to obtain facial key points that contain facial posture information (for convenience of description, referred to as fifth facial key points), and then using a facial 3D deformation statistical model to process the fifth facial key points to obtain fourth facial key points that do not contain facial posture information.
[0156] Step 430: Determine a plurality of fourth facial key points and audio features as sample data.
[0157] As can be seen from the description in steps 420 and 430, for each picture frame in the video stream, a sample data can be constructed, the sample data includes the fourth facial key point corresponding to the picture frame, and the audio feature of the audio frame corresponding to the picture frame, and the audio feature is the input value of the sample data, and the fourth facial key point is the label value. Here, the audio frame corresponding to the picture frame can be one audio frame or several consecutive audio frames.
[0158] In a specific implementation, the method of extracting audio features from the audio frame corresponding to the picture frame includes but is not limited to: using Fourier transform to extract Mel-frequency cepstral coefficients as the audio features corresponding to the audio frame, using a preset speech recognition model to extract audio features corresponding to the audio frame, etc.
[0159] Step 440, perform model training based on a number of sample data to obtain a first generator.
[0160] In a specific implementation, a supervised learning algorithm may be used, and a number of sample data may be used to train the initial model. When a preset training stop condition is met, the training is stopped to obtain the first generator.
[0161] Optionally, the supervised learning algorithm may be a support vector machine algorithm, a naive Bayes algorithm, a decision tree algorithm, or the like.
[0162] The above initial model can be a CNN model, an LSTM (Long Short-Term Memory) model, etc.
[0163] In one embodiment, the above-mentioned training stop condition may refer to the convergence of the model obtained by training, and the model convergence means that the model loss value meets the model convergence condition. Specifically, the audio features in the above-mentioned sample data are input into the initial first generator to obtain the predicted facial key points corresponding to the audio features, and the model loss value is determined based on the predicted facial key points corresponding to each audio feature and the fourth facial key points in each sample data. If the model loss value does not meet the preset model convergence condition, the model parameters of the current first generator are updated based on the current model loss value, and the first generator after the updated model parameters is iteratively trained until the model loss value meets the model convergence condition, that is, the trained model converges.
[0164] Furthermore, the above-mentioned determining the model loss value based on the predicted facial key points corresponding to each audio feature and the fourth facial key points in each sample data includes: determining first loss value information, such as L1 Loss value, based on the predicted facial key points corresponding to each audio feature and the fourth facial key points in each sample data, that is, using the L1 Loss loss function to achieve the first loss value information based on the predicted facial key points corresponding to each audio feature and the fourth facial key points in each sample data; and determining second loss value information, such as wing Loss value, based on the predicted facial key points corresponding to each audio feature and the fourth facial key points in each sample data, that is, using the wing Loss loss function to achieve the second loss value information based on the predicted facial key points corresponding to each audio feature and the fourth facial key points in each sample data; finally, determine the model loss value based on the first loss value information and the second loss value information.
[0165] The first loss value information and the second loss value information may be directly summed or weighted summed, and the summed result may be determined as the model loss value.
[0166] In another embodiment, the above training stop condition may refer to reaching a preset number of training iterations.
[0167] It can be seen that the technical solution provided by the embodiment of the present application, by extracting a number of picture frames and audio frames corresponding to the picture frames from the video stream, for each picture frame, extracting a number of fourth facial key points that do not contain facial posture information from the picture frame, and extracting audio features from the audio frame corresponding to the picture frame, determining the number of fourth facial key points and audio features as sample data, performing model training based on the number of sample data, and obtaining a first generator, can realize training to obtain a machine learning model for inferring a number of facial key points that do not contain facial posture information based on audio, that is, the first generator; and then using the first generator to participate in the technical solution of generating a digital human based on audio reasoning, can avoid the problem of unnatural posture of the final generated digital human due to the angle problem of the facial posture information inferred based on audio, that is, the posture of the digital human generated based on audio reasoning can be made more natural, thereby improving the user experience.
[0168] The present application also provides a digital human generation device. Figure 5 As shown, the digital human generation device provided in the embodiment of the present application may include the following modules:
[0169] A first key point determination module 510, configured to input the target audio into a trained first generator to obtain a plurality of first facial key points that do not contain facial posture information;
[0170] A parameter extraction module 520 is used to extract 3D face reconstruction parameters from the target face image;
[0171] A second key point determination module 530, configured to determine a plurality of second facial key points including facial posture information according to the plurality of first facial key points and the 3D face reconstruction parameters;
[0172] The digital human generation module 540 is used to generate a digital human image according to the second facial key points.
[0173] Optionally, the device further includes (not shown in the figure):
[0174] An edge line extraction module, used to extract face edge lines from the target face image;
[0175] The digital human generation module 540 in the embodiment of the present application includes (not shown in the figure):
[0176] A contour line determination submodule, used to determine a target face contour line based on the second face key points and the face edge line;
[0177] The model input submodule is used to input the target face contour line into the trained second generator to obtain a digital human image.
[0178] Optionally, the contour determination submodule is specifically used to:
[0179] A convolution operation is performed on a plurality of the second facial key points to obtain a facial key point feature vector, and a convolution operation is performed on the facial edge line to obtain an edge line feature vector; vector summation is performed on the facial key point feature vector and the edge line feature vector to obtain a hidden vector; and the hidden vector is decoded using a preset decoder to obtain a target facial contour line.
[0180] Optionally, the target audio includes one audio frame or several consecutive audio frames;
[0181] The model input submodule is specifically used to: when the target audio includes several continuous audio frames, input each audio frame included in the target audio into the trained first generator respectively, to obtain several third facial key points that do not contain facial posture information corresponding to each of the audio frames; determine the average value of the third facial key points corresponding to the several audio frames; and determine the average value as the first facial key point.
[0182] Optionally, the digital human generating module 540 is further configured to combine the digital human images corresponding to the target audios in a time sequence of the target audios to generate a digital human video.
[0183] Optionally, the device further includes (not shown in the figure):
[0184] A model training module, used for training to obtain the first generator;
[0185] The model training module specifically includes:
[0186] A frame extraction submodule, used to extract a number of picture frames and audio frames corresponding to the picture frames from a video stream;
[0187] A key point extraction submodule, for extracting a plurality of fourth face key points that do not contain face posture information from each of the picture frames;
[0188] An audio feature extraction submodule, used to extract audio features from the audio frame corresponding to the picture frame;
[0189] A sample construction submodule, used for determining a plurality of the fourth facial key points and the audio features as sample data;
[0190] The training submodule is used to perform model training based on a number of the sample data to obtain the first generator.
[0191] Optionally, the audio frame corresponding to the picture frame includes one audio frame or several consecutive audio frames.
[0192] Optionally, the key point extraction submodule is specifically used for:
[0193] The image frame is subjected to facial key point detection using a preset facial key point detection algorithm to obtain fifth facial key points containing facial posture information; the fifth facial key points are processed using a facial 3D deformation statistical model to obtain the fourth facial key points.
[0194] Optionally, the audio feature extraction submodule is specifically used to:
[0195] For the audio frame corresponding to the image frequency, using Fourier transform to extract Mel frequency cepstrum coefficients as audio features corresponding to the audio frame;
[0196] or,
[0197] For the audio frame corresponding to the picture frame, the audio features corresponding to the audio frame are extracted using a preset speech recognition model.
[0198] Optionally, the training submodule is specifically used for:
[0199] Inputting the audio features in each of the sample data into an initial first generator to obtain predicted facial key points corresponding to the audio features;
[0200] Determine a model loss value according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data;
[0201] If the model loss value does not meet the preset model convergence conditions, the model parameters of the first generator are updated based on the model loss value, and the first generator with updated model parameters is iteratively trained until the model loss value meets the model convergence conditions.
[0202] Optionally, the training submodule determines the model loss value according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data, including:
[0203] Determine first loss value information according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data;
[0204] Determine second loss value information according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data;
[0205] Based on the first loss value information and the second loss value information, a model loss value is determined.
[0206] It should be noted that the image processing device provided above can execute the image processing method provided in any embodiment of the present application, and has the corresponding functions and beneficial effects of the execution method.
[0207] In a specific implementation, the above-mentioned digital human generation device can be applied to electronic devices such as personal computers and servers, so that the electronic devices can generate digital humans based on target audio as image processing devices, and make the generated digital humans more natural in posture, thereby improving the user experience.
[0208] Furthermore, an embodiment of the present application also provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement the steps of the digital human generation method described in any one of the above method embodiments when executing the program stored in the memory.
[0209] The embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the digital human generation method described in any one of the above method embodiments are implemented.
[0210] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device, equipment, and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0211] In this article, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0212] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.
Claims
1. A method for generating a digital human, characterized in that: include: Input the target audio into the trained first generator to obtain a plurality of first facial key points that do not contain facial posture information; Extract 3D face reconstruction parameters from the target face image; Determining a plurality of second facial key points including facial posture information according to the plurality of first facial key points and the 3D face reconstruction parameters; Generating a digital human image according to the second facial key points; The target audio includes a plurality of continuous audio frames, and the target audio is input into the trained first generator to obtain a plurality of first facial key points that do not contain facial posture information, including: Input each audio frame included in the target audio into the trained first generator to obtain a plurality of third facial key points that do not contain facial posture information and correspond to each audio frame; Determine an average value of the third facial key points corresponding to a plurality of the audio frames; The average value is determined as the first facial key point.
2. The method according to claim 1, characterized in that The method further comprises: Extracting face edge lines from the target face image; Generating a digital human image according to the second facial key points includes: Determining a target face contour line according to the second face key points and the face edge line; The target face contour line is input into the trained second generator to obtain a digital human image.
3. The method according to claim 2, characterized in that The step of determining the target face contour line according to the second face key points and the face edge line comprises: Performing a convolution operation on a plurality of the second facial key points to obtain a facial key point feature vector, and performing a convolution operation on the facial edge line to obtain an edge line feature vector; Performing vector summation on the facial key point feature vector and the edge line feature vector to obtain a hidden vector; The hidden vector is decoded by using a preset decoder to obtain a target face contour line.
4. The method according to claim 1, characterized in that: The method further comprises: According to the time sequence of the target audios, the digital human images corresponding to the target audios are combined to generate a digital human video.
5. The method according to claim 1, characterized in that The first generator is trained in the following way: Extracting a plurality of picture frames and audio frames corresponding to the picture frames from a video stream; For each of the picture frames, extracting a plurality of fourth facial key points that do not contain facial posture information from the picture frame, and extracting audio features from the audio frame corresponding to the picture frame; Determine a plurality of the fourth facial key points and the audio features as sample data; Model training is performed based on a number of the sample data to obtain the first generator.
6. The method according to claim 5, characterized in that The step of extracting a plurality of fourth facial key points that do not contain facial posture information from the picture frame includes: Performing facial key point detection on the picture frame using a preset facial key point detection algorithm to obtain a fifth facial key point containing facial posture information; The fifth facial key point is processed using a facial 3D deformation statistical model to obtain the fourth facial key point.
7. The method according to claim 5, characterized in that The extracting audio features from the audio frame corresponding to the picture frame includes: For the audio frame corresponding to the image frequency, using Fourier transform to extract Mel frequency cepstrum coefficients as audio features corresponding to the audio frame; or, For the audio frame corresponding to the picture frame, the audio features corresponding to the audio frame are extracted using a preset speech recognition model.
8. The method according to claim 5, characterized in that The performing model training according to the plurality of sample data to obtain the first generator comprises: Inputting the audio features in each of the sample data into an initial first generator to obtain predicted facial key points corresponding to the audio features; Determine a model loss value according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data; If the model loss value does not meet the preset model convergence conditions, the model parameters of the first generator are updated based on the model loss value, and the first generator with updated model parameters is iteratively trained until the model loss value meets the model convergence conditions.
9. The method according to claim 8, characterized in that The determining of the model loss value according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data includes: Determine first loss value information according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data; Determine second loss value information according to the predicted facial key points corresponding to each of the audio features and the fourth facial key points in each of the sample data; Based on the first loss value information and the second loss value information, a model loss value is determined.
10. A digital human generation device, characterized in that: include: A first key point determination module, used for inputting the target audio into the trained first generator to obtain a plurality of first facial key points that do not contain facial posture information; A parameter extraction module is used to extract 3D face reconstruction parameters from the target face image; A second key point determination module, configured to determine a plurality of second face key points containing face posture information based on the plurality of first face key points and the 3D face reconstruction parameters; A digital human generation module, used to generate a digital human image according to the second facial key points; The target audio includes a plurality of continuous audio frames, and the target audio is input into the trained first generator to obtain a plurality of first facial key points that do not contain facial posture information, including: Input each audio frame included in the target audio into the trained first generator to obtain a plurality of third facial key points that do not contain facial posture information and correspond to each audio frame; Determine an average value of the third facial key points corresponding to a plurality of the audio frames; The average value is determined as the first facial key point.
11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the digital human generation method described in any one of claims 1 to 9 when executing the program stored in the memory.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the digital human generation method as described in any one of claims 1 to 9 are implemented.
