A digital human generation method, device, electronic device and storage medium

Abstract

The present invention relates to a digital human generation method, device, electronic device and storage medium. The digital human generation method may include: obtaining a first video containing a first object; extracting a first audio frame and a first video frame corresponding to the first audio frame in the first video, extracting a first remaining face picture without a target local area in the first video frame; generating first key point coordinates of key points of a face in a target local area of ​​the first object based on the first audio frame and a preset average face model; extracting a first fully connected layer parameter corresponding to the first audio frame in the preset average face model, and determining a first model parameter of a preset digital human generation model based on the first fully connected layer parameter; generating a digital human picture based on the first key point coordinates, the first remaining face picture and a preset digital human generation model whose model parameters are the first model parameters. The embodiment of the present invention can realize a faster process of generating a digital human picture.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for generating a digital human. Background Art

[0002] Virtual digital humans are "people" that exist in the digital world. They are three-dimensional "people" that are displayed by highly restoring real humans in the digital world through technologies such as motion capture, three-dimensional modeling, and speech synthesis.

[0003] If digital human video generation technology needs to achieve real-time interaction, at least two points need to be met: good generation effect and fast inference speed; good effect is a necessary prerequisite, and fast speed is a commercial need.

[0004] At present, the solution for generating digital human images generally processes three-dimensional data, and the speed of generating digital human images is relatively slow. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a digital human generation method, device, electronic device and storage medium.

[0006] In a first aspect, the present application provides a method for generating a digital human, comprising:

[0007] Obtain a first video containing a first object;

[0008] Extracting a first audio frame and a first video frame corresponding to the first audio frame from the first video, and extracting a first remaining face picture after removing a target local area from the first video frame;

[0009] generating first key point coordinates of key points of the face in the target local area of ​​the first object based on the first audio frame and a preset average face model;

[0010] Extracting first fully connected layer parameters corresponding to the first audio frame from the preset average face model, and determining first model parameters of a preset digital human generation model based on the first fully connected layer parameters;

[0011] A digital human image is generated based on the first key point coordinates, the first remaining face image, and the preset digital human generation model whose model parameters are the first model parameters.

[0012] Optionally, determining first model parameters of a preset digital human generation model based on the first fully connected layer parameters includes:

[0013] Performing deconvolution processing on the first fully connected layer parameters to obtain first intermediate parameters;

[0014] Acquire a second model parameter of the preset digital human generation model, where the second model parameter is a model parameter determined after the preset digital human generation model is trained;

[0015] The first intermediate parameter is combined with the second model parameter to obtain the first model parameter.

[0016] Optionally, combining the first intermediate parameter with the second model parameter to obtain the first model parameter includes:

[0017] The intermediate parameters are respectively merged with the second model parameters corresponding to the encoding layer of the preset digital human generation model, the second model parameters corresponding to the hidden layer of the preset digital human generation model, and the second model parameters corresponding to the decoding layer of the preset digital human generation model to obtain the first model parameters.

[0018] Optionally, the training method of the preset average face model includes:

[0019] extracting a plurality of second audio frames and second video frames corresponding to the second audio frames from a second video containing the first object;

[0020] Inputting the second audio frame into the preset average face model to obtain predicted coordinate values ​​of second facial key points in a second video frame corresponding to the second audio frame;

[0021] Extracting the label coordinate value of the second facial key point in the second video frame;

[0022] Calculate a first loss value based on the predicted coordinate value and the label coordinate value;

[0023] If the first loss value is greater than or equal to a preset first threshold, adjusting the model parameters of the preset average face model, and performing the step of inputting the second audio frame into the preset average face model to obtain the predicted coordinate value of the second face key point in the second video frame corresponding to the second audio frame;

[0024] If the first loss value is less than a preset first threshold, it is determined that the training of the preset average face model is completed.

[0025] Optionally, the training method of the preset digital human generation model includes:

[0026] Acquire a second video containing the first object;

[0027] Extracting a second audio frame and a second video frame corresponding to the first audio frame from the second video, and extracting a second remaining face picture without the target local area from the second video frame;

[0028] generating second key point coordinates of facial key points in the target local area of ​​the first object based on the second audio frame and a preset average face model;

[0029] Extracting second fully connected layer parameters corresponding to the second audio frame from the preset average face model, and determining third model parameters of the preset digital human generation model based on the second fully connected layer parameters, wherein the third model parameters are model parameters of the preset digital human generation model during this training;

[0030] Using the second key point coordinates and the second remaining face image to train the preset digital human generation model whose model parameters are the third model parameters, so that the preset digital human generation model outputs a digital human prediction image;

[0031] Calculate a second loss value based on the digital human prediction picture and the second video frame;

[0032] If the second loss value is greater than or equal to a preset second threshold, adjusting the model parameters of the preset digital human generation model, and executing the step of generating second key point coordinates of the facial key points in the target local area of ​​the first object based on the second audio frame and the preset average face model;

[0033] If the second loss value is less than a preset second threshold, it is determined that the training of the preset digital human generation model is completed.

[0034] Optionally, determining a third model parameter of a preset digital human generation model based on the second fully connected layer parameter includes:

[0035] When the training of the preset average face model is completed, obtaining the second fully connected layer parameters of the preset average face model;

[0036] Performing deconvolution processing on the second fully connected layer parameters to obtain second intermediate parameters;

[0037] Acquire a fourth model parameter of the preset digital human generation model, wherein the fourth model parameter is a model parameter of the preset digital human generation model before the current training;

[0038] The second intermediate parameter is combined with the fourth model parameter to obtain the third model parameter.

[0039] In a second aspect, the present application provides a digital human generation device, comprising:

[0040] An acquisition module, used to acquire a first video containing a first object;

[0041] A removal module, configured to extract a first audio frame and a first video frame corresponding to the first audio frame from the first video, and extract a first remaining face image of a target local area from the first video frame;

[0042] A generating module, configured to generate first key point coordinates of key points of a face in the target local area of ​​the first object based on the first audio frame and a preset average face model;

[0043] An extraction module, configured to extract first fully connected layer parameters corresponding to the first audio frame from the preset average face model, and determine first model parameters of a preset digital human generation model based on the first fully connected layer parameters;

[0044] A generation module is used to generate a digital human image based on the first key point coordinates, the first remaining face image and the preset digital human generation model whose model parameters are the first model parameters.

[0045] Optionally, the extraction module includes:

[0046] A deconvolution unit, configured to perform a deconvolution process on the first fully connected layer parameters to obtain first intermediate parameters;

[0047] A first acquisition unit, used to acquire a second model parameter of the preset digital human generation model, where the second model parameter is a model parameter determined after the preset digital human generation model is trained;

[0048] The first combining unit is used to combine the first intermediate parameter with the second model parameter to obtain the first model parameter.

[0049] Optionally, the combination unit is further used for:

[0050] The intermediate parameters are respectively merged with the second model parameters corresponding to the encoding layer of the preset digital human generation model, the second model parameters corresponding to the hidden layer of the preset digital human generation model, and the second model parameters corresponding to the decoding layer of the preset digital human generation model to obtain the first model parameters.

[0051] Optionally, the digital human generation device further includes an average face model training module, and the average face training module includes:

[0052] A first extraction unit, configured to extract a plurality of second audio frames and second video frames corresponding to the second audio frames from a second video containing the first object;

[0053] A first input unit, configured to input the second audio frame into the preset average face model to obtain predicted coordinate values ​​of second face key points in a second video frame corresponding to the second audio frame;

[0054] A second extraction unit, used to extract the label coordinate value of the second facial key point in the second video frame;

[0055] A first calculation unit, configured to calculate a first loss value based on the predicted coordinate value and the label coordinate value;

[0056] a first adjustment unit, configured to adjust the model parameters of the preset average face model if the first loss value is greater than or equal to a preset first threshold, and execute the step of inputting the second audio frame into the preset average face model to obtain the predicted coordinate value of the second face key point in the second video frame corresponding to the second audio frame;

[0057] The first determination unit is used to determine that the training of the preset average face model is completed if the first loss value is less than a preset first threshold.

[0058] Optionally, the digital human generation device further includes a digital human generation model training module, and the digital human generation model training module includes:

[0059] A third acquisition unit, configured to acquire a second video containing the first object;

[0060] A fourth extraction unit is used to extract a second audio frame and a second video frame corresponding to the first audio frame from the second video, and to extract a second remaining face picture from which the target local area is removed from the second video frame;

[0061] A first generating unit, configured to generate second key point coordinates of facial key points in the target local area of ​​the first object based on the second audio frame and a preset average face model;

[0062] an extraction and determination unit, configured to extract second fully connected layer parameters corresponding to the second audio frame from the preset average face model, and determine third model parameters of the preset digital human generation model based on the second fully connected layer parameters, wherein the third model parameters are model parameters of the preset digital human generation model during this training;

[0063] A training unit, used for training the preset digital human generation model whose model parameters are the third model parameters by using the second key point coordinates and the second remaining face picture, so that the preset digital human generation model outputs a digital human prediction picture;

[0064] A third calculation unit, configured to calculate a second loss value based on the digital human prediction picture and the second video frame;

[0065] A second adjustment unit is used to adjust the model parameters of the preset digital human generation model if the second loss value is greater than or equal to a preset second threshold value, and execute the step of generating second key point coordinates of the facial key points in the target local area of ​​the first object based on the second audio frame and the preset average face model;

[0066] The second determination unit is used to determine that the training of the preset digital human generation model is completed if the second loss value is less than a preset second threshold.

[0067] Optionally, the extraction determination unit is further configured to:

[0068] When the training of the preset average face model is completed, obtaining the second fully connected layer parameters of the preset average face model;

[0069] Performing deconvolution processing on the second fully connected layer parameters to obtain second intermediate parameters;

[0070] Acquire a fourth model parameter of the preset digital human generation model, wherein the fourth model parameter is a model parameter of the preset digital human generation model before the current training;

[0071] The second intermediate parameter is combined with the fourth model parameter to obtain the third model parameter.

[0072] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0073] Memory, used to store computer programs;

[0074] The processor is used to implement any digital human generation method described in the first aspect when executing the program stored in the memory.

[0075] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program of a digital human generation method is stored. When the digital human generation method program is executed by a processor, the steps of any digital human generation method described in the first aspect are implemented.

[0076] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0077] The embodiment of the present invention generates the first key point coordinates of the key points of the face in the target local area by using the first audio frame of the first object in the first video, and extracts the first remaining face image after removing the target local area in the face of the first object in the first video frame, and then extracts the first fully connected layer parameters corresponding to the first audio frame in the preset average face model, and determines the first model parameters of the preset digital human generation model based on the first fully connected layer parameters, and finally combines the first key point coordinates, the first remaining face image and the preset digital human generation model whose model parameters are the first model parameters to generate a digital human image. Since data processing is always performed in two-dimensional level, the embodiment of the present invention can realize the process of generating digital human images more quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0079] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0080] Figure 1 A flowchart of a method for generating a digital human provided in an embodiment of the present application;

[0081] Figure 2 A structural diagram of a digital human generation device provided in an embodiment of the present application;

[0082] Figure 3 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0084] At present, the scheme for generating digital human images generally processes three-dimensional data, and the speed of generating digital human images is relatively slow. Therefore, the embodiment of the present application provides a digital human video generation method, which can be applied to a computer.

[0085] like Figure 1 As shown, the digital human generation method may include the following steps:

[0086] Step S101, obtaining a first video containing a first object;

[0087] In the embodiment of the present invention, the first video may be a video recorded of the first object in real time, or a non-real-time video (ie, recorded in advance), and the first object may be a real person.

[0088] Step S102, extracting a first audio frame and a first video frame corresponding to the first audio frame from the first video, and extracting a first remaining face picture with a target local area removed from the first video frame;

[0089] In an embodiment of the present invention, the first video includes multiple first audio frames and multiple first video frames, and the audio frames can be extracted according to the duration corresponding to the video frames. That is, each first audio frame corresponds to a first video frame. Exemplarily, the first video frames can be extracted from the first video at a frame rate of 25 frames per second, then the duration of each first audio frame can be 0.04 seconds.

[0090] The target local area in the embodiment of the present invention may refer to a partial area in the face of the first object, such as the mouth area of ​​the first object, etc. Correspondingly, the first remaining face image after removing the target local area may be the image area after removing the mouth area.

[0091] Step S103, generating first key point coordinates of facial key points in the target local area of ​​the first object based on the first audio frame and a preset average face model;

[0092] In an embodiment of the present invention, the preset average face model may be an ALM model, which may be a neural network model pre-trained using an audio frame containing a first object and a video frame corresponding to the audio frame, and the facial key points in the target local area may refer to the facial key points in a partial area of ​​the face of the first object, such as the facial key points in the mouth area of ​​the first object.

[0093] Since the first audio frame is extracted from the first video containing the first object, the first audio frame can reflect the opening and closing state of the mouth of the first object. Furthermore, the first key point coordinates of the facial key points in the target local area of ​​the first object can be generated by using the first audio frame and the preset average face model.

[0094] Step S104, extracting first fully connected layer parameters corresponding to the first audio frame from the preset average face model, and determining first model parameters of a preset digital human generation model based on the first fully connected layer parameters;

[0095] In the embodiment of the present invention, different first audio frames correspond to different fully connected layer parameters. After the first audio frame is input into the preset average face model, the fully connected layer parameters of the preset average face model can be extracted.

[0096] The first model parameters of the preset digital human generation model include two parts. The first part of parameters corresponds to the fully connected layer parameters, which are used for adjustment in the later stage, and the second part of parameters corresponds to the pre-trained parameters. In this step, the first part of parameters is determined based on the first fully connected layer parameters.

[0097] Step S105, generating a digital human image based on the first key point coordinates, the first remaining face image, and the preset digital human generation model whose model parameters are the first model parameters.

[0098] In an embodiment of the present invention, since the coordinates of the first key point correspond to the target local area of ​​the first object, the first residual face image includes the remaining part after removing the target local area in the face of the first object, and combined with the preset digital human generation model whose model parameters are the first model parameters, a digital human image can be generated.

[0099] The embodiment of the present invention generates the first key point coordinates of the key points of the face in the target local area by using the first audio frame of the first object in the first video, and extracts the first remaining face image after removing the target local area in the face of the first object in the first video frame, and then extracts the first fully connected layer parameters corresponding to the first audio frame in the preset average face model, and determines the first model parameters of the preset digital human generation model based on the first fully connected layer parameters, and finally combines the first key point coordinates, the first remaining face image and the preset digital human generation model whose model parameters are the first model parameters to generate a digital human image. Since data processing is always performed in two-dimensional level, the embodiment of the present invention can realize the process of generating digital human images more quickly.

[0100] In yet another embodiment of the present invention, determining first model parameters of a preset digital human generation model based on the first fully connected layer parameters includes:

[0101] Step 201, performing deconvolution processing on the first fully connected layer parameters to obtain first intermediate parameters;

[0102] In the embodiment of the present invention, the first fully connected layer parameters are used as intermediate hidden variables to represent feature parameters in a target local area (such as a mouth area).

[0103] In this step, the first fully connected layer parameters may be subjected to deconvolution calculation to obtain first intermediate parameters, which may represent feature parameters within a target local area (eg, a mouth area).

[0104] Step 202: Obtain a second model parameter of the preset digital human generation model.

[0105] In the embodiment of the present invention, the second model parameter is a model parameter determined after the preset digital human generation model is trained;

[0106] Step 203: Combine the first intermediate parameter with the second model parameter to obtain the first model parameter.

[0107] In an embodiment of the present invention, the preset digital human generation model includes a coding layer, a hidden layer and a decoding layer. Accordingly, the second part of the parameters in the first model parameters include: second model parameters corresponding to the coding layer, second model parameters corresponding to the hidden layer and second model parameters corresponding to the decoding layer.

[0108] In this step, the first intermediate parameter can be merged with the second model parameter corresponding to the encoding layer of the preset digital human generation model, the second model parameter corresponding to the hidden layer of the preset digital human generation model, and the second model parameter corresponding to the decoding layer of the preset digital human generation model to obtain the first model parameter.

[0109] In another embodiment of the present invention, the training method of the preset average face model includes:

[0110] Step 301, extracting a plurality of second audio frames and second video frames corresponding to the second audio frames from a second video containing the first object;

[0111] In an embodiment of the present invention, the second video may be a video recorded of the first object, and the second video may include multiple second audio frames and multiple second video frames. The audio frames may be extracted according to the duration corresponding to the video frames, that is, each second audio frame corresponds to a second video frame. Exemplarily, the second video frames may be extracted from the second video at a frame rate of 25 frames per second, and the duration of each second audio frame may be 0.04 seconds.

[0112] Step 302, inputting the second audio frame into the preset average face model to obtain predicted coordinate values ​​of second facial key points in a second video frame corresponding to the second audio frame;

[0113] In an embodiment of the present invention, the preset average face model can be a CNN / LSTM model, the input of the preset average face model is an audio frame, and the output is the predicted coordinate value of the second face key point in the second video frame corresponding to the second audio frame, and these key points of the second face key points are used to represent the average face generated based on sound reasoning, without including angles.

[0114] Step 303, extracting the label coordinate value of the second facial key point in the second video frame;

[0115] In the embodiment of the present invention, the coordinate values ​​of the second facial key points in the second video frame can be used as training labels, so in this step, the coordinate values ​​of the second facial key points in the second video frame can be extracted to obtain label coordinate values.

[0116] Step 304, calculating a first loss value based on the predicted coordinate value and the label coordinate value;

[0117] In this step, the loss function can be used to calculate the deviation of the predicted coordinate value relative to the label coordinate value to obtain a first loss value. Exemplarily, the loss function is L1 loss and wing loss.

[0118] Step 305: if the first loss value is greater than or equal to a preset first threshold, adjusting the model parameters of the preset average face model, and performing the step of inputting the second audio frame into the preset average face model to obtain the predicted coordinate value of the second face key point in the second video frame corresponding to the second audio frame;

[0119] Step 306: If the first loss value is less than a preset first threshold, it is determined that the training of the preset average face model is completed.

[0120] In another embodiment of the present invention, the training method of the preset digital human generation model includes:

[0121] Step 401, obtaining a second video containing the first object;

[0122] In the embodiment of the present invention, the second video may be a video recorded of the first object, and the second video includes a plurality of second audio frames and a plurality of second video frames.

[0123] In this step, a second video containing the first object for training a preset digital human generation model may be obtained.

[0124] Step 402: extracting a second audio frame and a second video frame corresponding to the first audio frame from the second video, and extracting a second remaining face image from the second video frame without the target local area;

[0125] In this step, audio frames can be extracted according to the duration corresponding to the video frames, that is, each second audio frame corresponds to a second video frame. For example, the second video frames can be extracted from the second video at a frame rate of 25 frames per second, then the duration of each second audio frame can be 0.04 seconds.

[0126] The target local area in the embodiment of the present invention may refer to a partial area in the face of the first object, such as the mouth area of ​​the first object, etc. Correspondingly, the second remaining face image after removing the target local area may be the face image after removing the mouth area.

[0127] Step 403, generating second key point coordinates of facial key points in the target local area of ​​the first object based on the second audio frame and a preset average face model;

[0128] In an embodiment of the present invention, the preset average face model may be an ALM model, which may be a neural network model pre-trained using an audio frame containing a first object and a video frame corresponding to the audio frame, and the second key point coordinates of the facial key points in the target local area may refer to the coordinates of the facial key points in a partial area of ​​the face of the first object, such as the coordinates of the facial key points in the mouth area of ​​the first object.

[0129] Since the second audio frame is extracted from the second video containing the first object, the second audio frame can reflect the opening and closing state of the mouth of the first object. Furthermore, the second audio frame and the preset average face model can be used to generate the second key point coordinates of the facial key points in the target local area of ​​the first object.

[0130] Step 404, extracting the second fully connected layer parameters corresponding to the second audio frame from the preset average face model, and determining the third model parameters of the preset digital human generation model based on the second fully connected layer parameters, wherein the third model parameters are the model parameters of the preset digital human generation model during this training;

[0131] In an embodiment of the present invention, different second audio frames correspond to different second fully connected layer parameters. After the first audio frame is input into a preset average face model, the second fully connected layer parameters of the preset average face model can be extracted.

[0132] The first model parameters of the preset digital human generation model include two parts. The first part of parameters corresponds to the fully connected layer parameters, which are used for adjustment in the later stage, and the second part of parameters corresponds to the pre-trained parameters. In this step, the first part of parameters is determined based on the second fully connected layer parameters.

[0133] Step 405, using the second key point coordinates, the second remaining face image and the training model parameters as the preset digital human generation model of the third model parameters, so that the preset digital human generation model outputs a digital human prediction image;

[0134] In an embodiment of the present invention, since the coordinates of the second key point correspond to the target local area of ​​the first object, the second remaining face image includes the remaining part after removing the target local area of ​​the face of the first object, and combined with the preset digital human generation model whose model parameters are the second model parameters, a digital human prediction image can be generated.

[0135] Step 406, calculating a second loss value based on the digital human prediction image and the second video frame;

[0136] In the embodiment of the present invention, the difference between the digital human prediction picture and the second video frame can be calculated by a loss function to obtain a second loss value.

[0137] Step 407, if the second loss value is greater than or equal to a preset second threshold, adjusting the model parameters of the preset digital human generation model, and executing the step of generating second key point coordinates of facial key points in the target local area of ​​the first object based on the second audio frame and the preset average face model;

[0138] Step 408: If the second loss value is less than a preset second threshold, it is determined that the training of the preset digital human generation model is completed.

[0139] In another embodiment of the present invention, determining the third model parameters of the preset digital human generation model based on the second fully connected layer parameters in step 504 includes:

[0140] Step 501, when the training of the preset average face model is completed, obtaining the second fully connected layer parameters of the preset average face model;

[0141] In the embodiment of the present invention, the second fully connected layer parameters are used as intermediate hidden variables to represent feature parameters in a target local area (such as a mouth area).

[0142] Step 502, performing deconvolution processing on the second fully connected layer parameters to obtain second intermediate parameters;

[0143] In this step, the second fully connected layer parameters may be subjected to deconvolution calculation to obtain second intermediate parameters, which may represent feature parameters within the target local area (eg, the mouth area).

[0144] Step 503: Obtain a fourth model parameter of the preset digital human generation model.

[0145] In the embodiment of the present invention, the fourth model parameter is the model parameter of the preset digital human generation model before the current training;

[0146] Step 504: Combine the second intermediate parameter with the fourth model parameter to obtain the third model parameter.

[0147] In an embodiment of the present invention, the preset digital human generation model includes a coding layer, a hidden layer and a decoding layer. Accordingly, the second part of the parameters in the fourth model parameters include: the fourth model parameters corresponding to the coding layer, the fourth model parameters corresponding to the hidden layer and the fourth model parameters corresponding to the decoding layer.

[0148] In this step, the second intermediate parameter can be merged with the fourth model parameter corresponding to the encoding layer of the preset digital human generation model, the fourth model parameter corresponding to the hidden layer of the preset digital human generation model, and the fourth model parameter corresponding to the decoding layer of the preset digital human generation model to obtain the third model parameter.

[0149] In another embodiment of the present invention, a digital human generation device is also provided. Figure 2 As shown, including:

[0150] An acquisition module 11 is used to acquire a first video containing a first object;

[0151] A removal module 12 is used to extract a first audio frame and a first video frame corresponding to the first audio frame from the first video, and extract a first remaining face image for removing a target local area from the first video frame;

[0152] A generating module 13, configured to generate first key point coordinates of facial key points in the target local area of ​​the first object based on the first audio frame and a preset average face model;

[0153] An extraction module 14 is used to extract first fully connected layer parameters corresponding to the first audio frame from the preset average face model, and determine first model parameters of a preset digital human generation model based on the first fully connected layer parameters;

[0154] The generation module 15 is used to generate a digital human image based on the first key point coordinates, the first remaining face image and the preset digital human generation model whose model parameters are the first model parameters.

[0155] Optionally, the extraction module includes:

[0156] A deconvolution unit, configured to perform a deconvolution process on the first fully connected layer parameters to obtain first intermediate parameters;

[0157] A first acquisition unit, used to acquire a second model parameter of the preset digital human generation model, where the second model parameter is a model parameter determined after the preset digital human generation model is trained;

[0158] The first combining unit is used to combine the first intermediate parameter with the second model parameter to obtain the first model parameter.

[0159] Optionally, the combination unit is further used for:

[0160] The intermediate parameters are respectively merged with the second model parameters corresponding to the encoding layer of the preset digital human generation model, the second model parameters corresponding to the hidden layer of the preset digital human generation model, and the second model parameters corresponding to the decoding layer of the preset digital human generation model to obtain the first model parameters.

[0161] Optionally, the digital human generation device further includes an average face model training module, and the average face training module includes:

[0162] A first extraction unit, configured to extract a plurality of second audio frames and second video frames corresponding to the second audio frames from a second video containing the first object;

[0163] A first input unit, configured to input the second audio frame into the preset average face model to obtain predicted coordinate values ​​of second face key points in a second video frame corresponding to the second audio frame;

[0164] A second extraction unit, used to extract the label coordinate value of the second facial key point in the second video frame;

[0165] A first calculation unit, configured to calculate a first loss value based on the predicted coordinate value and the label coordinate value;

[0166] a first adjustment unit, configured to adjust the model parameters of the preset average face model if the first loss value is greater than or equal to a preset first threshold, and execute the step of inputting the second audio frame into the preset average face model to obtain the predicted coordinate value of the second face key point in the second video frame corresponding to the second audio frame;

[0167] The first determination unit is used to determine that the training of the preset average face model is completed if the first loss value is less than a preset first threshold.

[0168] Optionally, the digital human generation device further includes a digital human generation model training module, and the digital human generation model training module includes:

[0169] A third acquisition unit, configured to acquire a second video containing the first object;

[0170] A fourth extraction unit is used to extract a second audio frame and a second video frame corresponding to the first audio frame from the second video, and to extract a second remaining face picture from which the target local area is removed from the second video frame;

[0171] A first generating unit, configured to generate second key point coordinates of facial key points in the target local area of ​​the first object based on the second audio frame and a preset average face model;

[0172] an extraction and determination unit, configured to extract second fully connected layer parameters corresponding to the second audio frame from the preset average face model, and determine third model parameters of the preset digital human generation model based on the second fully connected layer parameters, wherein the third model parameters are model parameters of the preset digital human generation model during this training;

[0173] A training unit, used for training the preset digital human generation model whose model parameters are the third model parameters by using the second key point coordinates and the second remaining face picture, so that the preset digital human generation model outputs a digital human prediction picture;

[0174] A third calculation unit, configured to calculate a second loss value based on the digital human prediction picture and the second video frame;

[0175] A second adjustment unit is used to adjust the model parameters of the preset digital human generation model if the second loss value is greater than or equal to a preset second threshold value, and execute the step of generating second key point coordinates of the facial key points in the target local area of ​​the first object based on the second audio frame and the preset average face model;

[0176] The second determination unit is used to determine that the training of the preset digital human generation model is completed if the second loss value is less than a preset second threshold.

[0177] Optionally, the extraction determination unit is further configured to:

[0178] When the training of the preset average face model is completed, obtaining the second fully connected layer parameters of the preset average face model;

[0179] Performing deconvolution processing on the second fully connected layer parameters to obtain second intermediate parameters;

[0180] Acquire a fourth model parameter of the preset digital human generation model, wherein the fourth model parameter is a model parameter of the preset digital human generation model before the current training;

[0181] The second intermediate parameter is combined with the fourth model parameter to obtain the third model parameter.

[0182] In another embodiment of the present invention, an electronic device is provided, including a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140;

[0183] Memory 1130, used for storing computer programs;

[0184] The processor is used to implement the digital human generation method described in any of the above method embodiments when executing the computer program stored in the memory 1130.

[0185] In an electronic device provided by an embodiment of the present invention, a processor implements the steps of acquiring a first video including a first object by executing a program stored in a memory; extracting a first audio frame and a first video frame corresponding to the first audio frame from the first video, and extracting a first remaining face picture after removing a target local area from the first video frame; generating first key point coordinates of key facial points in the target local area of ​​the first object based on the first audio frame and a preset average face model; extracting first fully connected layer parameters corresponding to the first audio frame from the preset average face model, and determining first model parameters of a preset digital human generation model based on the first fully connected layer parameters; and generating a digital human picture based on the first key point coordinates, the first remaining face picture, and the preset digital human generation model whose model parameters are the first model parameters.

[0186] The embodiment of the present invention generates the first key point coordinates of the key points of the face in the target local area by using the first audio frame of the first object in the first video, and extracts the first remaining face image after removing the target local area in the face of the first object in the first video frame, and then extracts the first fully connected layer parameters corresponding to the first audio frame in the preset average face model, and determines the first model parameters of the preset digital human generation model based on the first fully connected layer parameters, and finally combines the first key point coordinates, the first remaining face image and the preset digital human generation model whose model parameters are the first model parameters to generate a digital human image. Since data processing is always performed in two-dimensional level, the embodiment of the present invention can realize the process of generating digital human images more quickly.

[0187] The communication bus 1140 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0188] The communication interface 1120 is used for communication between the above electronic device and other devices.

[0189] The memory 1130 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0190] The above-mentioned processor 1110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0191] In another embodiment of the present invention, a computer-readable storage medium is provided, on which a program of a digital human generation method is stored. When the program of the digital human generation method is executed by a processor, the steps of the digital human generation method described in any of the aforementioned method embodiments are implemented.

[0192] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0193] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating a digital human, characterized in that: include: Obtain a first video containing a first object; Extracting a first audio frame and a first video frame corresponding to the first audio frame from the first video, and extracting a first remaining face picture after removing a target local area from the first video frame; generating first key point coordinates of key points of the face in the target local area of ​​the first object based on the first audio frame and a preset average face model; Extracting first fully connected layer parameters corresponding to the first audio frame from the preset average face model, and determining first model parameters of a preset digital human generation model based on the first fully connected layer parameters; Generate a digital human image based on the first key point coordinates, the first remaining human face image, and the preset digital human generation model whose model parameters are the first model parameters; Among them, determining the first model parameters of the preset digital human generation model based on the first fully connected layer parameters includes: performing deconvolution processing on the first fully connected layer parameters to obtain first intermediate parameters; obtaining second model parameters of the preset digital human generation model, wherein the second model parameters are model parameters determined after the preset digital human generation model is trained; and combining the first intermediate parameters with the second model parameters to obtain the first model parameters.

2. The method for generating a digital human according to claim 1, characterized in that: Combining the first intermediate parameter with the second model parameter to obtain the first model parameter includes: The intermediate parameters are respectively merged with the second model parameters corresponding to the encoding layer of the preset digital human generation model, the second model parameters corresponding to the hidden layer of the preset digital human generation model, and the second model parameters corresponding to the decoding layer of the preset digital human generation model to obtain the first model parameters.

3. The method for generating a digital human according to claim 1, characterized in that: The training method of the preset average face model includes: extracting a plurality of second audio frames and second video frames corresponding to the second audio frames from a second video containing the first object; Inputting the second audio frame into the preset average face model to obtain predicted coordinate values ​​of second facial key points in a second video frame corresponding to the second audio frame; Extracting the label coordinate value of the second facial key point in the second video frame; Calculate a first loss value based on the predicted coordinate value and the label coordinate value; If the first loss value is greater than or equal to a preset first threshold, adjusting the model parameters of the preset average face model, and performing the step of inputting the second audio frame into the preset average face model to obtain the predicted coordinate value of the second face key point in the second video frame corresponding to the second audio frame; If the first loss value is less than a preset first threshold, it is determined that the training of the preset average face model is completed.

4. The method for generating a digital human according to claim 1, characterized in that: The training method of the preset digital human generation model includes: Acquire a second video containing the first object; Extracting a second audio frame and a second video frame corresponding to the first audio frame from the second video, and extracting a second remaining face picture without the target local area from the second video frame; generating second key point coordinates of facial key points in the target local area of ​​the first object based on the second audio frame and a preset average face model; Extracting second fully connected layer parameters corresponding to the second audio frame from the preset average face model, and determining third model parameters of the preset digital human generation model based on the second fully connected layer parameters, wherein the third model parameters are model parameters of the preset digital human generation model during this training; Using the second key point coordinates and the second remaining face image to train the preset digital human generation model whose model parameters are the third model parameters, so that the preset digital human generation model outputs a digital human prediction image; Calculate a second loss value based on the digital human prediction picture and the second video frame; If the second loss value is greater than or equal to a preset second threshold, adjusting the model parameters of the preset digital human generation model, and executing the step of generating second key point coordinates of the facial key points in the target local area of ​​the first object based on the second audio frame and the preset average face model; If the second loss value is less than a preset second threshold, it is determined that the training of the preset digital human generation model is completed.

5. The method for generating a digital human according to claim 4, characterized in that: Determining third model parameters of a preset digital human generation model based on the second fully connected layer parameters includes: When the training of the preset average face model is completed, obtaining the second fully connected layer parameters of the preset average face model; Performing deconvolution processing on the second fully connected layer parameters to obtain second intermediate parameters; Acquire a fourth model parameter of the preset digital human generation model, wherein the fourth model parameter is a model parameter of the preset digital human generation model before the current training; The second intermediate parameter is combined with the fourth model parameter to obtain the third model parameter.

6. A digital human generation device, characterized in that: include: An acquisition module, used to acquire a first video containing a first object; A removal module, configured to extract a first audio frame and a first video frame corresponding to the first audio frame from the first video, and extract a first remaining face image of a target local area from the first video frame; A generating module, configured to generate first key point coordinates of key points of a face in the target local area of ​​the first object based on the first audio frame and a preset average face model; An extraction module, configured to extract first fully connected layer parameters corresponding to the first audio frame from the preset average face model, and determine first model parameters of a preset digital human generation model based on the first fully connected layer parameters; A generation module, configured to generate a digital human image based on the first key point coordinates, the first remaining face image, and the preset digital human generation model whose model parameters are the first model parameters; Wherein, the extraction module comprises: A deconvolution unit, configured to perform a deconvolution process on the first fully connected layer parameters to obtain first intermediate parameters; A first acquisition unit, used to acquire a second model parameter of the preset digital human generation model, where the second model parameter is a model parameter determined after the preset digital human generation model is trained; The first combining unit is used to combine the first intermediate parameter with the second model parameter to obtain the first model parameter.

7. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the digital human generation method described in any one of claims 1 to 5 when executing the program stored in the memory.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program of a digital human generation method, and when the program of the digital human generation method is executed by a processor, the steps of the digital human generation method described in any one of claims 1 to 5 are implemented.

Images