Digital human video generation method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a method, device, electronic device and storage medium for generating a digital human video. The method includes: obtaining the audio of a target person, extracting the audio features corresponding to any audio frame in the audio, and obtaining a first average face 3D mesh; determining a sample video of the target person, extracting the first face posture parameter and the first background contour line of the target sample image in the sample video; using a preset decoder, processing the audio features, the first face posture parameter and the first average face 3D mesh to generate the first face 3D mesh; using a preset video generation model, processing the first face 3D mesh and the first background contour line to generate a digital human video. The digital human video is generated based on the audio of the target person, wherein the first average face 3D mesh, the first face posture parameter of the target sample image and the first background contour line are introduced, and the digital human posture in the generated digital human video is natural and accurate.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for generating a digital human video. Background Art

[0002] The technology for generating digital human videos is becoming more and more mature. If digital human video generation technology is to be applied in business and realize real-time interaction, at least two points need to be met: good generation effect and fast reasoning speed. Among them, good production effect is a necessary prerequisite, and fast reasoning speed is a business need.

[0003] At present, the solution of generating digital human videos based on sound reasoning mainly uses sound reasoning models to generate facial key points. However, the facial key points generated based on sound reasoning have angle problems, resulting in unnatural postures of the digital humans in the final generated digital human videos. Summary of the invention

[0004] In order to solve the technical problem that the facial key points generated based on sound reasoning have angle problems, resulting in unnatural postures of the digital human in the final generated digital human video, the embodiments of the present invention provide a digital human video generation method, device, electronic device and storage medium.

[0005] In a first aspect of an embodiment of the present invention, a method for generating a digital human video is provided, the method comprising:

[0006] Acquire audio of a target person, extract audio features corresponding to any audio frame in the audio, and acquire a first average face 3D mesh;

[0007] Determine a sample video of the target person, and extract a first face posture parameter and a first background contour line of a target sample image in the sample video;

[0008] Using a preset decoder, processing the audio feature, the first facial posture parameter, and the first average facial 3D mesh to generate a first facial 3D mesh;

[0009] The first face 3D mesh and the first background contour are processed using a preset video generation model to generate a digital human video.

[0010] In an optional implementation, obtaining a first average face 3D mesh includes:

[0011] Inputting the target sample image into a preset 3D model, and obtaining first 3D face reconstruction information output by the preset 3D model;

[0012] Extracting a first facial posture parameter and a first facial expression parameter from the first 3D face reconstruction information;

[0013] Determining the first facial posture parameter and the first facial expression parameter as a first average facial 3D mesh;

[0014] or,

[0015] Obtain a first average face 3D mesh set, and obtain any first average face 3D mesh in the first average face 3D mesh set.

[0016] In an optional implementation, the extracting the first face posture parameter of the target sample image in the sample video includes:

[0017] Inputting the target sample image in the sample video into a preset posture extraction model, and obtaining the first face posture parameter output by the preset posture extraction model;

[0018] The preset posture extraction model includes a preset neural grid model and a fully connected layer, and the preset neural grid model is connected to the fully connected layer.

[0019] In an optional implementation, the using a preset decoder to process the audio feature, the first facial posture parameter, and the first average facial 3D mesh to generate the first facial 3D mesh includes:

[0020] For any audio frame in the audio, input the audio feature corresponding to the audio frame into a preset neural grid model;

[0021] Obtaining a first sound hidden variable corresponding to the audio frame output by the preset neural grid model;

[0022] Inputting the first average face 3D mesh into the preset neural mesh model, and obtaining a first image hidden variable output by the preset neural mesh model;

[0023] The first sound hidden variable, the first image hidden variable and the first face posture parameter corresponding to the audio frame are synthesized in a channel synthesis manner to obtain a first synthesized variable;

[0024] The first synthesis variable is input into a preset decoder to obtain a first face 3D mesh corresponding to the audio frame output by the preset decoder.

[0025] In an optional implementation, inputting the first synthesis variable into a preset decoder to obtain a first face 3D mesh corresponding to the audio frame output by the preset decoder includes:

[0026] Inputting the first facial posture parameter into the preset neural grid model, and obtaining a first mouth surrounding feature vector of the first facial posture parameter output by the preset neural grid model;

[0027] The first synthesis variable and the first mouth surrounding feature vector are input into a preset decoder, and a first face 3D mesh corresponding to the audio frame output by the preset decoder is obtained.

[0028] In an optional implementation, the using a preset video generation model to process the first face 3D mesh and the first background contour to generate a digital human video includes:

[0029] Synthesize the first face 3D mesh corresponding to the audio frame with the first background contour line in the channel synthesis manner to obtain a first synthesized image corresponding to the audio frame;

[0030] Inputting the first synthetic image corresponding to the audio frame into a preset video generation model, and obtaining a digital human image corresponding to the audio frame output by the preset video generation model;

[0031] According to the time sequence of the audio, the digital human images corresponding to the audio frames are combined to generate a digital human video.

[0032] In an optional implementation, extracting the audio feature corresponding to any audio frame in the audio includes:

[0033] For any audio frame in the audio, using Fourier transform to extract Mel frequency cepstrum coefficients as audio features corresponding to the audio frame;

[0034] or,

[0035] For any audio frame in the audio, an audio feature corresponding to the audio frame is extracted using a preset speech recognition model.

[0036] In an optional embodiment, before executing the method, the method further includes:

[0037] Acquire a sample video of a target person, wherein a sample image in the sample video corresponds one-to-one to a sample audio frame of a sample audio in the sample video;

[0038] For any of the sample audio frames in the sample audio, extract the sample audio features corresponding to the sample audio frame, and obtain a second average face 3D mesh; and,

[0039] Extracting a second facial posture parameter and a second background contour line in the sample image corresponding to the sample audio frame;

[0040] Using a preset decoder, processing the sample audio feature, the second facial posture parameter, and the second average facial 3D mesh to generate a second facial 3D mesh;

[0041] The second face 3D mesh and the second background contour are processed using a preset video generation model to generate a digital human sample video.

[0042] In an optional implementation, obtaining a second average face 3D mesh includes:

[0043] Inputting the sample image corresponding to the sample audio frame into a preset 3D model, and obtaining second 3D face reconstruction information output by the preset 3D model;

[0044] Extracting second facial posture parameters and second facial expression parameters from the second 3D face reconstruction information;

[0045] Determining the second facial posture parameter and the second facial expression parameter as a second average facial 3D mesh;

[0046] or,

[0047] Obtain a second average face 3D mesh set, and obtain any second average face 3D mesh in the second average face 3D mesh set.

[0048] In an optional implementation, the extracting the second facial posture parameter in the sample image corresponding to the sample audio frame includes:

[0049] Inputting the sample image corresponding to the sample audio frame into a preset posture extraction model, and obtaining a second face posture parameter output by the preset posture extraction model;

[0050] The preset posture extraction model includes a preset neural grid model and a fully connected layer, and the preset neural grid model is connected to the fully connected layer.

[0051] In an optional implementation, the using a preset decoder to process the sample audio feature, the second face posture parameter, and the second average face 3D mesh to generate the second face 3D mesh includes:

[0052] For any of the sample audio frames in the sample audio, inputting the sample audio features corresponding to the sample audio frame into a preset neural grid model;

[0053] Obtaining a second sound hidden variable corresponding to the sample audio frame output by the preset neural grid model;

[0054] Inputting the second average face 3D mesh into the preset neural mesh model, and obtaining a second image hidden variable output by the preset neural mesh model;

[0055] The second sound hidden variable, the second image hidden variable and the second face posture parameter corresponding to the sample audio frame are synthesized in a channel synthesis manner to obtain a second synthesized variable;

[0056] The second synthetic variable is input into a preset decoder to obtain a second face 3D mesh corresponding to the sample audio frame output by the preset decoder.

[0057] In an optional implementation, inputting the second synthesized variable into a preset decoder to obtain a second face 3D mesh corresponding to the sample audio frame output by the preset decoder includes:

[0058] Inputting the second facial posture parameter into the preset neural grid model, and obtaining a second mouth surrounding feature vector of the second facial posture parameter output by the preset neural grid model;

[0059] The second synthesis variable and the second mouth surrounding feature vector are input into a preset decoder to obtain a second face 3D mesh corresponding to the audio frame output by the preset decoder.

[0060] In an optional implementation, the using a preset video generation model to process the second face 3D mesh and the second background contour to generate a digital human sample video includes:

[0061] Synthesize the second face 3D mesh corresponding to the sample audio frame with the second background contour line in the channel synthesis manner to obtain a second synthesized image corresponding to the sample audio frame;

[0062] Inputting the second synthetic image corresponding to the sample audio frame into a preset video generation model, and obtaining a digital human sample image corresponding to the sample audio frame output by the preset video generation model;

[0063] According to the time sequence of the sample audio, the digital human sample images corresponding to the sample audio frames are combined to generate a digital human sample video.

[0064] In an optional implementation, for any of the sample audio frames in the sample audio, extracting a sample audio feature corresponding to the sample audio frame includes:

[0065] For any of the sample audio frames in the sample audio, extracting Mel-frequency cepstrum coefficients using Fourier transform as sample audio features corresponding to the sample audio frame;

[0066] or,

[0067] For any of the sample audio frames in the sample audio, a preset speech recognition model is used to extract sample audio features corresponding to the sample audio frame.

[0068] In a second aspect of an embodiment of the present invention, a digital human video generating device is provided, the device comprising:

[0069] An audio acquisition module, used to acquire the audio of the target person, extract the audio features corresponding to any audio frame in the audio, and acquire a first average face 3D mesh;

[0070] A parameter and contour extraction module, used to determine a sample video of the target person, and extract a first face posture parameter and a first background contour of a target sample image in the sample video;

[0071] A mesh generation module, configured to process the audio feature, the first facial posture parameter, and the first average facial 3D mesh using a preset decoder to generate a first facial 3D mesh;

[0072] The video generation module is used to process the first face 3D mesh and the first background contour line using a preset video generation model to generate a digital human video.

[0073] In an optional implementation, the audio acquisition module is specifically used to:

[0074] Inputting the target sample image into a preset 3D model, and obtaining first 3D face reconstruction information output by the preset 3D model;

[0075] Extracting a first facial posture parameter and a first facial expression parameter from the first 3D face reconstruction information;

[0076] Determining the first facial posture parameter and the first facial expression parameter as a first average facial 3D mesh;

[0077] or,

[0078] Obtain a first average face 3D mesh set, and obtain any first average face 3D mesh in the first average face 3D mesh set.

[0079] In an optional implementation, the parameter and contour extraction module is specifically used for:

[0080] Inputting the target sample image in the sample video into a preset posture extraction model, and obtaining the first face posture parameter output by the preset posture extraction model;

[0081] The preset posture extraction model includes a preset neural grid model and a fully connected layer, and the preset neural grid model is connected to the fully connected layer.

[0082] In an optional implementation, the grid generation module specifically includes:

[0083] An audio frame input submodule, for inputting the audio features corresponding to any audio frame in the audio into a preset neural grid model;

[0084] A variable acquisition module, used to acquire a first sound hidden variable corresponding to the audio frame output by the preset neural grid model;

[0085] A mesh input submodule, used for inputting the first average face 3D mesh into the preset neural mesh model, and obtaining a first image hidden variable output by the preset neural mesh model;

[0086] a variable synthesis submodule, configured to synthesize the first sound hidden variable, the first image hidden variable and the first face posture parameter corresponding to the audio frame in a channel synthesis manner to obtain a first synthesized variable;

[0087] The mesh generation submodule is used to input the first synthesis variable into a preset decoder to obtain a first face 3D mesh corresponding to the audio frame output by the preset decoder.

[0088] In an optional implementation, the grid generation submodule is specifically used for:

[0089] Inputting the first facial posture parameter into the preset neural grid model, and obtaining a first mouth surrounding feature vector of the first facial posture parameter output by the preset neural grid model;

[0090] The first synthesis variable and the first mouth surrounding feature vector are input into a preset decoder, and a first face 3D mesh corresponding to the audio frame output by the preset decoder is obtained.

[0091] In an optional implementation, the video generation module is specifically used to:

[0092] Synthesize the first face 3D mesh corresponding to the audio frame with the first background contour line in the channel synthesis manner to obtain a first synthesized image corresponding to the audio frame;

[0093] Inputting the first synthetic image corresponding to the audio frame into a preset video generation model, and obtaining a digital human image corresponding to the audio frame output by the preset video generation model;

[0094] According to the time sequence of the audio, the digital human images corresponding to the audio frames are combined to generate a digital human video.

[0095] In an optional implementation, the audio acquisition module is specifically used to:

[0096] For any audio frame in the audio, using Fourier transform to extract Mel frequency cepstrum coefficients as audio features corresponding to the audio frame;

[0097] or,

[0098] For any audio frame in the audio, an audio feature corresponding to the audio frame is extracted using a preset speech recognition model.

[0099] In an optional embodiment, the device further comprises:

[0100] A sample video acquisition module, used to acquire a sample video of a target person, wherein a sample image in the sample video corresponds one-to-one to a sample audio frame of a sample audio in the sample video;

[0101] An audio feature extraction module, used for extracting a sample audio feature corresponding to any sample audio frame in the sample audio;

[0102] A mesh acquisition module, used to acquire a second average face 3D mesh; and

[0103] A posture parameter extraction module, used to extract a second face posture parameter and a second background contour line in the sample image corresponding to the sample audio frame;

[0104] a 3D mesh generation module, configured to process the sample audio feature, the second face posture parameter and the second average face 3D mesh using a preset decoder to generate a second face 3D mesh;

[0105] The sample video generation module is used to process the second face 3D mesh and the second background contour line using a preset video generation model to generate a digital human sample video.

[0106] In an optional implementation, the grid acquisition module is specifically used to:

[0107] Inputting the sample image corresponding to the sample audio frame into a preset 3D model, and obtaining second 3D face reconstruction information output by the preset 3D model;

[0108] Extracting second facial posture parameters and second facial expression parameters from the second 3D face reconstruction information;

[0109] Determining the second facial posture parameter and the second facial expression parameter as a second average facial 3D mesh;

[0110] or,

[0111] Obtain a second average face 3D mesh set, and obtain any second average face 3D mesh in the second average face 3D mesh set.

[0112] In an optional implementation, the posture parameter extraction module is specifically used for:

[0113] Inputting the sample image corresponding to the sample audio frame into a preset posture extraction model, and obtaining a second face posture parameter output by the preset posture extraction model;

[0114] The preset posture extraction model includes a preset neural grid model and a fully connected layer, and the preset neural grid model is connected to the fully connected layer.

[0115] In an optional implementation, the 3D mesh generation module specifically includes:

[0116] An audio feature input submodule, for inputting the sample audio feature corresponding to any sample audio frame in the sample audio into a preset neural grid model;

[0117] A hidden variable acquisition submodule, used to acquire a second sound hidden variable corresponding to the sample audio frame output by the preset neural grid model;

[0118] a 3D mesh input submodule, configured to input the second average face 3D mesh into the preset neural mesh model, and obtain a second image hidden variable output by the preset neural mesh model;

[0119] a hidden variable synthesis submodule, configured to synthesize the second sound hidden variable, the second image hidden variable and the second face posture parameter corresponding to the sample audio frame in a channel synthesis manner to obtain a second synthesized variable;

[0120] The 3D mesh generation submodule is used to input the second synthesis variable into a preset decoder to obtain a second face 3D mesh corresponding to the sample audio frame output by the preset decoder.

[0121] In an optional embodiment, the second facial posture parameter is input into the preset neural grid model, and a second mouth surrounding feature vector of the second facial posture parameter output by the preset neural grid model is obtained;

[0122] The second synthesis variable and the second mouth surrounding feature vector are input into a preset decoder to obtain a second face 3D mesh corresponding to the audio frame output by the preset decoder.

[0123] In an optional implementation, the sample video generation module is specifically used to:

[0124] Synthesize the second face 3D mesh corresponding to the sample audio frame with the second background contour line in the channel synthesis manner to obtain a second synthesized image corresponding to the sample audio frame;

[0125] Inputting the second synthetic image corresponding to the sample audio frame into a preset video generation model, and obtaining a digital human sample image corresponding to the sample audio frame output by the preset video generation model;

[0126] According to the time sequence of the sample audio, the digital human sample images corresponding to the sample audio frames are combined to generate a digital human sample video.

[0127] In an optional implementation, the audio feature extraction module is specifically used to:

[0128] For any of the sample audio frames in the sample audio, extracting Mel-frequency cepstrum coefficients using Fourier transform as sample audio features corresponding to the sample audio frame;

[0129] or,

[0130] For any of the sample audio frames in the sample audio, a preset speech recognition model is used to extract sample audio features corresponding to the sample audio frame.

[0131] In a third aspect of an embodiment of the present invention, there is further provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0132] Memory, used to store computer programs;

[0133] The processor is used to implement the digital human video generation method described in the first aspect when executing the program stored in the memory.

[0134] In a fourth aspect of the embodiments of the present invention, a storage medium is further provided, wherein instructions are stored in the storage medium, and when the instructions are executed on a computer, the computer executes the digital human video generation method described in the first aspect.

[0135] In a fifth aspect of the embodiments of the present invention, a computer program product comprising instructions is also provided. When the computer program product is run on a computer, the computer executes the digital human video generation method described in the first aspect.

[0136] The technical solution provided by the embodiment of the present invention obtains the audio of the target person, extracts the audio features corresponding to any audio frame in the audio, obtains the first average face 3D grid, determines the sample video of the target person, extracts the first face posture parameter and the first background contour line of the target sample image in the sample video, uses a preset decoder to process the audio features, the first face posture parameter and the first average face 3D grid to generate the first face 3D grid, uses a preset video generation model to process the first face 3D grid and the first background contour line to generate a digital human video. In this way, based on the audio of the target person, a corresponding first face 3D grid is generated, and then based on the first face 3D grid, a digital human video is generated, wherein the first average face 3D grid, the first face posture parameter of the target sample image and the first background contour line are introduced, and the digital human posture in the finally generated digital human video is natural and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0137] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0138] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0139] Figure 1 It is a schematic diagram of an implementation flow of a method for generating a digital human video in an embodiment of the present invention;

[0140] Figure 2 Schematic diagram of the structure of an Image2Pose model shown in an embodiment of the present invention;

[0141] Figure 3 Schematic diagram of an implementation flow of a method for generating a first face 3D mesh (corresponding to each audio frame) shown in an embodiment of the present invention;

[0142] Figure 4 Schematic diagram of an implementation flow of another method for generating a first face 3D mesh (corresponding to each audio frame) in an embodiment of the present invention;

[0143] Figure 5It is a schematic diagram of an implementation flow of another method for generating a digital human video according to an embodiment of the present invention;

[0144] Figure 6 A schematic diagram of an implementation flow of a model training method shown in an embodiment of the present invention;

[0145] Figure 7 Schematic diagram of an implementation flow of a method for generating a second face 3D mesh (corresponding to each sample audio frame) shown in an embodiment of the present invention;

[0146] Figure 8 It is a structural schematic diagram of a digital human video generating device shown in an embodiment of the present invention;

[0147] Fig. 9 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0148] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0149] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0150] like Figure 1 FIG. 1 is a schematic diagram of an implementation flow of a method for generating a digital human video provided by an embodiment of the present invention. The method is applied to a processor and may specifically include the following steps:

[0151] S101, obtaining audio of a target person, extracting audio features corresponding to any audio frame in the audio, and obtaining a first average face 3D mesh.

[0152] In an embodiment of the present invention, the target person can input an audio clip by himself, and the audio clip can be in WAV format (WAV is a lossless audio file format), or other formats. The frame rate can be 100, 50 or 25, and this is not limited in the embodiment of the present invention.

[0153] Therefore, in the model inference stage, the embodiment of the present invention can obtain the audio of the target person, where the target person can be any current person, that is, any user, and the embodiment of the present invention does not limit this. For example, the embodiment of the present invention can obtain the audio of user A.

[0154] In addition, in the embodiment of the present invention, for the audio of the target person, for any audio frame in the audio of the target person, the audio features corresponding to the audio frame are extracted, and the first average face 3D mesh is obtained accordingly.

[0155] For example, for the audio of user A, assuming that there are 10 audio frames, for any audio frame in the audio of user A, the audio features corresponding to the audio frame are extracted, and the first average face 3D mesh is obtained accordingly.

[0156] Among them, for the extraction of audio features, Fourier transform can be used, that is, for any audio frame in the audio of the target person, Mel frequency cepstrum coefficients (MFCC) are extracted using Fourier transform as the audio features corresponding to the audio frame.

[0157] or,

[0158] For the extraction of audio features, other preset speech recognition models may be used, such as Deepspeech / ASR / wav2Vector and other models. That is, for any audio frame in the audio of the target person, the preset speech recognition model is used to extract the audio features corresponding to the audio frame.

[0159] As for the first average face 3D mesh, there are some open source first average face 3D meshes on the market, which can form a first average face 3D mesh set, and then in the process of processing any audio frame in the audio, the first average face 3D mesh set is obtained, and any first average face 3D mesh in the first average face 3D mesh set is obtained.

[0160] or,

[0161] For the target sample image (i.e., target image) in the sample video, the target sample image is input into a preset 3D model, where the preset 3D model can be an open source 3D model (such as 3D face reconstruction models such as Deep3DFaceReconstruction / DECA / 3DMM), and the first 3D face reconstruction information output by the preset 3D model is obtained. The first face posture parameters and the first face expression parameters in the first 3D face reconstruction information are extracted, which can be regarded as an average face, that is, the first face posture parameters and the first face expression parameters are determined to be the first average face 3D mesh.

[0162] S102, determining a sample video of the target person, and extracting a first facial posture parameter and a first background contour line of a target sample image in the sample video.

[0163] In an embodiment of the present invention, during the model training phase, sample videos of the target person are prepared for participating in the model training, so that in the subsequent model reasoning phase, the sample videos of the target person can be used to participate in the generation of digital human videos.

[0164] Based on this, a sample video of the target person is determined, a sample image frame is randomly selected from the sample video of the target person as a target sample image, and a first face posture parameter and a first background contour line of the target sample image in the sample video are extracted.

[0165] For example, determine the sample video of user A, select the first frame sample image from the sample video of user A as the target sample image (ie, target image), extract the first face posture parameter (pose) and the first background contour line (ie, background canny line graph) of the target sample image in the sample video.

[0166] It should be noted that, for a sample video, including multiple frames of sample images, any one frame of sample image can be selected as the target sample image. For example, the first frame of sample image in the sample video is selected as the target sample image. In the subsequent processing of each audio frame, this target sample image is used, that is, one sample image is fixedly used. The embodiment of the present invention does not limit this. In this way, the posture of the digital human in the final generated digital human video is fixed, which is the same as the head posture of the target sample image in the sample video.

[0167] In addition, in order to ensure the continuity of the posture, a frame of sample image can be selected from the sample video of the target person in sequence (the order here follows the processing order of audio frames in the audio, for example, processing the first frame of audio frame, selecting the first frame of sample image as the target sample image, processing the second frame of audio frame, and selecting the first frame of sample image as the target sample image, thereby following the processing order of audio frames in the audio) as the target sample image. In this way, the continuity of the digital human posture in the finally generated digital human video can be ensured, which is the same as the head posture in the sample video.

[0168] In addition, for extracting the first facial posture parameter, a preset posture extraction model can be used, that is, the target sample image in the sample video is input into the preset posture extraction model to obtain the first facial posture parameter output by the preset posture extraction model. The preset posture extraction model can be trained based on the sample image in the sample video to obtain the preset posture extraction model.

[0169] For example, assuming that the preset pose extraction model is an Image2Pose model, a target sample image (e.g., a target image) in a sample video is input into the Image2Pose model, and a first face pose parameter output by the Image2Pose model is obtained, wherein the Image2Pose model includes a preset neural grid model (e.g., CNN) and a fully connected layer, and the preset neural grid model is connected to the fully connected layer, such as Figure 2 shown.

[0170] It should be noted that the preset posture extraction model includes a preset neural grid model and a fully connected layer, and the preset neural grid model is connected to the fully connected layer, wherein the fully connected layer parameters can be used to characterize the facial posture feature vector. In addition, for the extraction of the first background contour line, reference can be made to the more mature technologies on the market, and the first background contour line refers to the contour line corresponding to the key points of the background (area other than the face) in the target sample image.

[0171] S103: Using a preset decoder, process the audio feature, the first facial posture parameter, and the first average facial 3D mesh to generate a first facial 3D mesh.

[0172] In an embodiment of the present invention, the corresponding audio features extracted from any audio frame in the audio, the first average face 3D mesh, and the first face posture parameters of the target sample image are processed using a preset decoder to generate a corresponding first face 3D mesh.

[0173] In an embodiment of the present invention, a preset decoder is used to process the corresponding audio features extracted from any audio frame in the audio, the first average face 3D grid, and the first face posture parameters of the target sample image, and the face 3D model is discarded, which can effectively speed up the model inference speed.

[0174] For example, for the first audio frame in the audio of user A, the preset decoder is used to process the audio features corresponding to the first audio frame, the first average face 3D mesh, and the first face posture parameters of the target sample image to generate a corresponding first face 3D mesh;

[0175] For the second audio frame in the audio of user A, using a preset decoder, the audio features corresponding to the second audio frame, the first average face 3D mesh, and the first face posture parameters of the target sample image are processed to generate a corresponding first face 3D mesh;

[0176] For other audio frames in the audio of user A, the processing is similar to the above, so that the corresponding first face 3D meshes can be obtained, that is, the first average face 3D mesh and the first face posture parameters of the target sample image are involved in the processing of each audio frame, as shown in Table 1 below.

[0177]

[0178]

[0179] Table 1

[0180] In the embodiment of the present invention, in the process of processing each audio frame, as shown in FIG. Figure 3 As shown, the first face 3D mesh corresponding to each audio frame can be obtained in the following way:

[0181] S301, for any audio frame in the audio, input the audio feature corresponding to the audio frame into a preset neural grid model.

[0182] S302, obtaining a first sound hidden variable corresponding to the audio frame output by the preset neural grid model.

[0183] In an embodiment of the present invention, for any audio frame in the audio, the audio feature corresponding to the audio frame is input into a preset neural grid model, and the first sound hidden variable corresponding to the audio frame output by the preset neural grid model is obtained.

[0184] For example, taking the first audio frame in the audio of user A as an example, the audio features corresponding to the first audio frame are input into CNN, and the corresponding first sound hidden variable W1 is generated after CNN convolution, and the first sound hidden variable W1 corresponding to the first audio frame output by CNN is obtained.

[0185] It should be noted that the preset neural grid model may be, for example, a CNN, or of course, an RNN, and the embodiment of the present invention does not limit this.

[0186] S303: Input the first average face 3D mesh into the preset neural mesh model to obtain a first image hidden variable output by the preset neural mesh model.

[0187] For the first average human face 3D mesh, the first average human face 3D mesh is input into a preset neural mesh model to obtain a first image hidden variable output by the preset neural mesh model.

[0188] For example, the first average face 3D grid is input into CNN, and the corresponding first image hidden variable T1 is generated after CNN convolution, and the first image hidden variable T1 output by CNN is obtained.

[0189] S304: synthesize the first sound hidden variable, the first image hidden variable and the first face posture parameter corresponding to the audio frame in a channel synthesis manner to obtain a first synthesized variable.

[0190] In the embodiment of the present invention, the first face posture parameter of the target sample image in the sample video can be considered as a posture hidden variable (i.e., a posture feature vector). Based on this, for the first sound hidden variable corresponding to any audio frame in the audio, the first sound hidden variable, the first image hidden variable, and the first face posture parameter corresponding to the audio frame are synthesized in a channel synthesis manner to obtain the first synthesized variable corresponding to the audio frame.

[0191] For example, the first facial posture parameter of the target sample image in the sample video can be considered as the posture hidden variable I1. Taking the first sound hidden variable W1 corresponding to the first audio frame in the audio of user A as an example, the first sound hidden variable W1 corresponding to the first audio frame, the first image hidden variable T1 and the posture hidden variable I1 are synthesized according to the channels to obtain the first synthesized variable (W1+T1+I1) corresponding to the first audio frame.

[0192] It should be noted that, assuming that the first sound hidden variable W1 has 5 channels, and the first image hidden variable T1 and the posture hidden variable I1 also have 5 channels, then according to the channel synthesis method, the first sound hidden variable W1, the first image hidden variable T1 and the posture hidden variable I1 are synthesized into 15 channels, that is, W1+T1+I1.

[0193] S305: Input the first synthesis variable into a preset decoder to obtain a first face 3D mesh corresponding to the audio frame output by the preset decoder.

[0194] After the above steps, for any audio frame in the audio, the corresponding first synthesis variable can be obtained, so that for the first synthesis variable corresponding to any audio frame in the audio, the first synthesis variable corresponding to the audio frame can be input into the preset decoder to obtain the first face 3D mesh corresponding to the audio frame output by the preset decoder.

[0195] For example, taking the first synthetic variable (W1+T1+I1) corresponding to the first frame of audio in the audio of user A as an example, the first synthetic variable (W1+T1+I1) corresponding to the first frame of audio is input into the decoder for processing, and the first face 3D mesh (carrying angle) corresponding to the first frame of audio finally output by the decoder is obtained.

[0196] In addition, in order to make the final generated digital human video more natural and accurate, the embodiment of the present invention can input the first facial posture parameter of the target sample image in the sample video into the preset neural grid model, obtain the first mouth surrounding feature vector of the first facial posture parameter output by the preset neural grid model, and input the first mouth surrounding feature vector into the preset decoding. In addition, the first synthetic variable passes through several layers of CNN layers and is then input into the preset decoder. At the same time, some layers of the decoder obtain the weight vector and the vector (first mouth surrounding feature vector) obtained by the head posture extracted by CNN, and add them through the channel. In order to enhance the influence of the head posture on the posture vector, it is easier to generate a 3D face mesh with head posture information, and obtain the first 3D face mesh corresponding to the audio frame output by the preset decoder.

[0197] For example, taking the first synthetic variable (W1+T1+I1) corresponding to the first audio frame in the audio of user A as an example, the first facial posture parameter of the target sample image (i.e., the target image) in the sample video is input into CNN, the first mouth surrounding feature vector of the first facial posture parameter output by CNN is obtained, and the first mouth surrounding feature vector is input into the decoder for processing. In addition, the first synthetic variable is passed through several CNN layers ( Figure 4 (not shown), and then input to the decoder for processing. At the same time, some layers of the decoder obtain the weight vector and the vector obtained by the head posture extracted by CNN (the first mouth surrounding feature vector), and add them through the channel. In order to enhance the influence of the head posture on the posture vector, it is easier to generate a 3D face mesh with head posture information, and obtain the first face 3D mesh (carrying angle) corresponding to the first frame of audio frame finally output by the decoder, such as Figure 4 shown.

[0198] It should be noted that, for the first face 3D mesh, the angle carried therein refers to head postures such as raising the head or tilting the head back, and the embodiment of the present invention does not limit this.

[0199] S104: Using a preset video generation model, the first face 3D mesh and the first background contour are processed to generate a digital human video.

[0200] After the above steps, for any audio frame in the audio, there is a corresponding first face 3D mesh. The first face 3D mesh and the first background contour line corresponding to the audio frame are processed using the preset video generation model to generate a digital human video.

[0201] For example, as shown in Table 1 above, for any audio frame in the audio of user A, there is a first face 3D mesh corresponding to each frame. GAN is used to process the first face 3D mesh and the first background contour line corresponding to the audio frame to generate a digital human video, such as Figure 5 shown.

[0202] For any audio frame in the audio, the first face 3D mesh corresponding to the audio frame is synthesized with the first background contour line in a channel synthesis manner to obtain a first synthesized image corresponding to the audio frame;

[0203] Inputting the first synthetic image corresponding to the audio frame into a preset video generation model, obtaining a digital human image corresponding to the audio frame output by the preset video generation model, so that for any audio frame in the audio, there is a corresponding digital human image;

[0204] For the digital human images corresponding to any audio frames in the audio, the digital human images corresponding to any audio frames in the audio are combined according to the time sequence of the audio, and finally a digital human video can be generated.

[0205] For example, for the first audio frame in the audio of user A, the first face 3D mesh corresponding to the first audio frame is synthesized with the first background contour line in a channel synthesis manner to obtain a first synthesized image corresponding to the first audio frame;

[0206] Input the first synthesized image corresponding to the first audio frame into GAN, and obtain the digital human image corresponding to the first audio frame output by GAN; for other audio frames in the audio of user A, the digital human image corresponding to each audio frame can be obtained in the same way as the above processing type;

[0207] For the digital human images corresponding to each audio frame in the audio of user A, the digital human images corresponding to each audio frame in the audio of user A are combined according to the time sequence of the audio of user A, and finally a digital human video can be generated.

[0208] It should be noted that the preset video generation model may specifically be a GAN in the embodiment of the present invention. Of course, it may also be other types of preset video generation models, or networks such as transformer or VAE, which are not limited in the embodiment of the present invention.

[0209] Through the above description of the technical solution provided by the embodiment of the present invention, the audio of the target person is obtained, and for any audio frame in the audio, the audio features corresponding to the audio frame are extracted, and the first average face 3D mesh is obtained, the sample video of the target person is determined, and the first face posture parameters and the first background contour line of the target sample image in the sample video are extracted. The audio features, the first face posture parameters and the first average face 3D mesh are processed by using a preset decoder to generate a first face 3D mesh, and the first face 3D mesh and the first background contour line are processed by using a preset video generation model to generate a digital human video.

[0210] In this way, based on the audio of the target person, the corresponding first face 3D mesh is generated, and then based on the first face 3D mesh, a digital human video is generated, in which the first average face 3D mesh, the first face posture parameters of the target sample image and the first background contour line are introduced, and the digital human posture in the final generated digital human video is natural and accurate. In addition, the 3D face reconstruction model is abandoned, which can speed up the model reasoning speed.

[0211] In addition, when executing Figure 1 Before the digital human video generation method shown in the figure, the embodiment of the present invention also needs to perform model training in advance, such as Figure 6 As shown, a schematic diagram of an implementation flow of a model training provided by an embodiment of the present invention is applied to a processor, and specifically may include the following steps:

[0212] S601, obtaining a sample video of a target person, wherein a sample image in the sample video corresponds one-to-one to a sample audio frame of a sample audio in the sample video.

[0213] During the model training stage, a sample video of the target person may be prepared, wherein the sample video of the target person includes multiple frames of sample images and sample audio, that is, multiple frames of sample images and sample audio of the target person, where the sample images correspond one-to-one to the sample audio frames of the sample audio.

[0214] Based on this, a sample video of the target person is obtained to participate in the training of the model. For example, a sample video of user A is obtained to participate in the training of the model, wherein the sample video of user A includes multiple frames of sample images and sample audio, where the sample images correspond one to one with the sample audio frames of the sample audio.

[0215] S602: For any of the sample audio frames in the sample audio, extract a sample audio feature corresponding to the sample audio frame, and obtain a second average face 3D mesh.

[0216] In an embodiment of the present invention, the sample audio includes multiple sample audio frames. For any sample audio frame in the sample audio, the sample audio features corresponding to the sample audio frame are extracted, and the second average face 3D mesh is obtained accordingly.

[0217] For example, for the sample audio, there are 10 frames of sample audio frames. For any sample audio frame in the sample audio, the sample audio features corresponding to the sample audio frame are extracted, and the second average face 3D mesh is obtained accordingly.

[0218] Among them, for extracting the sample audio features, Fourier transform can be used, that is, for any sample audio frame in the sample audio, Mel frequency cepstrum coefficients (ie MFCC) are extracted using Fourier transform as the sample audio features corresponding to the sample audio frame.

[0219] or,

[0220] For the extraction of sample audio features, other preset speech recognition models may be used, such as Deepspeech / ASR / wav2Vector and other models, that is, for any sample audio frame in the sample audio, the preset speech recognition model is used to extract the sample audio features corresponding to the sample audio frame.

[0221] As for the second average face 3D mesh, there are some open source second average face 3D meshes on the market, which can form a second average face 3D mesh set, and then in the process of processing any sample audio frame in the sample audio, the second average face 3D mesh set is obtained, and any second average face 3D mesh in the second average face 3D mesh set is obtained.

[0222] or,

[0223] In the process of processing any sample audio frame in the sample audio, for the sample image corresponding to the sample audio frame in the sample video, the sample image is input into a preset 3D model, where the preset 3D model can be an open source 3D model (for example, 3D face reconstruction models such as Deep3DFaceReconstruction / DECA / 3DMM), and the second 3D face reconstruction information output by the preset 3D model is obtained. The second face posture parameters and the second face expression parameters in the second 3D face reconstruction information are extracted, which can be regarded as an average face, that is, the second face posture parameters and the second face expression parameters are determined as the second average face 3D grid.

[0224] S603: Extract a second facial posture parameter and a second background contour line in the sample image corresponding to the sample audio frame.

[0225] In an embodiment of the present invention, for any sample audio frame in the sample audio, the sample audio features corresponding to the sample audio frame are extracted, and a second average face 3D mesh is obtained. In addition, second face posture parameters and a second background contour line in a sample image corresponding to the sample audio frame are extracted.

[0226] For example, taking the first frame of sample audio in the sample audio as an example, the sample audio features corresponding to the first frame of sample audio are extracted, and the second average face 3D mesh is obtained. In addition, the second face posture parameters and the second background contour line in the first frame of sample image corresponding to the first frame of sample audio are extracted.

[0227] It should be noted that, for the sample video, including multiple frames of sample images, in the process of processing any sample audio frame in the sample audio, the second facial posture parameters and the second background contour line in the sample image corresponding to the sample audio frame are extracted, and the embodiment of the present invention is not limited to this.

[0228] In addition, for extracting the second facial posture parameters, a preset posture extraction model can be used, that is, the sample image corresponding to the sample audio frame is input into the preset posture extraction model to obtain the second facial posture parameters output by the preset posture extraction model. The preset posture extraction model can be trained based on the sample image in the sample video to obtain the preset posture extraction model.

[0229] For example, taking the first sample audio frame in the sample audio as an example, assuming that the preset posture extraction model is the Image2Pose model, the sample image corresponding to the first sample audio frame (i.e., the target image) is input into the Image2Pose model to obtain the second face posture parameter output by the Image2Pose model, wherein the Image2Pose model is as follows: Figure 2 shown.

[0230] It should be noted that after the preset posture extraction model is trained, the weight parameters of the preset posture extraction model are fixed (i.e., the fully connected layer parameters of the preset posture extraction model are fixed, and the fully connected layer parameters can be used to characterize the facial posture feature vector). In addition, for the extraction of the second background contour line, reference can be made to the more mature technologies on the market, and the second background contour line refers to the contour line corresponding to the key points of the background (area other than the face) in the sample image corresponding to the sample audio frame.

[0231] S604: Using a preset decoder, process the sample audio feature, the second facial posture parameter, and the second average facial 3D mesh to generate a second facial 3D mesh.

[0232] In an embodiment of the present invention, the sample audio features, the second facial posture parameters and the second average facial 3D mesh corresponding to any sample audio frame in the sample audio are processed using a preset decoder to generate a corresponding second facial 3D mesh.

[0233] For example, for a first sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A), a preset decoder is used to process the sample audio features corresponding to the first sample audio frame, the second average face 3D mesh, and the second face posture parameters in the sample image corresponding to the first sample audio frame to generate a corresponding second face 3D mesh;

[0234] For a second sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A), using a preset decoder, the sample audio features corresponding to the second sample audio frame, the second average face 3D mesh, and the second face posture parameters in the sample image corresponding to the second sample audio frame are processed to generate a corresponding second face 3D mesh;

[0235] For other sample audio frames in the sample audio of user A (i.e., the sample audio included in the sample video of user A), the processing is similar to the above process. In this way, the corresponding second face 3D meshes, i.e., the second average face 3D mesh and the second face posture parameters of the sample image corresponding to the i-th sample audio frame can be obtained to participate in the processing of the i-th sample audio frame, as shown in Table 2 below.

[0236]

[0237] Table 2

[0238] In the embodiment of the present invention, in the process of processing each sample audio frame, as shown in FIG. Figure 7 As shown, the second face 3D mesh corresponding to each sample audio frame can be obtained in the following way:

[0239] S701, for any of the sample audio frames in the sample audio, input the sample audio features corresponding to the sample audio frame into a preset neural grid model.

[0240] S702, obtaining a second sound hidden variable corresponding to the sample audio frame output by the preset neural grid model.

[0241] In an embodiment of the present invention, for any sample audio frame in the sample audio, the sample audio feature corresponding to the sample audio frame is input into a preset neural grid model to obtain a second sound hidden variable corresponding to the sample audio frame output by the preset neural grid model.

[0242] For example, taking the first frame of sample audio in the sample audio of user A (i.e., the sample audio included in the sample video of user A) as an example, the sample audio features corresponding to the first frame of sample audio are input into CNN, and the corresponding second sound hidden variable W1 is generated after CNN convolution, and the second sound hidden variable W1 corresponding to the first frame of sample audio output by CNN is obtained.

[0243] It should be noted that the preset neural grid model may be, for example, CNN, or other neural grid models such as RNN, and the embodiment of the present invention is not limited to this.

[0244] S703: Input the second average face 3D mesh into the preset neural mesh model to obtain a second image hidden variable output by the preset neural mesh model.

[0245] For the second average human face 3D mesh, the second average human face 3D mesh is input into a preset neural mesh model to obtain a second image hidden variable output by the preset neural mesh model.

[0246] For example, the second average face 3D mesh is input into CNN, and the corresponding second image hidden variable T1 is generated after CNN convolution, and the second image hidden variable T1 output by CNN is obtained.

[0247] S704: synthesize the second sound hidden variable, the second image hidden variable and the second facial posture parameter corresponding to the sample audio frame in a channel synthesis manner to obtain a second synthesized variable.

[0248] For the second sound hidden variable corresponding to any sample audio frame in the sample audio, the second sound hidden variable, the second image hidden variable and the second facial posture parameter (of the sample image corresponding to the sample audio frame) corresponding to the sample audio frame are synthesized in a channel synthesis manner to obtain the second synthesized variable corresponding to the sample audio frame.

[0249] For example, taking the second sound hidden variable W1 corresponding to the first frame of sample audio in the sample audio of user A (i.e., the sample audio included in the sample video of user A) as an example, the second sound hidden variable W1 corresponding to the first frame of sample audio, the second image hidden variable T1 and the second facial posture parameter of the sample image corresponding to the first frame of sample audio (which can be considered as the posture hidden variable I1) are synthesized according to channels to obtain the second synthesized variable (W1+T1+I1) corresponding to the first frame of sample audio.

[0250] It should be noted that, assuming that the second sound hidden variable W1 has 5 channels, and the second image hidden variable T1 and the second facial posture parameter (which can be considered as the posture hidden variable I1) also have 5 channels, then according to the channel synthesis method, the second sound hidden variable W1, the second image hidden variable T1 and the second facial posture parameter (which can be considered as the posture hidden variable I1) are synthesized into 15 channels, that is, W1+T1+I1.

[0251] S705: Input the second synthesis variable to a preset decoder to obtain a second face 3D mesh corresponding to the sample audio frame output by the preset decoder.

[0252] After the above steps, for any sample audio frame in the sample audio, the corresponding second synthetic variable can be obtained, so that for the second synthetic variable corresponding to any sample audio frame in the sample audio, the second synthetic variable corresponding to the sample audio frame can be input into the preset decoder to obtain the second face 3D mesh corresponding to the sample audio frame output by the preset decoder.

[0253] For example, taking the second synthetic variable (W1+T1+I1) corresponding to the first frame of sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A) as an example, the second synthetic variable (W1+T1+I1) corresponding to the first frame of sample audio frame is input into the decoder for processing, and the second face 3D mesh (carrying angle) corresponding to the first frame of sample audio frame finally output by the decoder is obtained.

[0254] In addition, the embodiment of the present invention can input the second facial posture parameters of the sample image corresponding to the sample audio frame in the sample video into the preset neural grid model, obtain the second mouth surrounding feature vector of the second facial posture parameters output by the preset neural grid model, and input the second mouth surrounding feature vector into the preset decoder. In addition, the second synthetic variable passes through several layers of CNN layers and is then input into the preset decoder. At the same time, some layers of the decoder obtain the weight vector and the vector (second mouth surrounding feature vector) obtained by the head posture extracted by CNN, and add them through the channel. In order to enhance the influence of the head posture on the posture vector, it is easier to generate a 3D facial mesh with head posture information, and obtain the second 3D facial mesh corresponding to the audio frame output by the preset decoder.

[0255] For example, taking the second synthetic variable (W1+T1+I1) corresponding to the first frame of sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A) as an example, the second facial posture parameter of the sample image corresponding to the first frame of the sample audio frame is input into CNN, and the second mouth surrounding feature vector of the second facial posture parameter output by CNN is obtained, and the second mouth surrounding feature vector is input into the decoder for processing. In addition, the second synthetic variable is passed through several layers of CNN and then input into the decoder for processing. At the same time, some layers of the decoder obtain the weight vector and the vector (second mouth surrounding feature vector) obtained by the head posture extracted by CNN, and the weight vectors are added through the channels to enhance the influence of the head posture on the posture vector, so that the 3D face mesh with head posture information is generated more easily, and the second 3D face mesh (carrying angle) corresponding to the first frame of sample audio frame finally output by the decoder is obtained.

[0256] S605: Using a preset video generation model, process the second face 3D mesh and the second background contour to generate a digital human sample video.

[0257] After the above steps, for any sample audio frame in the sample audio, there is a corresponding second face 3D mesh. The preset video generation model is used to process the second face 3D mesh and the second background contour line corresponding to the sample audio frame to generate a digital human sample video.

[0258] For example, as shown in Table 2 above, for any sample audio frame in the sample audio of user A, there is a corresponding second face 3D mesh. Using GAN, the second face 3D mesh and the second background contour line corresponding to the sample audio frame are processed to generate a digital human sample video.

[0259] For any sample audio frame in the sample audio, the second face 3D mesh corresponding to the sample audio frame is synthesized with the second background contour line in a channel synthesis manner to obtain a second synthesized image corresponding to the sample audio frame;

[0260] Inputting the second synthetic image corresponding to the sample audio frame into a preset video generation model, obtaining a digital human sample image corresponding to the sample audio frame output by the preset video generation model, so that each sample audio frame has a corresponding digital human sample image;

[0261] Each sample audio frame has a corresponding digital human sample image. According to the time sequence of the sample audio, the digital human sample images corresponding to each sample audio frame are combined to obtain the final digital human sample video.

[0262] For example, for a first sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A), a second face 3D mesh corresponding to the first sample audio frame is synthesized with a second background contour line in a channel synthesis manner to obtain a second synthesized image corresponding to the first sample audio frame;

[0263] The second synthetic image corresponding to the first sample audio frame is input into the GAN, and the second face 3D mesh corresponding to the first sample audio frame output by the GAN is obtained; for other sample audio frames in the sample audio of user A (i.e., the sample audio included in the sample video of user A), similar processing is performed, and the digital human sample images corresponding to each sample audio frame can be obtained in this way;

[0264] For the digital human sample images corresponding to each sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A), the digital human sample images corresponding to each sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A) are combined according to the time sequence of the sample audio of user A, and finally a digital human sample video can be generated.

[0265] It should be noted that the preset video generation model may specifically be a GAN in the embodiment of the present invention, and may of course be other types of preset video generation models, which is not limited in the embodiment of the present invention.

[0266] In addition, the above steps S601 to S605 can be repeatedly executed until the model training meets the termination conditions of the model training, such as the loss function converges, or the digital human sample video is highly similar to the sample video, which is not limited in this embodiment of the present invention.

[0267] Corresponding to the above method embodiment, the embodiment of the present invention also provides a digital human video generation device, such as Figure 8 As shown, the device may include: an audio acquisition module 810 , a parameter and contour extraction module 820 , a grid generation module 830 , and a video generation module 840 .

[0268] The audio acquisition module 810 is used to acquire the audio of the target person, extract the audio features corresponding to any audio frame in the audio, and acquire a first average face 3D mesh;

[0269] A parameter and contour extraction module 820 is used to determine a sample video of the target person, and extract a first face posture parameter and a first background contour of a target sample image in the sample video;

[0270] A mesh generation module 830 is used to process the audio feature, the first face posture parameter and the first average face 3D mesh using a preset decoder to generate a first face 3D mesh;

[0271] The video generation module 840 is used to process the first face 3D mesh and the first background contour line using a preset video generation model to generate a digital human video.

[0272] The embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 91, a communication interface 92, a memory 93 and a communication bus 94, wherein the processor 91, the communication interface 92, and the memory 93 communicate with each other through the communication bus 94.

[0273] Memory 93, used for storing computer programs;

[0274] The processor 91 is used to execute the program stored in the memory 93 to implement the following steps:

[0275] The audio of the target person is obtained, and for any audio frame in the audio, the audio features corresponding to the audio frame are extracted, and a first average face 3D mesh is obtained; a sample video of the target person is determined, and a first face posture parameter and a first background contour line of the target sample image in the sample video are extracted; using a preset decoder, the audio features, the first face posture parameter and the first average face 3D mesh are processed to generate a first face 3D mesh; using a preset video generation model, the first face 3D mesh and the first background contour line are processed to generate a digital human video.

[0276] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0277] The communication interface is used for communication between the above electronic device and other devices.

[0278] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0279] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0280] In another embodiment of the present invention, a storage medium is provided. The storage medium stores instructions. When the instructions are executed on a computer, the computer executes the digital human video generation method described in any one of the above embodiments.

[0281] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the digital human video generation method described in any one of the above embodiments.

[0282] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium, or transmitted from one storage medium to another storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0283] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0284] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0285] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for generating a digital human video, characterized in that: The method comprises: Acquire audio of a target person, extract audio features corresponding to any audio frame in the audio, and acquire a first average face 3D mesh; Determine a sample video of the target person, and extract a first face posture parameter and a first background contour line of a target sample image in the sample video; Using a preset decoder, processing the audio feature, the first facial posture parameter, and the first average facial 3D mesh to generate a first facial 3D mesh; Using a preset video generation model, the first face 3D mesh and the first background contour are processed to generate a digital human video; The using of a preset decoder to process the audio feature, the first facial posture parameter, and the first average facial 3D mesh to generate a first facial 3D mesh includes: For any audio frame in the audio, input the audio feature corresponding to the audio frame into a preset neural grid model; Obtaining a first sound hidden variable corresponding to the audio frame output by the preset neural grid model; Inputting the first average face 3D mesh into the preset neural mesh model, and obtaining a first image hidden variable output by the preset neural mesh model; Synthesize the first sound hidden variable, the first image hidden variable, and the first face posture parameter corresponding to the audio frame in a channel synthesis manner to obtain a first synthesized variable; Inputting the first synthesis variable into a preset decoder, and obtaining a first face 3D mesh corresponding to the audio frame output by the preset decoder; The step of inputting the first synthesis variable into a preset decoder to obtain a first face 3D mesh corresponding to the audio frame output by the preset decoder includes: Inputting the first facial posture parameter into the preset neural grid model, and obtaining a first mouth surrounding feature vector of the first facial posture parameter output by the preset neural grid model; Inputting the first synthesis variable and the first feature vector around the mouth into a preset decoder, and obtaining a first face 3D mesh corresponding to the audio frame output by the preset decoder; The step of using a preset video generation model to process the first face 3D mesh and the first background contour line to generate a digital human video includes: Synthesize the first face 3D mesh corresponding to the audio frame with the first background contour line in the channel synthesis manner to obtain a first synthesized image corresponding to the audio frame; Inputting the first synthetic image corresponding to the audio frame into a preset video generation model, and obtaining a digital human image corresponding to the audio frame output by the preset video generation model; According to the time sequence of the audio, the digital human images corresponding to the audio frames are combined to generate a digital human video.

2. The method according to claim 1, characterized in that: The obtaining of a first average face 3D mesh comprises: Inputting the target sample image into a preset 3D model, and obtaining first 3D face reconstruction information output by the preset 3D model; Extracting a first facial posture parameter and a first facial expression parameter from the first 3D face reconstruction information; Determining the first facial posture parameter and the first facial expression parameter as a first average facial 3D mesh; or, Obtain a first average face 3D mesh set, and obtain any first average face 3D mesh in the first average face 3D mesh set.

3. The method according to claim 1, characterized in that The step of extracting a first face posture parameter of a target sample image in the sample video includes: Inputting the target sample image in the sample video into a preset posture extraction model, and obtaining the first face posture parameter output by the preset posture extraction model; The preset posture extraction model includes a preset neural grid model and a fully connected layer, and the preset neural grid model is connected to the fully connected layer.

4. The method according to claim 1, characterized in that The extracting, for any audio frame in the audio, an audio feature corresponding to the audio frame includes: For any audio frame in the audio, using Fourier transform to extract Mel frequency cepstrum coefficients as audio features corresponding to the audio frame; or, For any audio frame in the audio, an audio feature corresponding to the audio frame is extracted using a preset speech recognition model.

5. The method according to claim 1, characterized in that Before executing the method, the method further includes: Acquire a sample video of a target person, wherein a sample image in the sample video corresponds one-to-one to a sample audio frame of a sample audio in the sample video; For any of the sample audio frames in the sample audio, extract the sample audio features corresponding to the sample audio frame, and obtain a second average face 3D mesh; and, Extracting a second facial posture parameter and a second background contour line in the sample image corresponding to the sample audio frame; Using a preset decoder, processing the sample audio feature, the second facial posture parameter, and the second average facial 3D mesh to generate a second facial 3D mesh; The second face 3D mesh and the second background contour are processed using a preset video generation model to generate a digital human sample video.

6. The method according to claim 5, characterized in that The obtaining of the second average face 3D mesh comprises: Inputting the sample image corresponding to the sample audio frame into a preset 3D model, and obtaining second 3D face reconstruction information output by the preset 3D model; Extracting second facial posture parameters and second facial expression parameters from the second 3D face reconstruction information; Determining the second facial posture parameter and the second facial expression parameter as a second average facial 3D mesh; or, Obtain a second average face 3D mesh set, and obtain any second average face 3D mesh in the second average face 3D mesh set.

7. The method according to claim 5, characterized in that The step of extracting a second facial posture parameter from the sample image corresponding to the sample audio frame comprises: Inputting the sample image corresponding to the sample audio frame into a preset posture extraction model, and obtaining a second face posture parameter output by the preset posture extraction model; The preset posture extraction model includes a preset neural grid model and a fully connected layer, and the preset neural grid model is connected to the fully connected layer.

8. The method according to claim 5, characterized in that The method of using a preset decoder to process the sample audio feature, the second face posture parameter, and the second average face 3D mesh to generate a second face 3D mesh includes: For any of the sample audio frames in the sample audio, inputting the sample audio features corresponding to the sample audio frame into a preset neural grid model; Obtaining a second sound hidden variable corresponding to the sample audio frame output by the preset neural grid model; Inputting the second average face 3D mesh into the preset neural mesh model, and obtaining a second image hidden variable output by the preset neural mesh model; The second sound hidden variable, the second image hidden variable and the second face posture parameter corresponding to the sample audio frame are synthesized in a channel synthesis manner to obtain a second synthesized variable; The second synthetic variable is input into a preset decoder to obtain a second face 3D mesh corresponding to the sample audio frame output by the preset decoder.

9. The method according to claim 8, characterized in that The step of inputting the second synthesized variable into a preset decoder to obtain a second face 3D mesh corresponding to the sample audio frame output by the preset decoder includes: Inputting the second facial posture parameter into the preset neural grid model, and obtaining a second mouth surrounding feature vector of the second facial posture parameter output by the preset neural grid model; The second synthesis variable and the second mouth surrounding feature vector are input into a preset decoder to obtain a second face 3D mesh corresponding to the audio frame output by the preset decoder.

10. The method according to claim 8 or 9, characterized in that: The using of a preset video generation model to process the second face 3D mesh and the second background contour to generate a digital human sample video includes: Synthesize the second face 3D mesh corresponding to the sample audio frame with the second background contour line in the channel synthesis manner to obtain a second synthesized image corresponding to the sample audio frame; Inputting the second synthetic image corresponding to the sample audio frame into a preset video generation model, and obtaining a digital human sample image corresponding to the sample audio frame output by the preset video generation model; According to the time sequence of the sample audio, the digital human sample images corresponding to the sample audio frames are combined to generate a digital human sample video.

11. The method according to claim 5, characterized in that The extracting, for any of the sample audio frames in the sample audio, a sample audio feature corresponding to the sample audio frame comprises: For any of the sample audio frames in the sample audio, extracting Mel-frequency cepstrum coefficients using Fourier transform as sample audio features corresponding to the sample audio frame; or, For any of the sample audio frames in the sample audio, a preset speech recognition model is used to extract sample audio features corresponding to the sample audio frame.

12. A digital human video generation device, characterized in that: The device comprises: An audio acquisition module, used to acquire the audio of the target person, extract the audio features corresponding to any audio frame in the audio, and acquire a first average face 3D mesh; A parameter and contour extraction module, used to determine a sample video of the target person, and extract a first face posture parameter and a first background contour of a target sample image in the sample video; A mesh generation module, configured to process the audio feature, the first facial posture parameter, and the first average facial 3D mesh using a preset decoder to generate a first facial 3D mesh; A video generation module, used to process the first face 3D mesh and the first background contour line using a preset video generation model to generate a digital human video; The using of a preset decoder to process the audio feature, the first facial posture parameter, and the first average facial 3D mesh to generate a first facial 3D mesh includes: For any audio frame in the audio, input the audio feature corresponding to the audio frame into a preset neural grid model; Obtaining a first sound hidden variable corresponding to the audio frame output by the preset neural grid model; Inputting the first average face 3D mesh into the preset neural mesh model, and obtaining a first image hidden variable output by the preset neural mesh model; Synthesize the first sound hidden variable, the first image hidden variable, and the first face posture parameter corresponding to the audio frame in a channel synthesis manner to obtain a first synthesized variable; Inputting the first synthesis variable into a preset decoder, and obtaining a first face 3D mesh corresponding to the audio frame output by the preset decoder; The step of inputting the first synthesis variable into a preset decoder to obtain a first face 3D mesh corresponding to the audio frame output by the preset decoder includes: Inputting the first facial posture parameter into the preset neural grid model, and obtaining a first mouth surrounding feature vector of the first facial posture parameter output by the preset neural grid model; Inputting the first synthesis variable and the first feature vector around the mouth into a preset decoder, and obtaining a first face 3D mesh corresponding to the audio frame output by the preset decoder; The step of using a preset video generation model to process the first face 3D mesh and the first background contour line to generate a digital human video includes: Synthesize the first face 3D mesh corresponding to the audio frame with the first background contour line in the channel synthesis manner to obtain a first synthesized image corresponding to the audio frame; Inputting the first synthetic image corresponding to the audio frame into a preset video generation model, and obtaining a digital human image corresponding to the audio frame output by the preset video generation model; According to the time sequence of the audio, the digital human images corresponding to the audio frames are combined to generate a digital human video.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 11 when executing a program stored in a memory.

14. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images