Digital human video generation method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a method, device, electronic device and storage medium for generating a digital human video. The method includes: obtaining the audio of a target person, extracting the audio features corresponding to any audio frame in the audio, and obtaining a first average face 3D mesh; determining a sample video of the target person, extracting the first face posture parameter and the first background contour line of the target sample image in the sample video; using a preset face 3D model, processing the audio features, the first face posture parameter and the first average face 3D mesh to generate the first face 3D mesh; using a preset video generation model, processing the first face 3D mesh and the first background contour line to generate a digital human video. A digital human video is generated based on the audio of the target person, wherein the first average face 3D mesh, the first face posture parameter of the target sample image and the first background contour line are introduced, and the digital human posture in the generated digital human video is natural and accurate.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for generating a digital human video. Background Art

[0002] The technology for generating digital human videos is becoming more and more mature. If digital human video generation technology is to be applied in business and realize real-time interaction, at least two points need to be met: good generation effect and fast reasoning speed. Among them, good production effect is a necessary prerequisite, and fast reasoning speed is a business need.

[0003] At present, the solution of generating digital human videos based on sound reasoning mainly uses sound reasoning models to generate facial key points. However, the facial key points generated based on sound reasoning have angle problems, resulting in unnatural postures of the digital humans in the final generated digital human videos. Summary of the invention

[0004] In order to solve the technical problem that the facial key points generated based on sound reasoning have angle problems, resulting in unnatural postures of the digital human in the final generated digital human video, the embodiments of the present invention provide a digital human video generation method, device, electronic device and storage medium.

[0005] In a first aspect of an embodiment of the present invention, a method for generating a digital human video is provided, the method comprising:

[0006] Acquire audio of a target person, extract audio features corresponding to any audio frame in the audio, and acquire a first average face 3D mesh;

[0007] Determine a sample video of the target person, and extract a first face posture parameter and a first background contour line of a target sample image in the sample video;

[0008] Using a preset 3D face model, the audio feature, the first face posture parameter, and the first average 3D face mesh are processed to generate a first 3D face mesh;

[0009] The first face 3D mesh and the first background contour are processed using a preset video generation model to generate a digital human video.

[0010] In an optional implementation, obtaining a first average face 3D mesh includes:

[0011] Inputting the target sample image into a preset 3D model, and obtaining first 3D face reconstruction information output by the preset 3D model;

[0012] Extracting a first facial posture parameter and a first facial expression parameter from the first 3D face reconstruction information;

[0013] Determining the first facial posture parameter and the first facial expression parameter as a first average facial 3D mesh;

[0014] or,

[0015] Obtain a first average face 3D mesh set, and obtain any first average face 3D mesh in the first average face 3D mesh set.

[0016] In an optional implementation, the using a preset 3D face model to process the audio feature, the first face posture parameter, and the first average 3D face mesh to generate the first 3D face mesh includes:

[0017] For any audio frame in the audio, process the audio feature corresponding to the audio frame and the first average face 3D mesh to generate a first intermediate face 3D mesh;

[0018] The first intermediate 3D face mesh and the first face posture parameter are input into a preset 3D face model, and a first 3D face mesh corresponding to the audio frame output by the preset 3D face model is obtained.

[0019] In an optional implementation, the processing of the audio features corresponding to the audio frame and the first average face 3D mesh to generate a first intermediate face 3D mesh includes:

[0020] Inputting the audio feature corresponding to the audio frame into a preset neural grid model, and obtaining a first sound hidden variable corresponding to the audio frame output by the preset neural grid model;

[0021] Inputting the first average face 3D mesh into the preset neural mesh model, and obtaining a first image hidden variable output by the preset neural mesh model;

[0022] The first sound hidden variable and the first image hidden variable corresponding to the audio frame are synthesized in a channel synthesis manner to obtain a first synthesized variable;

[0023] The first synthesis variable is input into a preset decoder to obtain a first intermediate face 3D mesh corresponding to the audio frame output by the preset decoder.

[0024] In an optional implementation, the using a preset video generation model to process the first face 3D mesh and the first background contour to generate a digital human video includes:

[0025] Synthesize the first face 3D mesh corresponding to the audio frame with the first background contour line in the channel synthesis manner to obtain a first synthesized image corresponding to the audio frame;

[0026] Inputting the first synthetic image corresponding to the audio frame into a preset video generation model, and obtaining a digital human image corresponding to the audio frame output by the preset video generation model;

[0027] According to the time sequence of the audio, the digital human images corresponding to the audio frames are combined to generate a digital human video.

[0028] In an optional implementation, extracting the audio feature corresponding to any audio frame in the audio includes:

[0029] For any audio frame in the audio, using Fourier transform to extract Mel frequency cepstrum coefficients as audio features corresponding to the audio frame;

[0030] or,

[0031] For any audio frame in the audio, an audio feature corresponding to the audio frame is extracted using a preset speech recognition model.

[0032] In an optional embodiment, before executing the method, the method further includes:

[0033] Acquire a sample video of a target person, wherein a sample image in the sample video corresponds one-to-one to a sample audio frame of a sample audio in the sample video;

[0034] For any of the sample audio frames in the sample audio, extract the sample audio features corresponding to the sample audio frame, and obtain a second average face 3D mesh; and,

[0035] Extracting a second facial posture parameter and a second background contour line in the sample image corresponding to the sample audio frame;

[0036] Using a preset 3D face model, the sample audio feature, the second face posture parameter, and the second average 3D face mesh are processed to generate a second 3D face mesh;

[0037] The second face 3D mesh and the second background contour are processed using a preset video generation model to generate a digital human sample video.

[0038] In an optional implementation, obtaining a second average face 3D mesh includes:

[0039] Inputting the sample image corresponding to the sample audio frame into a preset 3D model, and obtaining second 3D face reconstruction information output by the preset 3D model;

[0040] Extracting second facial posture parameters and second facial expression parameters from the second 3D face reconstruction information;

[0041] Determining the second facial posture parameter and the second facial expression parameter as a second average facial 3D mesh;

[0042] or,

[0043] Obtain a second average face 3D mesh set, and obtain any second average face 3D mesh in the second average face 3D mesh set.

[0044] In an optional implementation, the using a preset 3D face model to process the sample audio feature, the second face posture parameter, and the second average 3D face mesh to generate the second 3D face mesh includes:

[0045] For any of the sample audio frames in the sample audio, processing the sample audio features corresponding to the sample audio frame and the second average face 3D mesh to generate a second intermediate face 3D mesh;

[0046] The second intermediate face 3D mesh and the second face posture parameter are input into a preset face 3D model, and a second face 3D mesh corresponding to the sample audio frame output by the preset face 3D model is obtained.

[0047] In an optional implementation, the processing of the sample audio features corresponding to the sample audio frame and the second average face 3D mesh to generate a second intermediate face 3D mesh includes:

[0048] Inputting the sample audio feature corresponding to the sample audio frame into a preset neural grid model, and obtaining a second sound hidden variable corresponding to the sample audio frame output by the preset neural grid model;

[0049] Inputting the second average face 3D mesh into the preset neural mesh model, and obtaining a second image hidden variable output by the preset neural mesh model;

[0050] The second sound hidden variable and the second image hidden variable corresponding to the sample audio frame are synthesized in a channel synthesis manner to obtain a second synthesized variable;

[0051] The second synthesis variable is input into a preset decoder to obtain a second intermediate face 3D mesh corresponding to the sample audio frame output by the preset decoder.

[0052] In an optional implementation, the using a preset video generation model to process the second face 3D mesh and the second background contour to generate a digital human sample video includes:

[0053] Synthesize the second face 3D mesh corresponding to the sample audio frame with the second background contour line in the channel synthesis manner to obtain a second synthesized image corresponding to the sample audio frame;

[0054] Inputting the second synthetic image corresponding to the sample audio frame into a preset video generation model, and obtaining a digital human sample image corresponding to the sample audio frame output by the preset video generation model;

[0055] According to the time sequence of the sample audio, the digital human sample images corresponding to the sample audio frames are combined to generate a digital human sample video.

[0056] In an optional implementation, for any of the sample audio frames in the sample audio, extracting a sample audio feature corresponding to the sample audio frame includes:

[0057] For any of the sample audio frames in the sample audio, extracting Mel-frequency cepstrum coefficients using Fourier transform as sample audio features corresponding to the sample audio frame;

[0058] or,

[0059] For any of the sample audio frames in the sample audio, a preset speech recognition model is used to extract sample audio features corresponding to the sample audio frame.

[0060] In a second aspect of an embodiment of the present invention, a digital human video generating device is provided, the device comprising:

[0061] An audio acquisition module, used to acquire the audio of the target person, extract the audio features corresponding to any audio frame in the audio, and acquire a first average face 3D mesh;

[0062] A parameter and contour extraction module, used to determine a sample video of the target person, and extract a first face posture parameter and a first background contour of a target sample image in the sample video;

[0063] A mesh generation module, configured to process the audio feature, the first face posture parameter and the first average face 3D mesh using a preset face 3D model to generate a first face 3D mesh;

[0064] The video generation module is used to process the first face 3D mesh and the first background contour line using a preset video generation model to generate a digital human video.

[0065] In an optional implementation, the audio acquisition module is specifically used to:

[0066] Inputting the target sample image into a preset 3D model, and obtaining first 3D face reconstruction information output by the preset 3D model;

[0067] Extracting a first facial posture parameter and a first facial expression parameter from the first 3D face reconstruction information;

[0068] Determining the first facial posture parameter and the first facial expression parameter as a first average facial 3D mesh;

[0069] or,

[0070] Obtain a first average face 3D mesh set, and obtain any first average face 3D mesh in the first average face 3D mesh set.

[0071] In an optional implementation, the grid generation module specifically includes:

[0072] an intermediate mesh generation submodule, configured to process, for any audio frame in the audio, the audio features corresponding to the audio frame and the first average face 3D mesh, to generate a first intermediate face 3D mesh;

[0073] The face mesh generation submodule is used to input the first intermediate face 3D mesh and the first face posture parameter into a preset face 3D model, and obtain a first face 3D mesh corresponding to the audio frame output by the preset face 3D model.

[0074] In an optional implementation, the audio feature corresponding to the audio frame is input into a preset neural grid model, and a first sound hidden variable corresponding to the audio frame output by the preset neural grid model is obtained;

[0075] Inputting the first average face 3D mesh into the preset neural mesh model, and obtaining a first image hidden variable output by the preset neural mesh model;

[0076] The first sound hidden variable and the first image hidden variable corresponding to the audio frame are synthesized in a channel synthesis manner to obtain a first synthesized variable;

[0077] The first synthesis variable is input into a preset decoder to obtain a first intermediate face 3D mesh corresponding to the audio frame output by the preset decoder.

[0078] In an optional implementation, the video generation module is specifically used to:

[0079] Synthesize the first face 3D mesh corresponding to the audio frame with the first background contour line in the channel synthesis manner to obtain a first synthesized image corresponding to the audio frame;

[0080] Inputting the first synthetic image corresponding to the audio frame into a preset video generation model, and obtaining a digital human image corresponding to the audio frame output by the preset video generation model;

[0081] According to the time sequence of the audio, the digital human images corresponding to the audio frames are combined to generate a digital human video.

[0082] In an optional implementation, the audio acquisition module is specifically used to:

[0083] For any audio frame in the audio, using Fourier transform to extract Mel frequency cepstrum coefficients as audio features corresponding to the audio frame;

[0084] or,

[0085] For any audio frame in the audio, an audio feature corresponding to the audio frame is extracted using a preset speech recognition model.

[0086] In an optional embodiment, the device further comprises:

[0087] A sample video acquisition module, used to acquire a sample video of a target person, wherein a sample image in the sample video corresponds one-to-one to a sample audio frame of a sample audio in the sample video;

[0088] An audio feature extraction module, used for extracting a sample audio feature corresponding to any sample audio frame in the sample audio;

[0089] A mesh acquisition module, used to acquire a second average face 3D mesh; and

[0090] A posture and background extraction module, used to extract a second face posture parameter and a second background contour line in the sample image corresponding to the sample audio frame;

[0091] a 3D mesh generation module, configured to process the sample audio features, the second face posture parameters and the second average face 3D mesh using a preset face 3D model to generate a second face 3D mesh;

[0092] The sample video generation module is used to process the second face 3D mesh and the second background contour line using a preset video generation model to generate a digital human sample video.

[0093] In an optional implementation, the grid acquisition module is specifically used to:

[0094] Inputting the sample image corresponding to the sample audio frame into a preset 3D model, and obtaining second 3D face reconstruction information output by the preset 3D model;

[0095] Extracting second facial posture parameters and second facial expression parameters from the second 3D face reconstruction information;

[0096] Determining the second facial posture parameter and the second facial expression parameter as a second average facial 3D mesh;

[0097] or,

[0098] Obtain a second average face 3D mesh set, and obtain any second average face 3D mesh in the second average face 3D mesh set.

[0099] In an optional implementation, the 3D mesh generation module specifically includes:

[0100] an intermediate 3D mesh generation submodule, configured to process, for any of the sample audio frames in the sample audio, the sample audio features corresponding to the sample audio frame and the second average face 3D mesh, to generate a second intermediate face 3D mesh;

[0101] The face 3D mesh generation submodule is used to input the second intermediate face 3D mesh and the second face posture parameter into a preset face 3D model, and obtain a second face 3D mesh corresponding to the sample audio frame output by the preset face 3D model.

[0102] In an optional implementation, the intermediate 3D grid generation submodule is specifically used for:

[0103] Inputting the sample audio feature corresponding to the sample audio frame into a preset neural grid model, and obtaining a second sound hidden variable corresponding to the sample audio frame output by the preset neural grid model;

[0104] Inputting the second average face 3D mesh into the preset neural mesh model, and obtaining a second image hidden variable output by the preset neural mesh model;

[0105] The second sound hidden variable and the second image hidden variable corresponding to the sample audio frame are synthesized in a channel synthesis manner to obtain a second synthesized variable;

[0106] The second synthesis variable is input into a preset decoder to obtain a second intermediate face 3D mesh corresponding to the sample audio frame output by the preset decoder.

[0107] In an optional implementation, the sample video generation module is specifically used to:

[0108] Synthesize the second face 3D mesh corresponding to the sample audio frame with the second background contour line in the channel synthesis manner to obtain a second synthesized image corresponding to the sample audio frame;

[0109] Inputting the second synthetic image corresponding to the sample audio frame into a preset video generation model, and obtaining a digital human sample image corresponding to the sample audio frame output by the preset video generation model;

[0110] According to the time sequence of the sample audio, the digital human sample images corresponding to the sample audio frames are combined to generate a digital human sample video.

[0111] In an optional implementation, the audio feature extraction module is specifically used to:

[0112] For any of the sample audio frames in the sample audio, extracting Mel-frequency cepstrum coefficients using Fourier transform as sample audio features corresponding to the sample audio frame;

[0113] or,

[0114] For any of the sample audio frames in the sample audio, a preset speech recognition model is used to extract sample audio features corresponding to the sample audio frame.

[0115] In a third aspect of an embodiment of the present invention, there is further provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0116] Memory, used to store computer programs;

[0117] The processor is used to implement the digital human video generation method described in the first aspect when executing the program stored in the memory.

[0118] In a fourth aspect of the embodiments of the present invention, a storage medium is further provided, wherein instructions are stored in the storage medium, and when the instructions are executed on a computer, the computer executes the digital human video generation method described in the first aspect.

[0119] In a fifth aspect of the embodiments of the present invention, a computer program product comprising instructions is also provided. When the computer program product is run on a computer, the computer executes the digital human video generation method described in the first aspect.

[0120] The technical solution provided by the embodiment of the present invention obtains the audio of the target person, extracts the audio features corresponding to any audio frame in the audio, obtains the first average face 3D grid, determines the sample video of the target person, extracts the first face posture parameter and the first background contour line of the target sample image in the sample video, uses the preset face 3D model to process the audio features, the first face posture parameter and the first average face 3D grid to generate the first face 3D grid, uses the preset video generation model to process the first face 3D grid and the first background contour line to generate a digital human video. In this way, based on the audio of the target person, a corresponding first face 3D grid is generated, and then based on the first face 3D grid, a digital human video is generated, wherein the first average face 3D grid, the first face posture parameter of the target sample image and the first background contour line are introduced, and the digital human posture in the finally generated digital human video is natural and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0122] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0123] Figure 1 It is a schematic diagram of an implementation flow of a method for generating a digital human video in an embodiment of the present invention;

[0124] Figure 2 It is a schematic diagram of an implementation flow of a method for generating a first intermediate face 3D mesh (corresponding to each audio frame) shown in an embodiment of the present invention;

[0125] Figure 3 It is a schematic diagram of an implementation flow of another method for generating a digital human video according to an embodiment of the present invention;

[0126] Figure 4 A schematic diagram of an implementation flow of a model training method shown in an embodiment of the present invention;

[0127] Figure 5It is a schematic diagram of an implementation flow of a second intermediate face 3D mesh generation method (corresponding to each sample audio frame) shown in an embodiment of the present invention;

[0128] Figure 6 It is a structural schematic diagram of a digital human video generating device shown in an embodiment of the present invention;

[0129] Figure 7 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0130] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0131] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0132] like Figure 1 FIG. 1 is a schematic diagram of an implementation flow of a method for generating a digital human video provided by an embodiment of the present invention. The method is applied to a processor and may specifically include the following steps:

[0133] S101, obtaining audio of a target person, extracting audio features corresponding to any audio frame in the audio, and obtaining a first average face 3D mesh.

[0134] In an embodiment of the present invention, the target person can input an audio clip by himself, and the audio clip can be in WAV format (WAV is a lossless audio file format), or other formats. The frame rate can be 100, 50 or 25, and this is not limited in the embodiment of the present invention.

[0135] Therefore, in the model inference stage, the embodiment of the present invention can obtain the audio of the target person, where the target person can be any current person, that is, any user, and the embodiment of the present invention does not limit this. For example, the embodiment of the present invention can obtain the audio of user A.

[0136] In addition, in the embodiment of the present invention, for the audio of the target person, for any audio frame in the audio of the target person, the audio features corresponding to the audio frame are extracted, and the first average face 3D mesh is obtained accordingly.

[0137] For example, for the audio of user A, assuming that there are 10 audio frames, for any audio frame in the audio of user A, the audio features corresponding to the audio frame are extracted, and the first average face 3D mesh is obtained accordingly.

[0138] Among them, for the extraction of audio features, Fourier transform can be used, that is, for any audio frame in the audio of the target person, Mel frequency cepstrum coefficients (MFCC) are extracted using Fourier transform as the audio features corresponding to the audio frame.

[0139] or,

[0140] For the extraction of audio features, other preset speech recognition models may be used, such as Deepspeech / ASR / wav2Vector and other models. That is, for any audio frame in the audio of the target person, the preset speech recognition model is used to extract the audio features corresponding to the audio frame.

[0141] As for the first average face 3D mesh, there are some open source first average face 3D meshes on the market, which can form a first average face 3D mesh set, and then in the process of processing any audio frame in the audio, the first average face 3D mesh set is obtained, and any first average face 3D mesh in the first average face 3D mesh set is obtained.

[0142] or,

[0143] For the target sample image (i.e., target image) in the sample video, the target sample image is input into a preset 3D model, where the preset 3D model can be an open source 3D model (such as 3D face reconstruction models such as Deep3DFaceReconstruction / DECA / 3DMM), and the first 3D face reconstruction information output by the preset 3D model is obtained. The first face posture parameters and the first face expression parameters in the first 3D face reconstruction information are extracted, which can be regarded as an average face, that is, the first face posture parameters and the first face expression parameters are determined to be the first average face 3D mesh.

[0144] S102, determining a sample video of the target person, and extracting a first facial posture parameter and a first background contour line of a target sample image in the sample video.

[0145] In an embodiment of the present invention, during the model training phase, sample videos of the target person are prepared for participating in the model training, so that in the subsequent model reasoning phase, the sample videos of the target person can be used to participate in the generation of digital human videos.

[0146] Based on this, a sample video of the target person is determined, a sample image frame is randomly selected from the sample video of the target person as a target sample image, and a first face posture parameter and a first background contour line of the target sample image in the sample video are extracted.

[0147] For example, determine the sample video of user A, select the first frame sample image from the sample video of user A as the target sample image (ie, target image), extract the first face posture parameter (pose) and the first background contour line (ie, background canny line graph) of the target sample image in the sample video.

[0148] It should be noted that, for a sample video, including multiple frames of sample images, any one frame of sample image can be selected as the target sample image. For example, the first frame of sample image in the sample video is selected as the target sample image. In the subsequent processing of each audio frame, this target sample image is used, that is, one sample image is fixedly used. The embodiment of the present invention does not limit this. In this way, the posture of the digital human in the final generated digital human video is fixed, which is the same as the head posture of the target sample image in the sample video.

[0149] In addition, in order to ensure the continuity of the posture, a frame of sample image can be selected from the sample video of the target person in sequence (the order here follows the processing order of audio frames in the audio, for example, processing the first frame of audio frame, selecting the first frame of sample image as the target sample image, processing the second frame of audio frame, and selecting the first frame of sample image as the target sample image, thereby following the processing order of audio frames in the audio) as the target sample image. In this way, the continuity of the digital human posture in the finally generated digital human video can be ensured, which is the same as the head posture in the sample video.

[0150] In addition, for the extraction of the first face posture parameter and the first background contour line, reference can be made to relatively mature technologies on the market, and the embodiments of the present invention will not be described in detail here. It should be emphasized that the first background contour line refers to the contour line corresponding to the key point of the background (the area other than the face) in the target sample image.

[0151] S103: Using a preset 3D face model, the audio feature, the first face posture parameter, and the first average 3D face mesh are processed to generate a first 3D face mesh.

[0152] In an embodiment of the present invention, the corresponding audio features extracted from any audio frame in the audio, the first average face 3D mesh, and the first face posture parameters of the target sample image are processed using a preset face 3D model to generate a corresponding first face 3D mesh.

[0153] For example, for the first audio frame in the audio of user A, the preset 3D face model is used to process the audio features corresponding to the first audio frame, the first average 3D face mesh, and the first face posture parameters of the target sample image to generate a corresponding first 3D face mesh;

[0154] For the second audio frame in the audio of user A, the preset 3D face model is used to process the audio features corresponding to the second audio frame, the first average 3D face mesh, and the first face posture parameters of the target sample image to generate a corresponding first 3D face mesh;

[0155] For other audio frames in the audio of user A, the processing is similar to the above, so that the corresponding first face 3D meshes can be obtained, that is, the first average face 3D mesh and the first face posture parameters of the target sample image are involved in the processing of each audio frame, as shown in Table 1 below.

[0156]

[0157] Table 1

[0158] Among them, for any audio frame in the audio, the audio features corresponding to the audio frame and the first average face 3D mesh are processed to generate a first intermediate face 3D mesh, and the first intermediate face 3D mesh and the first face posture parameters are input into the preset face 3D model to obtain the first face 3D mesh corresponding to the audio frame output by the preset face 3D model.

[0159] For example, for the first audio frame in the audio of user A, the audio features corresponding to the first audio frame and the first average face 3D mesh are processed to generate a first intermediate face 3D mesh, the first intermediate face 3D mesh and the first face posture parameter are input into the FLAME model, and the first face 3D mesh corresponding to the first audio frame output by the FLAME model is obtained;

[0160] For the second audio frame in the audio of user A, the audio features corresponding to the second audio frame and the first average face 3D mesh are processed to generate a first intermediate face 3D mesh, the first intermediate face 3D mesh and the first face posture parameter are input into the FLAME model, and the first face 3D mesh corresponding to the second audio frame output by the FLAME model is obtained;

[0161] For other audio frames in the audio of user A, similar to the above processing, the corresponding first face 3D meshes can be obtained, that is, each audio frame has a corresponding first face 3D mesh, which means that the first average face 3D mesh and the first face posture parameters of the target sample image participate in the processing of each audio frame, as shown in Table 1 above.

[0162] It should be noted that the first intermediate face 3D mesh does not carry an angle, while the first face 3D mesh carries an angle, which refers to head postures such as raising the head, tilting the head, and shaking the head, which is not limited in the embodiment of the present invention. In addition, the preset face 3D model here can be a FLAME model, and of course it can also be a 3DMM / Deep3DFaceReconstruction / DECA, which is not limited in the embodiment of the present invention.

[0163] In the embodiment of the present invention, during the process of processing each audio frame, as shown in FIG. Figure 2 As shown, the first intermediate face 3D mesh corresponding to each audio frame can be obtained in the following way:

[0164] S201, inputting the audio feature corresponding to the audio frame into a preset neural grid model, and obtaining a first sound hidden variable corresponding to the audio frame output by the preset neural grid model.

[0165] In an embodiment of the present invention, for any audio frame in the audio, the audio feature corresponding to the audio frame is input into a preset neural grid model, and the first sound hidden variable corresponding to the audio frame output by the preset neural grid model is obtained.

[0166] For example, taking the first audio frame in the audio of user A as an example, the audio features corresponding to the first audio frame are input into CNN, and the corresponding first sound hidden variable W1 is generated after CNN convolution, and the first sound hidden variable W1 corresponding to the first audio frame output by CNN is obtained.

[0167] It should be noted that the preset neural grid model may be, for example, a CNN, or of course, an RNN, and the embodiment of the present invention does not limit this.

[0168] S202: Input the first average face 3D mesh into the preset neural mesh model to obtain a first image hidden variable output by the preset neural mesh model.

[0169] For the first average human face 3D mesh, the first average human face 3D mesh is input into a preset neural mesh model to obtain a first image hidden variable output by the preset neural mesh model.

[0170] For example, the first average face 3D grid is input into CNN, and the corresponding first image hidden variable T1 is generated after CNN convolution, and the first image hidden variable T1 output by CNN is obtained.

[0171] S203: synthesize the first sound hidden variable and the first image hidden variable corresponding to the audio frame in a channel synthesis manner to obtain a first synthesized variable.

[0172] For a first sound hidden variable corresponding to any audio frame in the audio, the first sound hidden variable corresponding to the audio frame is synthesized with the first image hidden variable in a channel synthesis manner to obtain a first synthesized variable corresponding to the audio frame.

[0173] For example, taking the first sound hidden variable W1 corresponding to the first audio frame in the audio of user A as an example, the first sound hidden variable W1 corresponding to the first audio frame and the first image hidden variable T1 are synthesized according to the channel to obtain the first synthesized variable (W1+T1) corresponding to the first audio frame.

[0174] It should be noted that, assuming that the first sound hidden variable W1 has 5 channels, and the first image hidden variable T1 also has 5 channels, then according to the channel synthesis method, the first sound hidden variable W1 and the first image hidden variable T1 are synthesized into 10 channels, namely W1+T1.

[0175] S204: Input the first synthesis variable to a preset decoder to obtain a first intermediate face 3D mesh corresponding to the audio frame output by the preset decoder.

[0176] After the above steps, for any audio frame in the audio, the corresponding first synthetic variable can be obtained, so that for the first synthetic variable corresponding to any audio frame in the audio, the first synthetic variable corresponding to the audio frame can be input into the preset decoder to obtain the first intermediate face 3D mesh corresponding to the audio frame output by the preset decoder.

[0177] For example, taking the first synthetic variable (W1+T1) corresponding to the first frame of audio in the audio of user A as an example, the first synthetic variable (W1+T1) corresponding to the first frame of audio is input into the decoder for processing, and the first intermediate face 3D mesh (without angle) corresponding to the first frame of audio finally output by the decoder is obtained.

[0178] S104: Using a preset video generation model, the first face 3D mesh and the first background contour are processed to generate a digital human video.

[0179] After the above steps, for any audio frame in the audio, there is a corresponding first face 3D mesh. The first face 3D mesh and the first background contour line corresponding to the audio frame are processed using the preset video generation model to generate a digital human video.

[0180] For example, as shown in Table 1 above, for any audio frame in the audio of user A, there is a first face 3D mesh corresponding to each frame. GAN is used to process the first face 3D mesh and the first background contour line corresponding to the audio frame to generate a digital human video, such as Figure 3 shown.

[0181] For any audio frame in the audio, the first face 3D mesh corresponding to the audio frame is synthesized with the first background contour line in a channel synthesis manner to obtain a first synthesized image corresponding to the audio frame;

[0182] Inputting the first synthetic image corresponding to the audio frame into a preset video generation model, obtaining a digital human image corresponding to the audio frame output by the preset video generation model, so that for any audio frame in the audio, there is a corresponding digital human image;

[0183] For the digital human images corresponding to any audio frames in the audio, the digital human images corresponding to any audio frames in the audio are combined according to the time sequence of the audio, and finally a digital human video can be generated.

[0184] For example, for the first audio frame in the audio of user A, the first face 3D mesh corresponding to the first audio frame is synthesized with the first background contour line in a channel synthesis manner to obtain a first synthesized image corresponding to the first audio frame;

[0185] Input the first synthesized image corresponding to the first audio frame into GAN, and obtain the digital human image corresponding to the first audio frame output by GAN; for other audio frames in the audio of user A, the digital human image corresponding to each audio frame can be obtained in the same way as the above processing type;

[0186] For the digital human images corresponding to each audio frame in the audio of user A, the digital human images corresponding to each audio frame in the audio of user A are combined according to the time sequence of the audio of user A, and finally a digital human video can be generated.

[0187] It should be noted that the preset video generation model may specifically be a GAN in the embodiment of the present invention. Of course, it may also be other types of preset video generation models, or networks such as transformer or VAE, which are not limited in the embodiment of the present invention.

[0188] Through the above description of the technical solution provided by the embodiment of the present invention, the audio of the target person is obtained, and for any audio frame in the audio, the audio features corresponding to the audio frame are extracted, and the first average face 3D mesh is obtained, the sample video of the target person is determined, and the first face posture parameters and the first background contour line of the target sample image in the sample video are extracted. The audio features, the first face posture parameters and the first average face 3D mesh are processed using a preset face 3D model to generate a first face 3D mesh, and the first face 3D mesh and the first background contour line are processed using a preset video generation model to generate a digital human video.

[0189] In this way, based on the audio of the target person, a corresponding first face 3D mesh is generated, and then based on the first face 3D mesh, a digital human video is generated, wherein the first average face 3D mesh, the first face posture parameter of the target sample image and the first background contour line are introduced, and the posture of the digital human in the digital human video finally generated is natural and accurate. In addition, the FLAME model is used in the embodiment of the present invention, so that the posture of the digital human in the digital human video finally generated can be further natural and accurate.

[0190] In addition, when executing Figure 1 Before the digital human video generation method shown in the figure, the embodiment of the present invention also needs to perform model training in advance, such as Figure 4 As shown, a schematic diagram of an implementation flow of a model training provided by an embodiment of the present invention is applied to a processor, and specifically may include the following steps:

[0191] S401, obtaining a sample video of a target person, wherein a sample image in the sample video corresponds one-to-one to a sample audio frame of a sample audio in the sample video.

[0192] During the model training stage, a sample video of the target person may be prepared, wherein the sample video of the target person includes multiple frames of sample images and sample audio, that is, multiple frames of sample images and sample audio of the target person, where the sample images correspond one-to-one to the sample audio frames of the sample audio.

[0193] Based on this, a sample video of the target person is obtained to participate in the training of the model. For example, a sample video of user A is obtained to participate in the training of the model, wherein the sample video of user A includes multiple frames of sample images and sample audio, where the sample images correspond one to one with the sample audio frames of the sample audio.

[0194] S402: For any of the sample audio frames in the sample audio, extract a sample audio feature corresponding to the sample audio frame, and obtain a second average face 3D mesh.

[0195] In an embodiment of the present invention, the sample audio includes multiple sample audio frames. For any sample audio frame in the sample audio, the sample audio features corresponding to the sample audio frame are extracted, and the second average face 3D mesh is obtained accordingly.

[0196] For example, for the sample audio, there are 10 frames of sample audio frames. For any sample audio frame in the sample audio, the sample audio features corresponding to the sample audio frame are extracted, and the second average face 3D mesh is obtained accordingly.

[0197] Among them, for extracting the sample audio features, Fourier transform can be used, that is, for any sample audio frame in the sample audio, Mel frequency cepstrum coefficients (ie MFCC) are extracted using Fourier transform as the sample audio features corresponding to the sample audio frame.

[0198] or,

[0199] For the extraction of sample audio features, other preset speech recognition models may be used, such as Deepspeech / ASR / wav2Vector and other models, that is, for any sample audio frame in the sample audio, the preset speech recognition model is used to extract the sample audio features corresponding to the sample audio frame.

[0200] As for the second average face 3D mesh, there are some open source second average face 3D meshes on the market, which can form a second average face 3D mesh set, and then in the process of processing any sample audio frame in the sample audio, the second average face 3D mesh set is obtained, and any second average face 3D mesh in the second average face 3D mesh set is obtained.

[0201] or,

[0202] In the process of processing any sample audio frame in the sample audio, for the sample image corresponding to the sample audio frame in the sample video, the sample image is input into a preset 3D model, where the preset 3D model can be an open source 3D model (for example, 3D face reconstruction models such as Deep3DFaceReconstruction / DECA / 3DMM), and the second 3D face reconstruction information output by the preset 3D model is obtained. The second face posture parameters and the second face expression parameters in the second 3D face reconstruction information are extracted, which can be regarded as an average face, that is, the second face posture parameters and the second face expression parameters are determined as the second average face 3D grid.

[0203] S403: Extract a second facial posture parameter and a second background contour line in the sample image corresponding to the sample audio frame.

[0204] In an embodiment of the present invention, for any sample audio frame in the sample audio, the sample audio features corresponding to the sample audio frame are extracted, and a second average face 3D mesh is obtained. In addition, second face posture parameters and a second background contour line in a sample image corresponding to the sample audio frame are extracted.

[0205] For example, taking the first frame of sample audio in the sample audio as an example, the sample audio features corresponding to the first frame of sample audio are extracted, and the second average face 3D mesh is obtained. In addition, the second face posture parameters and the second background contour line in the first frame of sample image corresponding to the first frame of sample audio are extracted.

[0206] It should be noted that, for the sample video, including multiple frames of sample images, in the process of processing any sample audio frame in the sample audio, the second facial posture parameters and the second background contour line in the sample image corresponding to the sample audio frame are extracted, and the embodiment of the present invention is not limited to this.

[0207] In addition, for the extraction of the second facial posture parameter and the second background contour line, reference can be made to relatively mature technologies on the market, and the embodiments of the present invention will not be described in detail here. It should be emphasized that the second background contour line refers to the contour line corresponding to the key point of the background (the area other than the face) in the sample image corresponding to the sample audio frame.

[0208] S404: Using a preset 3D face model, the sample audio feature, the second face posture parameter, and the second average 3D face mesh are processed to generate a second 3D face mesh.

[0209] In an embodiment of the present invention, the sample audio features, the second facial posture parameters and the second average facial 3D mesh corresponding to any sample audio frame in the sample audio are processed using a preset facial 3D model to generate a corresponding second facial 3D mesh.

[0210] For example, for a first sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A), a preset face 3D model is used to process the sample audio features corresponding to the first sample audio frame, the second average face 3D mesh, and the second face posture parameters in the sample image corresponding to the first sample audio frame to generate a corresponding second face 3D mesh;

[0211] For the second sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A), using a preset 3D face model, the sample audio features corresponding to the second sample audio frame, the second average 3D face mesh, and the second face posture parameters in the sample image corresponding to the second sample audio frame are processed to generate a corresponding second 3D face mesh;

[0212] For other sample audio frames in the sample audio of user A (i.e., the sample audio included in the sample video of user A), the processing is similar to the above process. In this way, the corresponding second face 3D meshes, i.e., the second average face 3D mesh and the second face posture parameters of the sample image corresponding to the i-th sample audio frame can be obtained to participate in the processing of the i-th sample audio frame, as shown in Table 2 below.

[0213]

[0214] Table 2

[0215] Among them, for any sample audio frame in the sample audio, the sample audio features corresponding to the sample audio frame and the second average face 3D mesh are processed to generate a second intermediate face 3D mesh, and the second intermediate face 3D mesh and the second face posture parameters (in the sample image corresponding to the sample audio frame) are input into the preset face 3D model to obtain the second face 3D mesh corresponding to the sample audio frame output by the preset face 3D model.

[0216] For example, for the first sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A), the sample audio features and the second average face 3D mesh corresponding to the first sample audio frame are processed to generate a corresponding second intermediate face 3D mesh, the second intermediate face 3D mesh and the second face posture parameter (in the sample image corresponding to the first sample audio frame) are input into the FLAME model, and the second face 3D mesh corresponding to the first sample audio frame output by the FLAME model is obtained;

[0217] For the second sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A), the sample audio features and the second average face 3D mesh corresponding to the second sample audio frame are processed to generate a corresponding second intermediate face 3D mesh, the second intermediate face 3D mesh and the second face posture parameter (in the sample image corresponding to the second sample audio frame) are input into the FLAME model, and the second face 3D mesh corresponding to the second sample audio frame output by the FLAME model is obtained;

[0218] For other sample audio frames in the sample audio of user A (i.e., the sample audio included in the sample video of user A), similar to the above processing, the corresponding second face 3D meshes, i.e., the second average face 3D mesh and the second face posture parameters of the sample image corresponding to the i-th sample audio frame can be obtained to participate in the processing of the i-th sample audio frame, as shown in Table 2 above.

[0219] It should be noted that the second intermediate face 3D mesh does not carry an angle, while the second face 3D mesh carries an angle, which refers to head postures such as raising the head, tilting the head, and shaking the head, which is not limited in the embodiment of the present invention. In addition, the preset face 3D model here can be a FLAME model, and of course it can also be a 3DMM / Deep3DFaceReconstruction / DECA, which is not limited in the embodiment of the present invention.

[0220] In the embodiment of the present invention, during the process of processing each sample audio frame, as shown in FIG. Figure 5 As shown, the second intermediate face 3D mesh corresponding to each sample audio frame can be obtained in the following way:

[0221] S501, inputting the sample audio feature corresponding to the sample audio frame into a preset neural grid model, and obtaining a second sound hidden variable corresponding to the sample audio frame output by the preset neural grid model.

[0222] In an embodiment of the present invention, for any sample audio frame in the sample audio, the sample audio feature corresponding to the sample audio frame is input into a preset neural grid model to obtain a second sound hidden variable corresponding to the sample audio frame output by the preset neural grid model.

[0223] For example, taking the first frame of sample audio in the sample audio of user A (i.e., the sample audio included in the sample video of user A) as an example, the sample audio features corresponding to the first frame of sample audio are input into CNN, and the corresponding second sound hidden variable W1 is generated after CNN convolution, and the second sound hidden variable W1 corresponding to the first frame of sample audio output by CNN is obtained.

[0224] It should be noted that the preset neural grid model may be, for example, CNN, or other neural grid models such as RNN, and the embodiment of the present invention is not limited to this.

[0225] S502: Input the second average face 3D mesh into the preset neural mesh model to obtain a second image hidden variable output by the preset neural mesh model.

[0226] For the second average human face 3D mesh, the second average human face 3D mesh is input into a preset neural mesh model to obtain a second image hidden variable output by the preset neural mesh model.

[0227] For example, the second average face 3D mesh is input into CNN, and the corresponding second image hidden variable T1 is generated after CNN convolution, and the second image hidden variable T1 output by CNN is obtained.

[0228] S503: synthesize the second sound hidden variable and the second image hidden variable corresponding to the sample audio frame in a channel synthesis manner to obtain a second synthesized variable.

[0229] For the second sound hidden variable corresponding to any sample audio frame in the sample audio, the second sound hidden variable corresponding to the sample audio frame is synthesized with the second image hidden variable in a channel synthesis manner to obtain the second synthesized variable corresponding to the sample audio frame.

[0230] For example, taking the second sound hidden variable W1 corresponding to the first frame sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A) as an example, the second sound hidden variable W1 corresponding to the first frame sample audio frame is synthesized with the second image hidden variable T1 according to the channel to obtain the second synthesized variable (W1+T1) corresponding to the first frame sample audio frame.

[0231] It should be noted that, assuming that the second sound hidden variable W1 has 5 channels, and the second image hidden variable T1 also has 5 channels, the second sound hidden variable W1 and the second image hidden variable T1 are synthesized into 10 channels, namely W1+T1, in accordance with the channel synthesis method.

[0232] S504: Input the second synthesis variable to a preset decoder to obtain a second intermediate face 3D mesh corresponding to the sample audio frame output by the preset decoder.

[0233] After the above steps, for any sample audio frame in the sample audio, the corresponding second synthetic variable can be obtained, so that for the second synthetic variable corresponding to any sample audio frame in the sample audio, the second synthetic variable corresponding to the sample audio frame can be input into the preset decoder to obtain the second intermediate face 3D mesh corresponding to the sample audio frame output by the preset decoder.

[0234] For example, taking the second synthetic variable (W1+T1) corresponding to the first frame of sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A) as an example, the second synthetic variable (W1+T1) corresponding to the first frame of sample audio frame is input into the decoder for processing, and the second intermediate face 3D mesh (without angle) corresponding to the first frame of sample audio frame finally output by the decoder is obtained.

[0235] S405: Using a preset video generation model, process the second face 3D mesh and the second background contour to generate a digital human sample video.

[0236] After the above steps, for any sample audio frame in the sample audio, there is a corresponding second face 3D mesh. The preset video generation model is used to process the second face 3D mesh and the second background contour line corresponding to the sample audio frame to generate a digital human sample video.

[0237] For example, as shown in Table 2 above, for any sample audio frame in the sample audio of user A, there is a corresponding second face 3D mesh. Using GAN, the second face 3D mesh and the second background contour line corresponding to the sample audio frame are processed to generate a digital human sample video.

[0238] For any sample audio frame in the sample audio, the second face 3D mesh corresponding to the sample audio frame is synthesized with the second background contour line in a channel synthesis manner to obtain a second synthesized image corresponding to the sample audio frame;

[0239] Inputting the second synthetic image corresponding to the sample audio frame into a preset video generation model, obtaining a digital human sample image corresponding to the sample audio frame output by the preset video generation model, so that each sample audio frame has a corresponding digital human sample image;

[0240] Each sample audio frame has a corresponding digital human sample image. According to the time sequence of the sample audio, the digital human sample images corresponding to each sample audio frame are combined to obtain the final digital human sample video.

[0241] For example, for a first sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A), a second face 3D mesh corresponding to the first sample audio frame is synthesized with a second background contour line in a channel synthesis manner to obtain a second synthesized image corresponding to the first sample audio frame;

[0242] The second synthetic image corresponding to the first sample audio frame is input into the GAN, and the second face 3D mesh corresponding to the first sample audio frame output by the GAN is obtained; for other sample audio frames in the sample audio of user A (i.e., the sample audio included in the sample video of user A), similar processing is performed, and the digital human sample images corresponding to each sample audio frame can be obtained in this way;

[0243] For the digital human sample images corresponding to each sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A), the digital human sample images corresponding to each sample audio frame in the sample audio of user A (i.e., the sample audio included in the sample video of user A) are combined according to the time sequence of the sample audio of user A, and finally a digital human sample video can be generated.

[0244] It should be noted that the preset video generation model may specifically be a GAN in the embodiment of the present invention, and may of course be other types of preset video generation models, which is not limited in the embodiment of the present invention.

[0245] In addition, the above steps S301 to S305 can be repeatedly executed until the model training meets the termination conditions of the model training, such as the loss function converges, or the digital human sample video is highly similar to the sample video, which is not limited in this embodiment of the present invention.

[0246] Corresponding to the above method embodiment, the embodiment of the present invention also provides a digital human video generation device, such as Figure 6 As shown, the device may include: an audio acquisition module 610 , a parameter and contour extraction module 620 , a grid generation module 630 , and a video generation module 640 .

[0247] The audio acquisition module 610 is used to acquire the audio of the target person, extract the audio features corresponding to any audio frame in the audio, and acquire a first average face 3D mesh;

[0248] A parameter and contour extraction module 620 is used to determine a sample video of the target person, and extract a first face posture parameter and a first background contour of a target sample image in the sample video;

[0249] A mesh generation module 630 is used to process the audio feature, the first face posture parameter and the first average face 3D mesh using a preset face 3D model to generate a first face 3D mesh;

[0250] The video generation module 640 is used to process the first face 3D mesh and the first background contour line using a preset video generation model to generate a digital human video.

[0251] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, it includes a processor 71, a communication interface 72, a memory 73 and a communication bus 74, wherein the processor 71, the communication interface 72, and the memory 73 communicate with each other through the communication bus 74.

[0252] A memory 73, for storing computer programs;

[0253] The processor 71 is used to execute the program stored in the memory 73 to implement the following steps:

[0254] The audio of the target person is obtained, and for any audio frame in the audio, the audio features corresponding to the audio frame are extracted, and a first average face 3D mesh is obtained; a sample video of the target person is determined, and a first face posture parameter and a first background contour line of the target sample image in the sample video are extracted; using a preset face 3D model, the audio features, the first face posture parameter and the first average face 3D mesh are processed to generate a first face 3D mesh; using a preset video generation model, the first face 3D mesh and the first background contour line are processed to generate a digital human video.

[0255] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0256] The communication interface is used for communication between the above electronic device and other devices.

[0257] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0258] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0259] In another embodiment of the present invention, a storage medium is provided. The storage medium stores instructions. When the instructions are executed on a computer, the computer executes the digital human video generation method described in any one of the above embodiments.

[0260] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the digital human video generation method described in any one of the above embodiments.

[0261] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium, or transmitted from one storage medium to another storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0262] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0263] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0264] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for generating a digital human video, characterized in that: The method comprises: Acquire audio of a target person, extract audio features corresponding to any audio frame in the audio, and acquire a first average face 3D mesh; Determine a sample video of the target person, and extract a first face posture parameter and a first background contour line of a target sample image in the sample video; Using a preset 3D face model, the audio feature, the first face posture parameter, and the first average 3D face mesh are processed to generate a first 3D face mesh; Using a preset video generation model, the first face 3D mesh and the first background contour are processed to generate a digital human video; The step of obtaining a first average face 3D mesh includes: Inputting the target sample image into a preset 3D model, and obtaining first 3D face reconstruction information output by the preset 3D model; Extracting a first facial posture parameter and a first facial expression parameter from the first 3D face reconstruction information; Determining the first facial posture parameter and the first facial expression parameter as a first average facial 3D mesh; or, Obtain a first average face 3D mesh set, and obtain any first average face 3D mesh in the first average face 3D mesh set.

2. The method according to claim 1, characterized in that The using a preset 3D face model to process the audio feature, the first face posture parameter, and the first average 3D face mesh to generate a first 3D face mesh includes: For any audio frame in the audio, process the audio feature corresponding to the audio frame and the first average face 3D mesh to generate a first intermediate face 3D mesh; The first intermediate 3D face mesh and the first face posture parameter are input into a preset 3D face model, and a first 3D face mesh corresponding to the audio frame output by the preset 3D face model is obtained.

3. The method according to claim 2, characterized in that The processing of the audio feature corresponding to the audio frame and the first average face 3D mesh to generate a first intermediate face 3D mesh includes: Inputting the audio feature corresponding to the audio frame into a preset neural grid model, and obtaining a first sound hidden variable corresponding to the audio frame output by the preset neural grid model; Inputting the first average face 3D mesh into the preset neural mesh model, and obtaining a first image hidden variable output by the preset neural mesh model; The first sound hidden variable and the first image hidden variable corresponding to the audio frame are synthesized in a channel synthesis manner to obtain a first synthesized variable; The first synthesis variable is input into a preset decoder to obtain a first intermediate face 3D mesh corresponding to the audio frame output by the preset decoder.

4. The method according to claim 2, characterized in that: The using a preset video generation model to process the first face 3D mesh and the first background contour to generate a digital human video includes: Synthesize the first face 3D mesh corresponding to the audio frame with the first background contour line in the channel synthesis manner to obtain a first synthesized image corresponding to the audio frame; Inputting the first synthetic image corresponding to the audio frame into a preset video generation model, and obtaining a digital human image corresponding to the audio frame output by the preset video generation model; According to the time sequence of the audio, the digital human images corresponding to the audio frames are combined to generate a digital human video.

5. The method according to claim 1, characterized in that The extracting, for any audio frame in the audio, an audio feature corresponding to the audio frame includes: For any audio frame in the audio, using Fourier transform to extract Mel frequency cepstrum coefficients as audio features corresponding to the audio frame; or, For any audio frame in the audio, an audio feature corresponding to the audio frame is extracted using a preset speech recognition model.

6. The method according to claim 1, characterized in that Before executing the method, the method further includes: Acquire a sample video of a target person, wherein a sample image in the sample video corresponds one-to-one to a sample audio frame of a sample audio in the sample video; For any of the sample audio frames in the sample audio, extract the sample audio features corresponding to the sample audio frame, and obtain a second average face 3D mesh; and, Extracting a second facial posture parameter and a second background contour line in the sample image corresponding to the sample audio frame; Using a preset 3D face model, the sample audio feature, the second face posture parameter, and the second average 3D face mesh are processed to generate a second 3D face mesh; The second face 3D mesh and the second background contour are processed using a preset video generation model to generate a digital human sample video.

7. The method according to claim 6, characterized in that The obtaining of the second average face 3D mesh comprises: Inputting the sample image corresponding to the sample audio frame into a preset 3D model, and obtaining second 3D face reconstruction information output by the preset 3D model; Extracting second facial posture parameters and second facial expression parameters from the second 3D face reconstruction information; Determining the second facial posture parameter and the second facial expression parameter as a second average facial 3D mesh; or, Obtain a second average face 3D mesh set, and obtain any second average face 3D mesh in the second average face 3D mesh set.

8. The method according to claim 6, characterized in that The method of using a preset 3D face model to process the sample audio feature, the second face posture parameter, and the second average 3D face mesh to generate a second 3D face mesh includes: For any of the sample audio frames in the sample audio, processing the sample audio features corresponding to the sample audio frame and the second average face 3D mesh to generate a second intermediate face 3D mesh; The second intermediate 3D face mesh and the second face posture parameter are input into a preset 3D face model, and a second 3D face mesh corresponding to the sample audio frame output by the preset 3D face model is obtained.

9. The method according to claim 8, characterized in that The processing of the sample audio features corresponding to the sample audio frame and the second average face 3D mesh to generate a second intermediate face 3D mesh includes: Inputting the sample audio feature corresponding to the sample audio frame into a preset neural grid model, and obtaining a second sound hidden variable corresponding to the sample audio frame output by the preset neural grid model; Inputting the second average face 3D mesh into the preset neural mesh model, and obtaining a second image hidden variable output by the preset neural mesh model; The second sound hidden variable and the second image hidden variable corresponding to the sample audio frame are synthesized in a channel synthesis manner to obtain a second synthesized variable; The second synthesis variable is input into a preset decoder to obtain a second intermediate face 3D mesh corresponding to the sample audio frame output by the preset decoder.

10. The method according to claim 8, characterized in that The using of a preset video generation model to process the second face 3D mesh and the second background contour to generate a digital human sample video includes: Synthesize the second face 3D mesh corresponding to the sample audio frame with the second background contour line in the channel synthesis manner to obtain a second synthesized image corresponding to the sample audio frame; Inputting the second synthetic image corresponding to the sample audio frame into a preset video generation model, and obtaining a digital human sample image corresponding to the sample audio frame output by the preset video generation model; According to the time sequence of the sample audio, the digital human sample images corresponding to the sample audio frames are combined to generate a digital human sample video.

11. The method according to claim 6, characterized in that The extracting, for any of the sample audio frames in the sample audio, a sample audio feature corresponding to the sample audio frame comprises: For any of the sample audio frames in the sample audio, extract Mel frequency cepstrum coefficients using Fourier transform as sample audio features corresponding to the sample audio frame; or, For any of the sample audio frames in the sample audio, a preset speech recognition model is used to extract sample audio features corresponding to the sample audio frame.

12. A digital human video generation device, characterized in that: The device comprises: An audio acquisition module, used to acquire the audio of the target person, extract the audio features corresponding to any audio frame in the audio, and acquire a first average face 3D mesh; A parameter and contour extraction module, used to determine a sample video of the target person, and extract a first face posture parameter and a first background contour of a target sample image in the sample video; A mesh generation module, configured to process the audio feature, the first face posture parameter and the first average face 3D mesh using a preset face 3D model to generate a first face 3D mesh; A video generation module, used to process the first face 3D mesh and the first background contour line using a preset video generation model to generate a digital human video; Among them, the audio acquisition module is specifically used to: input the target sample image into a preset 3D model, and obtain the first 3D face reconstruction information output by the preset 3D model; extract the first face posture parameter and the first face expression parameter in the first 3D face reconstruction information; determine the first face posture parameter and the first face expression parameter as the first average face 3D grid; or, obtain a first average face 3D grid set, and obtain any first average face 3D grid in the first average face 3D grid set.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 11 when executing a program stored in a memory.

14. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images