Digital human video generation method, device, electronic device and storage medium

Abstract

The disclosed embodiments disclose a method, device, electronic device and storage medium for generating a digital human video. The method comprises: obtaining a target audio and a target face image; for an audio frame in the target audio, inputting an audio frame sequence corresponding to the audio frame and a target area image in the target face image into a pre-trained end-to-end model to generate a target image corresponding to the audio frame, wherein the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the target audio, the target area image is an area image other than the mouth area image in the target face image, and the target image corresponding to the audio frame is used to indicate that the person indicated by the target face image emits the audio indicated by the audio frame; based on the generated target image, a digital human video is generated. The disclosed embodiments can improve the efficiency of digital human generation.

Description

Technical Field

[0001] The present disclosure relates to the technical field of digital human video generation, and in particular to a digital human video generation method, device, electronic device and storage medium. Background Art

[0002] Digital human generation technology is becoming more mature. Existing solutions include digital human generation methods based on pix2pix, pix2pixHD, and video2videosynthesis. Specifically, a large number of digital human generation technologies have emerged, such as digital human generation methods based on pix2pix, pix2pixHD, Vid2Vid, few shot video2video, NERF, StyleGAN, etc.

[0003] However, in the existing solutions, if the generated facial key points are inaccurate and the generated sketch is poor, the final generated digital human image will be poor. Summary of the invention

[0004] In view of this, in order to solve some or all of the above technical problems, the embodiments of the present disclosure provide a method, device, electronic device and storage medium for generating a digital human video.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating a digital human video, the method comprising:

[0006] Obtain target audio and target face image;

[0007] For the audio frame in the target audio, the audio frame sequence corresponding to the audio frame and the target area image in the target facial image are input into a pre-trained end-to-end model to generate a target image corresponding to the audio frame, wherein the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the target audio, the target area image is an area image excluding the mouth area image in the target facial image, and the target image corresponding to the audio frame is used to indicate that the person indicated by the target facial image emits the audio indicated by the audio frame;

[0008] Based on the generated target image, a digital human video is generated.

[0009] Optionally, in the method of any embodiment of the present disclosure, the end-to-end model includes a first sub-model, a second sub-model and a third sub-model, the input data of the first sub-model is an audio frame sequence corresponding to the audio frame, the output data of the first sub-model is a first hidden vector, the input data of the second sub-model is a target area image in the target face image, the output data of the second sub-model is a second hidden vector, the input data of the third sub-model includes the first hidden vector and the second hidden vector, and the output data of the third sub-model includes the target image; and

[0010] The method of inputting the audio frame sequence corresponding to the audio frame and the target area image in the target face image into a pre-trained end-to-end model to generate a target image corresponding to the audio frame includes:

[0011] Inputting the audio frame sequence corresponding to the audio frame into the first sub-model to obtain a first hidden vector;

[0012] Inputting the target area image in the target face image into the second sub-model to obtain a second hidden vector;

[0013] Merging the first hidden vector and the second hidden vector to obtain a merged vector;

[0014] The merged vector is input into the third sub-model to obtain a target image corresponding to the audio frame.

[0015] Optionally, in the method of any embodiment of the present disclosure, the end-to-end model is trained in the following manner:

[0016] Get video data;

[0017] Extracting audio frames and facial images corresponding to the audio frames from the video data, using the audio frame sequence corresponding to the extracted audio frames as sample audio, and using the extracted facial images as sample facial images;

[0018] A machine learning algorithm is used to take the sample audio as input data of the generator in the generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio. If the discriminator in the generative adversarial network determines that the target image generated by the generator meets the preset training end conditions, the current generator is used as an end-to-end model.

[0019] Optionally, in the method of any embodiment of the present disclosure, the sample audio is used as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio, and if the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, the current generator is used as an end-to-end model, including:

[0020] Obtaining an initial generative adversarial network, wherein the initial generative adversarial network includes a first sub-model, a second sub-model, a third sub-model, and a fourth sub-model, the input data of the fourth sub-model is the first hidden vector, and the output data of the fourth sub-model is the mouth key point;

[0021] Perform the first training step as follows:

[0022] Inputting the sample audio into a first sub-model included in the initial generative adversarial network to obtain a first hidden vector corresponding to the sample audio;

[0023] Inputting the first hidden vector corresponding to the sample audio into the fourth sub-model to obtain the predicted mouth key points corresponding to the sample audio;

[0024] Calculating a first function value of a first preset loss function based on the predicted mouth key points corresponding to the sample audio and the mouth key points extracted from the sample face image corresponding to the sample audio;

[0025] If the calculated first function value is less than or equal to the first preset threshold, the model parameters of the first sub-model included in the current initial generative adversarial network are determined as the model parameters of the first sub-model included in the trained end-to-end model.

[0026] Optionally, in the method of any embodiment of the present disclosure, the sample audio is used as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio, and if the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, the current generator is used as an end-to-end model, further comprising:

[0027] If the calculated first function value is greater than the above-mentioned first preset threshold, the model parameters of the first sub-model and the model parameters of the fourth sub-model included in the current initial generative adversarial network are updated, and the above-mentioned first training step is continued based on the initial generative adversarial network after the model parameters are updated.

[0028] Optionally, in the method of any embodiment of the present disclosure, the sample audio is used as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio, and if the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, the current generator is used as an end-to-end model, further comprising:

[0029] Perform the second training step as follows:

[0030] Inputting the sample audio into a first sub-model included in the initial generative adversarial network to obtain a first hidden vector corresponding to the sample audio;

[0031] Inputting the target area image in the sample face image corresponding to the sample audio into the second sub-model included in the initial generative adversarial network to obtain a second hidden vector corresponding to the sample audio;

[0032] Merging a first hidden vector corresponding to the sample audio and a second hidden vector corresponding to the sample audio to obtain a merged vector corresponding to the sample audio;

[0033] Inputting the merged vector corresponding to the sample audio into the third sub-model included in the initial generative adversarial network to obtain a predicted target image corresponding to the sample audio;

[0034] Calculating a second function value of a second preset loss function based on a predicted target image corresponding to the sample audio and a target image extracted from a sample face image corresponding to the sample audio;

[0035] If the calculated second function value is less than or equal to a preset threshold, the model parameters of the second sub-model included in the current initial generative adversarial network are determined as the model parameters of the second sub-model included in the trained end-to-end model, and the model parameters of the third sub-model included in the current initial generative adversarial network are determined as the model parameters of the third sub-model included in the trained end-to-end model.

[0036] Optionally, in the method of any embodiment of the present disclosure, the sample audio is used as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio, and if the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, the current generator is used as an end-to-end model, further comprising:

[0037] If the calculated second function value is greater than the above-mentioned second preset threshold, the model parameters of the second sub-model and the model parameters of the third sub-model included in the current initial generative adversarial network are updated, and the above-mentioned second training step is continued based on the initial generative adversarial network after the model parameters are updated.

[0038] Optionally, in the method of any embodiment of the present disclosure, the preset training end condition includes at least one of the following:

[0039] A function value of a preset loss function calculated based on an audio frame sequence corresponding to the audio frame is less than or equal to a first preset value;

[0040] A function value of the preset loss function calculated based on the audio frame sequence corresponding to the non-audio frame is greater than or equal to a second preset value.

[0041] Optionally, in the method of any embodiment of the present disclosure, the second sub-model is an encoder, and the third sub-model is a decoder corresponding to the encoder.

[0042] Optionally, in the method of any embodiment of the present disclosure, the audio frame sequence corresponding to the audio frame includes the audio frame and a preset number of consecutive audio frames preceding the audio frame in the target audio.

[0043] In a second aspect, an embodiment of the present disclosure provides a digital human video generation device, the device comprising:

[0044] An acquisition unit, configured to acquire target audio and target face image;

[0045] An input unit is configured to input, for an audio frame in the target audio, an audio frame sequence corresponding to the audio frame and a target area image in the target facial image into a pre-trained end-to-end model to generate a target image corresponding to the audio frame, wherein the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the target audio, the target area image is an area image excluding a mouth area image in the target facial image, and the target image corresponding to the audio frame is used to indicate that the person indicated by the target facial image emits the audio indicated by the audio frame;

[0046] The generating unit is configured to generate a digital human video based on the generated target image.

[0047] Optionally, in the device of any embodiment of the present disclosure, the end-to-end model includes a first sub-model, a second sub-model and a third sub-model, the input data of the first sub-model is an audio frame sequence corresponding to the audio frame, the output data of the first sub-model is a first hidden vector, the input data of the second sub-model is a target area image in the target face image, the output data of the second sub-model is a second hidden vector, the input data of the third sub-model includes the first hidden vector and the second hidden vector, and the output data of the third sub-model includes the target image; and

[0048] The above generation unit is further configured to:

[0049] Inputting the audio frame sequence corresponding to the audio frame into the first sub-model to obtain a first hidden vector;

[0050] Inputting the target area image in the target face image into the second sub-model to obtain a second hidden vector;

[0051] Merging the first hidden vector and the second hidden vector to obtain a merged vector;

[0052] The merged vector is input into the third sub-model to obtain a target image corresponding to the audio frame.

[0053] Optionally, in the device of any embodiment of the present disclosure, the end-to-end model is trained in the following manner:

[0054] Get video data;

[0055] Extracting audio frames and facial images corresponding to the audio frames from the video data, using the audio frame sequence corresponding to the extracted audio frames as sample audio, and using the extracted facial images as sample facial images;

[0056] A machine learning algorithm is used to take the sample audio as input data of the generator in the generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio. If the discriminator in the generative adversarial network determines that the target image generated by the generator meets the preset training end conditions, the current generator is used as an end-to-end model.

[0057] Optionally, in the device of any embodiment of the present disclosure, the sample audio is used as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio, and if the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, the current generator is used as an end-to-end model, including:

[0058] Obtaining an initial generative adversarial network, wherein the initial generative adversarial network includes a first sub-model, a second sub-model, a third sub-model, and a fourth sub-model, the input data of the fourth sub-model is the first hidden vector, and the output data of the fourth sub-model is the mouth key point;

[0059] Perform the first training step as follows:

[0060] Inputting the sample audio into a first sub-model included in the initial generative adversarial network to obtain a first hidden vector corresponding to the sample audio;

[0061] Inputting the first hidden vector corresponding to the sample audio into the fourth sub-model to obtain the predicted mouth key points corresponding to the sample audio;

[0062] Calculating a first function value of a first preset loss function based on the predicted mouth key points corresponding to the sample audio and the mouth key points extracted from the sample face image corresponding to the sample audio;

[0063] If the calculated first function value is less than or equal to the first preset threshold, the model parameters of the first sub-model included in the current initial generative adversarial network are determined as the model parameters of the first sub-model included in the trained end-to-end model.

[0064] Optionally, in the apparatus of any embodiment of the present disclosure, the sample audio is used as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio, and if the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, the current generator is used as an end-to-end model, further comprising:

[0065] If the calculated first function value is greater than the above-mentioned first preset threshold, the model parameters of the first sub-model and the model parameters of the fourth sub-model included in the current initial generative adversarial network are updated, and the above-mentioned first training step is continued based on the initial generative adversarial network after the model parameters are updated.

[0066] Optionally, in the apparatus of any embodiment of the present disclosure, the sample audio is used as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio, and if the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, the current generator is used as an end-to-end model, further comprising:

[0067] Perform the second training step as follows:

[0068] Inputting the sample audio into a first sub-model included in the initial generative adversarial network to obtain a first hidden vector corresponding to the sample audio;

[0069] Inputting the target area image in the sample face image corresponding to the sample audio into the second sub-model included in the initial generative adversarial network to obtain a second hidden vector corresponding to the sample audio;

[0070] Merging a first hidden vector corresponding to the sample audio and a second hidden vector corresponding to the sample audio to obtain a merged vector corresponding to the sample audio;

[0071] Inputting the merged vector corresponding to the sample audio into the third sub-model included in the initial generative adversarial network to obtain a predicted target image corresponding to the sample audio;

[0072] Calculating a second function value of a second preset loss function based on a predicted target image corresponding to the sample audio and a target image extracted from a sample face image corresponding to the sample audio;

[0073] If the calculated second function value is less than or equal to a preset threshold, the model parameters of the second sub-model included in the current initial generative adversarial network are determined as the model parameters of the second sub-model included in the trained end-to-end model, and the model parameters of the third sub-model included in the current initial generative adversarial network are determined as the model parameters of the third sub-model included in the trained end-to-end model.

[0074] Optionally, in the apparatus of any embodiment of the present disclosure, the sample audio is used as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio, and if the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, the current generator is used as an end-to-end model, further comprising:

[0075] If the calculated second function value is greater than the above-mentioned second preset threshold, the model parameters of the second sub-model and the model parameters of the third sub-model included in the current initial generative adversarial network are updated, and the above-mentioned second training step is continued based on the initial generative adversarial network after the model parameters are updated.

[0076] Optionally, in the device of any embodiment of the present disclosure, the preset training end condition includes at least one of the following:

[0077] A function value of a preset loss function calculated based on an audio frame sequence corresponding to the audio frame is less than or equal to a first preset value;

[0078] A function value of the preset loss function calculated based on the audio frame sequence corresponding to the non-audio frame is greater than or equal to a second preset value.

[0079] Optionally, in the device of any embodiment of the present disclosure, the second sub-model is an encoder, and the third sub-model is a decoder corresponding to the encoder.

[0080] Optionally, in the device of any embodiment of the present disclosure, the audio frame sequence corresponding to the audio frame includes the audio frame and a preset number of consecutive audio frames before the audio frame in the above-mentioned target audio.

[0081] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0082] Memory for storing computer programs;

[0083] The processor is used to execute the computer program stored in the above-mentioned memory, and when the above-mentioned computer program is executed, the method of any embodiment of the digital human video generation method of the above-mentioned first aspect of the present disclosure is implemented.

[0084] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable medium, and when the computer program is executed by a processor, the method of any embodiment of the digital human video generation method of the first aspect is implemented.

[0085] In a fifth aspect, an embodiment of the present disclosure provides a computer program, which includes a computer-readable code. When the computer-readable code is executed on a device, the processor in the device executes instructions for implementing the steps in the method of any embodiment of the digital human video generation method in the first aspect mentioned above.

[0086] Based on the digital human video generation method provided by the above embodiment of the present disclosure, by acquiring the target audio and the target face image, then, for the audio frame in the above target audio, the audio frame sequence corresponding to the audio frame and the target area image in the above target face image are input into the pre-trained end-to-end model to generate the target image corresponding to the audio frame, wherein the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the above target audio, the above target area image is an area image other than the mouth area image in the above target face image, and the target image corresponding to the audio frame is used to indicate that the person indicated by the above target face image emits the audio indicated by the audio frame, and finally, based on the generated target image, the digital human video is generated. Thus, the end-to-end model is adopted to directly obtain the target image for generating the digital human video, thereby improving the efficiency of generating the digital human video by increasing the speed of generating the target image.

[0087] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0089] Figure 1 is an exemplary system architecture diagram of a digital human video generation method or a digital human video generation device provided by an embodiment of the present disclosure;

[0090] Figure 2 is a flow chart of a method for generating a digital human video provided by an embodiment of the present disclosure;

[0091] Figure 3 It is for Figure 2 A schematic diagram of an application scenario of an embodiment of the present invention;

[0092] Figure 4A is a flow chart of another method for generating a digital human video provided by an embodiment of the present disclosure;

[0093] Figure 4B It is a structural schematic diagram of a mouth area image generation model in a digital human video generation method provided by an embodiment of the present disclosure;

[0094] Figure 5 It is a structural schematic diagram of a digital human video generating device provided by an embodiment of the present disclosure;

[0095] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0096] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0097] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish objects such as different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate the logical order between them.

[0098] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0099] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0100] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0101] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0102] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0103] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the above-mentioned technologies, methods, and equipment should be considered as part of the specification.

[0104] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0105] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0106] Figure 1 It is an exemplary system architecture diagram of a digital human video generation method or a digital human video generation device provided in an embodiment of the present disclosure.

[0107] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0108] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send data (such as target audio and target face image), etc. The terminal devices 101, 102, 103 can be installed with various client applications, such as audio and video processing software.

[0109] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or they can be implemented as a single software or software module. No specific limitation is made here.

[0110] The server 105 may be a server that provides various services, such as a background server that processes data sent by the terminal devices 101, 102, and 103. As an example, the server 105 may be a cloud server.

[0111] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0112] It should also be noted that the digital human video generation method provided in the embodiments of the present disclosure can be executed by a server, or by a terminal device, or by a server and a terminal device in cooperation with each other. Accordingly, the various parts (such as various units, sub-units, modules, and sub-modules) included in the digital human video generation device can be all set in the server, or all set in the terminal device, or can be set in the server and the terminal device respectively.

[0113] It should be understood that Figure 1 The number of terminal devices, networks and servers in the system is only illustrative. According to the implementation requirements, there may be any number of terminal devices, networks and servers. When the electronic device on which the digital human video generation method is running does not need to transmit data with other electronic devices, the system architecture may only include the electronic device (such as a server or terminal device) on which the digital human video generation method is running.

[0114] Figure 2 A process 200 of a digital human video generation method provided by an embodiment of the present disclosure is shown. The digital human video generation method comprises the following steps:

[0115] Step 201, obtaining target audio and target face image.

[0116] In this embodiment, the execution subject of the digital human video generation method (for example Figure 1 The server or terminal device shown) can obtain the target audio and target face image from other electronic devices or locally.

[0117] The target audio may be any audio. The target audio may be used to generate a sound indicated by the target audio in the digital human video generated in the subsequent steps. For example, the target audio may be a voice audio, or may be an audio generated by a machine by converting text.

[0118] The target face image may be any face image. For example, the target face image may be a captured image containing a face, or may be a frame of a face image extracted from a video.

[0119] In some cases, there may be no association between the target audio and the target facial image. For example, the target audio may be the audio emitted by a first person, and the target facial image may be the facial image of a second person, where the second person may be a person different from the first person; or the target audio may be the audio emitted by a first person at a first moment, and the target facial image may be the facial image of the first person at a second moment, where the second moment may be any moment different from the first moment.

[0120] Step 202, for the audio frame in the above-mentioned target audio, input the audio frame sequence corresponding to the audio frame and the target area image in the above-mentioned target face image into a pre-trained end-to-end model to generate a target image corresponding to the audio frame.

[0121] In this embodiment, the above-mentioned execution entity can input the audio frame in the above-mentioned target audio, the audio frame sequence corresponding to the audio frame and the target area image in the above-mentioned target face image into a pre-trained end-to-end model to generate a target image corresponding to the audio frame.

[0122] Among them, the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the above-mentioned target audio. The above-mentioned target area image is the area image excluding the mouth area image in the above-mentioned target face image. The target image corresponding to the audio frame is used to indicate that the person indicated by the above-mentioned target face image emits the audio indicated by the audio frame. The above-mentioned end-to-end model can characterize the correspondence between the audio frame sequence corresponding to the audio frame, the target area image in the target face image, and the target image corresponding to the audio frame.

[0123] Here, the audio frame sequence corresponding to the audio frame may be a sequence consisting of a preset number of audio frames including the audio frame in the target audio. For example, the audio frame sequence may include the audio frame and the first 4 audio frames of the audio frame, or the audio frame sequence may include the audio frame and the first 2 audio frames of the audio frame and the last 2 audio frames of the audio frame.

[0124] Optionally, the audio frame sequence corresponding to the audio frame includes the audio frame and a preset number of consecutive audio frames before the audio frame in the above-mentioned target audio.

[0125] In some optional implementations of this embodiment, the end-to-end model includes a first sub-model, a second sub-model and a third sub-model. The input data of the first sub-model is an audio frame sequence corresponding to an audio frame. The output data of the first sub-model is a first hidden vector. The input data of the second sub-model is a target area image in the target face image. The output data of the second sub-model is a second hidden vector. The input data of the third sub-model includes the first hidden vector and the second hidden vector. The output data of the third sub-model includes a target image.

[0126] On this basis, the execution subject may perform step 202 in the following manner, so as to input the audio frame sequence corresponding to the audio frame and the target area image in the target face image into a pre-trained end-to-end model to generate a target image corresponding to the audio frame:

[0127] In the first step, the audio frame sequence corresponding to the audio frame is input into the first sub-model to obtain the first hidden vector.

[0128] The first sub-model may include a model structure such as CNN (Convolutional Neural Networks), LSTM (Long Short-Term Memory Networks), etc. As an example, the first sub-model may include 2 CNN layers and 2 LSTM layers. The first hidden vector may be a sound coding vector, that is, a vector output by the intermediate layer.

[0129] In the second step, the target area image in the target face image is input into the second sub-model to obtain a second hidden vector.

[0130] The second sub-model may include a model structure such as CNN, LSTM, etc. As an example, the second sub-model may include 4 CNN layers. The second hidden vector may be a vector of a target area image (e.g., a target area image in a target face image, or a target area image in a sample face image) (e.g., a vector of a latent space output encoded by a sketch encoder).

[0131] The third step is to merge the first hidden vector and the second hidden vector to obtain a merged vector.

[0132] The fourth step is to input the merged vector into the third sub-model to obtain a target image corresponding to the audio frame.

[0133] The second sub-model may include a model structure such as CNN, LSTM, etc. As an example, the third sub-model may include 4 CNN layers. The third sub-model may characterize the correspondence between the merged vector and the target image.

[0134] It can be understood that in the above optional implementation, the target image corresponding to the audio frame is generated by the first sub-model, the second sub-model and the third sub-model included in the end-to-end model, so that the generation effect of the digital human video can be improved by improving the accuracy of the generated target image. In addition, in some cases, in the above optional implementation, during the use of the end-to-end model, since key point extraction, denormalization processing and other operations are not required, the accuracy of the digital human video generation can be improved.

[0135] In some cases of the above optional implementations, the above end-to-end model is trained in the following manner:

[0136] Step 1: Get video data.

[0137] The video data may be any video data containing voice and facial images. In the video data, each video frame contains an audio frame and a facial image, that is, each audio frame has a corresponding facial image. For example, in the video data within one second, if the video within one second contains 5 frames, that is, it contains 5 audio frames and 5 facial images, and the audio frames correspond to the facial images one by one.

[0138] Step 2: extract audio frames and face images corresponding to the audio frames from the above video data, use the audio frame sequence corresponding to the extracted audio frames as sample audio, and use the extracted face images as sample face images.

[0139] Step three, using a machine learning algorithm, taking the sample audio as the input data of the generator in the generative adversarial network, and obtaining a target image generated by the above generator corresponding to the sample audio. If the discriminator in the above generative adversarial network determines that the target image generated by the above generator meets the preset training end conditions, the current generator is used as an end-to-end model.

[0140] Among them, the above-mentioned preset training end conditions may include at least one of the following: the calculated loss function value is less than or equal to the preset threshold, and the probability that the mouth area image generated by the above-mentioned generator is the mouth area image of the sample face image corresponding to the sample audio is 50%.

[0141] It can be understood that in the above situation, an end-to-end model is obtained based on a generative adversarial network, which can improve the generation effect of digital human videos by improving the accuracy of the target image generated by the generator.

[0142] In some cases, the above-mentioned preset training end condition also includes at least one of the following:

[0143] The first item is that a function value of a preset loss function calculated based on an audio frame sequence corresponding to the audio frame is less than or equal to a first preset value.

[0144] The audio frame sequence corresponding to the audio frame may be a sequence of a preset number of audio frames including the audio frame in the target audio. For example, the audio frame sequence may include the audio frame and the first four audio frames of the audio frame.

[0145] The second item is that the function value of the preset loss function calculated based on the audio frame sequence corresponding to the non-audio frame is greater than or equal to a second preset value.

[0146] The audio frame sequence corresponding to the non-audio frame (hereinafter referred to as the target frame) may be a sequence consisting of other audio frames except the audio frame sequence corresponding to the audio frame. For example, the audio frame sequence corresponding to the audio frame may be a sequence consisting of a preset number of audio frames randomly selected from the video data or the target video. The audio frame sequence corresponding to the non-audio frame may include the target frame or may not include the target frame.

[0147] In some cases, the number of audio frames included in the audio frame sequence corresponding to the non-audio frames and the number of audio frames included in the audio frame sequence corresponding to the audio frames may be equal.

[0148] It can be understood that in the above case, the audio frame sequence corresponding to the audio frame (for example, the current frame and the previous 4 frames), and the target image in the sample face image, are input into the discriminator, and the smaller the loss, the better. Specifically, the 26 key points generated by the current frame and the previous 4 frames of audio reasoning and the 26 key points of the real face mouth of the current frame are input into the discriminator to obtain the function value of the preset loss function. The smaller the function value, the better, which indicates that the adversarially generated mouth is more realistic, that is, the effect of the digital human video is better. The audio frame sequence corresponding to the non-audio frame is input into the discriminator, and the larger the function value of the preset loss function, the better. Specifically, the 5 frames of audio that are not corresponding to the current frame (for example, the 26 key points generated by the other 5 frames of audio reasoning and the 26 key points of the real face mouth of the current frame are input into the discriminator to obtain the function value of the preset loss function. The larger the function value, the better, which indicates that the adversarially generated mouth of the adversarial generator is more realistic, that is, the generation effect of the digital human video is better.

[0149] Optionally, the second sub-model is an encoder, and the third sub-model is a decoder corresponding to the encoder.

[0150] In some application scenarios of the above situation, the sample audio is used as input data of the generator in the generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio. If the discriminator in the generative adversarial network determines that the target image generated by the generator meets the preset training end condition, the current generator is used as an end-to-end model, including:

[0151] First, an initial generative adversarial network is obtained, wherein the initial generative adversarial network includes a first sub-model, a second sub-model, a third sub-model and a fourth sub-model, the input data of the fourth sub-model is the first hidden vector, and the output data of the fourth sub-model is the mouth key point.

[0152] Afterwards, the following first training step (including step 1 to step 4) is performed:

[0153] Step 1: Input the sample audio into the first sub-model included in the initial generative adversarial network to obtain a first hidden vector corresponding to the sample audio.

[0154] Step 2: Input the first hidden vector corresponding to the sample audio into the fourth sub-model to obtain the predicted mouth key points corresponding to the sample audio.

[0155] Step three, calculating a first function value of a first preset loss function based on the predicted mouth key points corresponding to the sample audio and the mouth key points extracted from the sample face image corresponding to the sample audio.

[0156] Step four: If the calculated first function value is less than or equal to the first preset threshold, the model parameters of the first sub-model included in the current initial generative adversarial network are determined as the model parameters of the first sub-model included in the trained end-to-end model.

[0157] Optionally, if the calculated first function value is greater than the above-mentioned first preset threshold, the model parameters of the first sub-model and the model parameters of the fourth sub-model included in the current initial generative adversarial network are updated, and the above-mentioned first training step is continued to be executed based on the initial generative adversarial network after the model parameters are updated.

[0158] It can be understood that in the above optional implementation method, the size of the first function value is used to determine whether the model parameters of the first sub-model and the model parameters of the fourth sub-model in the generative adversarial network can be used for reasoning, and the generator in the trained generative adversarial network is used to generate digital human videos, which improves the generation effect of digital human videos. When using the generator, there is no need to use the second sub-model to obtain key points, which can improve the generation efficiency of digital human videos.

[0159] Optionally, the above-mentioned step of training to obtain an end-to-end model may also include executing the following second training step (including the first step to the sixth step).

[0160] The first step is to input the sample audio into the first sub-model included in the initial generative adversarial network to obtain a first hidden vector corresponding to the sample audio.

[0161] The second step is to input the target area image in the sample face image corresponding to the sample audio into the second sub-model included in the initial generative adversarial network to obtain a second hidden vector corresponding to the sample audio.

[0162] In the third step, the first hidden vector corresponding to the sample audio and the second hidden vector corresponding to the sample audio are merged to obtain a merged vector corresponding to the sample audio.

[0163] The fourth step is to input the merged vector corresponding to the sample audio into the third sub-model included in the initial generative adversarial network to obtain a predicted target image corresponding to the sample audio.

[0164] The fifth step is to calculate a second function value of a second preset loss function based on the predicted target image corresponding to the sample audio and the target image extracted from the sample face image corresponding to the sample audio.

[0165] The sixth step, if the calculated second function value is less than or equal to a preset threshold, the model parameters of the second sub-model included in the current initial generative adversarial network are determined as the model parameters of the second sub-model included in the trained end-to-end model, and the model parameters of the third sub-model included in the current initial generative adversarial network are determined as the model parameters of the third sub-model included in the trained end-to-end model.

[0166] Optionally, if the calculated second function value is greater than the above-mentioned second preset threshold, the model parameters of the second sub-model and the model parameters of the third sub-model included in the current initial generative adversarial network are updated, and the above-mentioned second training step is continued to be executed based on the initial generative adversarial network after the model parameters are updated.

[0167] It can be understood that here, after fixing the model parameters of the first sub-model and the model parameters of the fourth sub-model, the size of the second function value is used to judge whether the model parameters of the third sub-model can be used for reasoning, and the generator in the trained generative adversarial network is used to generate digital human videos, which improves the generation effect of digital human videos. When using the generator, there is no need to use the second sub-model to obtain key points, which can further improve the generation efficiency of digital human videos.

[0168] Step 203: Generate a digital human video based on the generated target image.

[0169] In this embodiment, the execution entity may generate a digital human video based on each generated target image.

[0170] Continue to see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an application scenario of the method for generating a digital human video according to this embodiment. Figure 3 In the example, the server 310 (i.e., the execution subject) first obtains the target audio 301 and the target face image 304. The server 310 inputs the audio frame sequence 303 corresponding to the audio frame 302 and the target region image 305 in the target face image 304 into the pre-trained end-to-end model 306 for the audio frame 302 in the target audio 301, and generates a target image 307 corresponding to the audio frame 302, wherein the audio frame sequence 303 corresponding to the audio frame 302 is a sequence of continuous audio frames containing the audio frame 302 in the target audio 301, the target region image 305 is a region image other than the mouth region image in the target face image 304, and the target image 307 corresponding to the audio frame 302 is used to indicate that the person indicated by the target face image emits the audio indicated by the audio frame. The server 310 generates a digital human video 308 based on the generated target image 307.

[0171] The method provided by the above embodiment of the present disclosure obtains target audio and target face image, and then, for the audio frame in the above target audio, inputs the audio frame sequence corresponding to the audio frame and the target area image in the above target face image into the pre-trained end-to-end model to generate a target image corresponding to the audio frame, wherein the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the above target audio, the above target area image is an area image other than the mouth area image in the above target face image, and the target image corresponding to the audio frame is used to indicate that the person indicated by the above target face image emits the audio indicated by the audio frame, and finally, based on the generated target image, a digital human video is generated. Thus, the end-to-end model is used to directly obtain the target image for generating the digital human video, thereby improving the efficiency of generating the digital human video by increasing the speed of generating the target image.

[0172] Further references Figure 4A , which shows a process 400 of another embodiment of a method for generating a digital human video. The process of the method for generating a digital human video includes the following steps:

[0173] Step 401, obtaining target audio and target face image.

[0174] Step 402: for an audio frame in the target audio, input the audio frame sequence corresponding to the audio frame into the first sub-model to obtain a first hidden vector, input the target area image in the target face image into the second sub-model to obtain a second hidden vector, merge the first hidden vector and the second hidden vector to obtain a merged vector, input the merged vector into the third sub-model to obtain a target image corresponding to the audio frame.

[0175] Among them, the input data of the above-mentioned first sub-model is the audio frame sequence corresponding to the audio frame, the output data of the above-mentioned first sub-model is the first hidden vector, the input data of the above-mentioned second sub-model is the target area image in the above-mentioned target face image, the output data of the above-mentioned second sub-model is the second hidden vector, the input data of the above-mentioned third sub-model includes the above-mentioned first hidden vector and the above-mentioned second hidden vector, and the output data of the above-mentioned third sub-model includes the target image.

[0176] Step 403: Generate a digital human video based on the generated target image.

[0177] As an example, the method for generating a digital human video in this embodiment can be performed as follows:

[0178] First, let me introduce the format of the data:

[0179] In the digital human video generation method of this embodiment, the size of the face sketch is 512*512*1; the size of the target face image is 512*512*3; the size of the face sketch and the target face image combined is 512*512*4.

[0180] Please refer to the following Figure 4B Introduce the implementation process of the specific solution:

[0181] After obtaining the user audio (that is, the target audio mentioned above), the user audio is processed by the encoder (that is, the first sub-model mentioned above) to generate a sound coding vector LM1 (that is, the middle layer (hidden space) of the CNN or LSTM, the first hidden vector mentioned above), and then the sound coding vector LM is synthesized with the original image (that is, the target area image, that is, the merged vector) vector LM2 (that is, the latent space encoded by the sketch encoder) in a channel synthesis manner to obtain a channel synthesis vector LM3 (containing the features of moth (mouth) and face image (face image)), and then the channel synthesis vector LM3 is processed by the decoder (that is, the third sub-model) (that is, input into the GAN generation model for decoding) to obtain a digital human fake image (that is, the target image), and then the digital human video is output (a video contains multiple frames of images).

[0182] During the training phase, the following steps can be performed:

[0183] The training is divided into two phases:

[0184] In the first stage, the sound (that is, the sample audio mentioned above) passes through CNN and LSTM, collectively referred to as the model LMEncoder (that is, the first sub-model mentioned above), and passes through full connection (that is, the third sub-model mentioned above) to obtain 26 inferred key points (that is, the above-mentioned mouth key points, for example, which may include 20 key points of the mouth and 6 key points of the chin). The first function value of the first preset loss function is calculated for the 26 inferred key points and the real key points (that is, the mouth key points extracted from the sample face image corresponding to the sample audio), and LMEncoder is trained.

[0185] In the second stage, when the first function value of the first preset loss function of the 26 key points is stable (for example, the calculated first function value is less than or equal to the first preset threshold), the model parameters of LMEncoder are fixed, and the encoder and decoder LipGAN training begins. The specific process is as follows:

[0186] First, prepare video data, which includes audio (ie, sample audio) and pictures (ie, sample face images corresponding to the sample audio).

[0187] Afterwards, the data is processed at a frame rate of 25 frames per second, features are extracted from the audio, and facial key points and corresponding canny lines are extracted from the image; that is, for each video frame, audio features are extracted from the video audio (sample audio), and 68 facial key points are extracted from the video image (that is, the sample face image corresponding to the sample audio). Among them, the method of extracting audio features can use Fourier transform to extract MFCC / use deepspeech model to extract audio features / use other algorithms (ASR model-sound recognition) to extract audio features.

[0188] Then, if Figure 4B As shown in the figure, after the sound passes through CNN and LSTM, a sound coding vector LM1 is generated, and then 26 mouth key points are generated through the fully connected layer (that is, 26 key points are generated by reasoning). Then, the 26 inferred key points and the 26 key points of the real face mouth are used to calculate the loss (that is, the first function value) to train LMEncoder.

[0189] Then, when the loss (the first function value) is stable, the LMEncoder parameters (i.e., the first sub-model) are fixed, that is, after the LMEncoder model is trained, the encoder and decoder LipGAN (i.e., the second sub-model and the third sub-model) are trained. Specifically, in the hidden layer, the hidden vector of the face image (removing the mouth part of the face) (i.e., the original image vector LM2) and the hidden vector of the real voice (i.e., the voice encoding vector LM1) are merged to become 1024*1*1 (i.e., the channel synthesis vector LM3, which contains the features of the moth (mouth) and the face image), and then the decoder outputs the generated image (i.e., the target image).

[0190] It should be noted that in the first and second stages, a mouth image of a frame can be trained using one frame of audio data or multiple frames of audio data. Specifically, when using N frames of audio data to train a mouth image (i.e., 26 key points of a human face), such as when training the key points of the human face and mouth of the t-th frame, the audio data corresponding to the t-th frame, t-1, t-2...t-(N-1) frames can be used to train the 26 key points of the human face and mouth of the t-th frame, thereby improving the generation effect of the human face and mouth image, and making the generation effect of the digital human image better. N can be greater than 1, and the larger N is, the better the mouth generation effect is. For example, the current audio frame and the first 4 frames of the current audio, and the picture of the current frame without the mouth part (i.e., the target area image) can be used to output the final target image.

[0191] In addition, a new discriminator (i.e., the fourth sub-model) loss function can be added to LipGAN to ensure the stability of image generation;

[0192] Among them, the current frame and the previous 4 frames (that is, the audio frame sequence corresponding to the audio frame), as well as the current true picture (that is, the target area image), are input into the discriminator, and the smaller the loss, the better. Specifically, the 26 key points generated by the current frame and the previous 4 frames of audio inference and the 26 key points of the real face mouth in the current frame are input into the discriminator to calculate the loss. The smaller the loss, the better, which means that the mouth generated by the adversarial method is more realistic, that is, the effective effect is better.

[0193] The other five frames (i.e., the audio frame sequence corresponding to the non-audio frame) and the current frame image (i.e., the target area image) are input into the discriminator. The larger the loss, the better. Specifically, the 5 frames of audio that are not corresponding to the current frame (i.e., the 26 key points generated by the other 5 frames of audio inference and the 26 key points of the real face mouth in the current frame) are input into the discriminator to calculate the loss. The larger the loss, the better, which means that the adversarial generated mouth of the adversarial generator is more realistic, that is, the better the effect.

[0194] In the inference (application) phase:

[0195] First, input the current frame and the previous 4 frames of audio (that is, the audio frame sequence corresponding to the audio frame) / or extract the audio features, input them into the model LMEncoder (that is, the first sub-model), and obtain the hidden vector LM1 (that is, the first hidden vector).

[0196] Then, the current image excluding the mouth area (ie, the target area image) is obtained, and the hidden vector IM2 (ie, the second hidden vector) is obtained through the encoder.

[0197] Finally, the hidden vectors LM1 and IM2 are merged to obtain the hidden vector (i.e. the channel synthesis vector LM3 (i.e. the merged vector), which contains the features of moth (mouth) and face image (face image)) and input into the decoder (i.e. the third sub-model), and the final image (i.e. the target image) is output. Then the digital human video is output.

[0198] Among them, the sound inference model can be used to extract the audio features of the audio. The format of the input sound can be wav format, and the frame rate can be 100, 50 or 25. Among them, wav is a lossless audio file format. For the sound features, it can be MFCC, or features extracted by models such as Deepspeech / ASR / wav2Vector. The sound inference model can be LSMT, BERT (Bidirectional Encoder Representations from Transformers), Transfromer (Transformer model), CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), etc. 3DMM is a 3D deformation statistical model of the face; it is a relatively basic three-dimensional statistical model, which was first proposed to solve the problem of restoring three-dimensional shapes from two-dimensional face graphics; the author collected 200 three-dimensional face head information, and used this set of data as the basis of PCA (principal component analysis) to obtain the principal component information that can represent the shape and texture of the face.

[0199] In this embodiment, the specific execution method of the above steps 401 to 403 can refer to the above Figure 2 The relevant description of the corresponding embodiment is not repeated here. In addition, in addition to the above-mentioned contents, the embodiments of the present disclosure may also include Figure 2 The same or similar features and effects as those in the corresponding embodiments will not be described in detail here.

[0200] The digital human video generation method in this embodiment uses an end-to-end approach to generate digital human videos, inputs audio, and directly generates a target image for generating digital human videos in combination with the latent space encoded by the sketch encoder, that is, it does not need to obtain key points and denormalize, and has high efficiency; further, it is not necessary to extract audio features, which further improves efficiency; and when audio features are extracted, the effect is better. In addition, the loss function of the new discriminator (i.e., the fourth sub-model) is used to maintain the stability of target image generation.

[0201] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a digital human video generation device, which corresponds to the method embodiment described above. In addition to the features described below, the device embodiment may also include the same or corresponding features as the method embodiment described above, and produce the same or corresponding effects as the method embodiment described above. The device can be specifically applied to various electronic devices.

[0202] like Figure 5 As shown, the digital human video generation device 500 of this embodiment. The above-mentioned device 500 includes: an acquisition unit 501, an input unit 502 and a generation unit 503. Among them, the acquisition unit 501 is configured to acquire the target audio and the target face image; the input unit 502 is configured to input the audio frame corresponding to the audio frame in the above-mentioned target audio and the target area image in the above-mentioned target face image into the pre-trained end-to-end model to generate a target image corresponding to the audio frame, wherein the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the above-mentioned target audio, the above-mentioned target area image is an area image in the above-mentioned target face image except the mouth area image, and the target image corresponding to the audio frame is used to indicate that the person indicated by the above-mentioned target face image emits the audio indicated by the audio frame; the generation unit 503 is configured to generate a digital human video based on the generated target image.

[0203] In this embodiment, the acquisition unit 501 of the digital human video generating device 500 can acquire the target audio and the target face image.

[0204] In this embodiment, the input unit 502 can input the audio frame in the above-mentioned target audio, the audio frame sequence corresponding to the audio frame and the target area image in the above-mentioned target facial image into a pre-trained end-to-end model, and generate a target image corresponding to the audio frame, wherein the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the above-mentioned target audio, the above-mentioned target area image is an area image excluding the mouth area image in the above-mentioned target facial image, and the target image corresponding to the audio frame is used to indicate that the person indicated by the above-mentioned target facial image emits the audio indicated by the audio frame.

[0205] In this embodiment, the generating unit 503 may generate a digital human video based on the generated target image.

[0206] In some optional implementations of this embodiment, the end-to-end model includes a first sub-model, a second sub-model and a third sub-model, the input data of the first sub-model is an audio frame sequence corresponding to the audio frame, the output data of the first sub-model is a first hidden vector, the input data of the second sub-model is a target area image in the target face image, the output data of the second sub-model is a second hidden vector, the input data of the third sub-model includes the first hidden vector and the second hidden vector, and the output data of the third sub-model includes the target image; and

[0207] The above generation unit is further configured to:

[0208] Inputting the audio frame sequence corresponding to the audio frame into the first sub-model to obtain a first hidden vector;

[0209] Inputting the target area image in the target face image into the second sub-model to obtain a second hidden vector;

[0210] Merging the first hidden vector and the second hidden vector to obtain a merged vector;

[0211] The merged vector is input into the third sub-model to obtain a target image corresponding to the audio frame.

[0212] In some optional implementations of this embodiment, the end-to-end model is trained in the following manner:

[0213] Get video data;

[0214] Extracting audio frames and facial images corresponding to the audio frames from the video data, using the audio frame sequence corresponding to the extracted audio frames as sample audio, and using the extracted facial images as sample facial images;

[0215] A machine learning algorithm is used to take the sample audio as input data of the generator in the generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio. If the discriminator in the generative adversarial network determines that the target image generated by the generator meets the preset training end conditions, the current generator is used as an end-to-end model.

[0216] In some optional implementations of this embodiment, the sample audio is used as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio. If the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, the current generator is used as an end-to-end model, including:

[0217] Obtaining an initial generative adversarial network, wherein the initial generative adversarial network includes a first sub-model, a second sub-model, a third sub-model, and a fourth sub-model, the input data of the fourth sub-model is the first hidden vector, and the output data of the fourth sub-model is the mouth key point;

[0218] Perform the first training step as follows:

[0219] Inputting the sample audio into a first sub-model included in the initial generative adversarial network to obtain a first hidden vector corresponding to the sample audio;

[0220] Inputting the first hidden vector corresponding to the sample audio into the fourth sub-model to obtain the predicted mouth key points corresponding to the sample audio;

[0221] Calculating a first function value of a first preset loss function based on the predicted mouth key points corresponding to the sample audio and the mouth key points extracted from the sample face image corresponding to the sample audio;

[0222] If the calculated first function value is less than or equal to the first preset threshold, the model parameters of the first sub-model included in the current initial generative adversarial network are determined as the model parameters of the first sub-model included in the trained end-to-end model.

[0223] In some optional implementations of this embodiment, the sample audio is used as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio. If the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, the current generator is used as an end-to-end model, further comprising:

[0224] If the calculated first function value is greater than the above-mentioned first preset threshold, the model parameters of the first sub-model and the model parameters of the fourth sub-model included in the current initial generative adversarial network are updated, and the above-mentioned first training step is continued based on the initial generative adversarial network after the model parameters are updated.

[0225] In some optional implementations of this embodiment, the sample audio is used as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio. If the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, the current generator is used as an end-to-end model, further comprising:

[0226] Perform the second training step as follows:

[0227] Inputting the sample audio into a first sub-model included in the initial generative adversarial network to obtain a first hidden vector corresponding to the sample audio;

[0228] Inputting the target area image in the sample face image corresponding to the sample audio into the second sub-model included in the initial generative adversarial network to obtain a second hidden vector corresponding to the sample audio;

[0229] Merging a first hidden vector corresponding to the sample audio and a second hidden vector corresponding to the sample audio to obtain a merged vector corresponding to the sample audio;

[0230] Inputting the merged vector corresponding to the sample audio into the third sub-model included in the initial generative adversarial network to obtain a predicted target image corresponding to the sample audio;

[0231] Calculating a second function value of a second preset loss function based on a predicted target image corresponding to the sample audio and a target image extracted from a sample face image corresponding to the sample audio;

[0232] If the calculated second function value is less than or equal to a preset threshold, the model parameters of the second sub-model included in the current initial generative adversarial network are determined as the model parameters of the second sub-model included in the trained end-to-end model, and the model parameters of the third sub-model included in the current initial generative adversarial network are determined as the model parameters of the third sub-model included in the trained end-to-end model.

[0233] In some optional implementations of this embodiment, the sample audio is used as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio. If the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, the current generator is used as an end-to-end model, further comprising:

[0234] If the calculated second function value is greater than the above-mentioned second preset threshold, the model parameters of the second sub-model and the model parameters of the third sub-model included in the current initial generative adversarial network are updated, and the above-mentioned second training step is continued based on the initial generative adversarial network after the model parameters are updated.

[0235] In some optional implementations of this embodiment, the preset training end condition includes at least one of the following:

[0236] A function value of a preset loss function calculated based on an audio frame sequence corresponding to the audio frame is less than or equal to a first preset value;

[0237] A function value of the preset loss function calculated based on the audio frame sequence corresponding to the non-audio frame is greater than or equal to a second preset value.

[0238] In some optional implementations of this embodiment, the second sub-model is an encoder, and the third sub-model is a decoder corresponding to the encoder.

[0239] In some optional implementations of this embodiment, the audio frame sequence corresponding to the audio frame includes the audio frame and a preset number of consecutive audio frames preceding the audio frame in the target audio.

[0240] In the device 500 provided by the above embodiment of the present disclosure, the acquisition unit 501 can acquire the target audio and the target face image, and then the input unit 502 can input the audio frame corresponding to the audio frame and the target area image in the target face image into the pre-trained end-to-end model for the audio frame in the target audio, and generate the target image corresponding to the audio frame, wherein the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the target audio, the target area image is an area image other than the mouth area image in the target face image, and the target image corresponding to the audio frame is used to indicate that the person indicated by the target face image emits the audio indicated by the audio frame, and finally, the generation unit 503 can generate the digital human video based on the generated target image. Thus, the end-to-end model is used to directly obtain the target image for generating the digital human video, thereby improving the efficiency of generating the digital human video by increasing the speed of generating the target image.

[0241] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure is provided. Figure 6 The electronic device 600 shown includes: at least one processor 601, a memory 602, and at least one network interface 604 and other user interfaces 603. The various components in the electronic device 600 are coupled together via a bus system 605. It is understood that the bus system 605 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 605 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 605 is not shown in FIG. Figure 6 Various buses are labeled as bus system 605 .

[0242] The user interface 603 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0243] It can be understood that the memory 602 in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 602 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0244] In some implementations, the memory 602 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 6021 and an application 6022 .

[0245] The operating system 6021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 6022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present disclosure can be included in the application 6022.

[0246] In the embodiment of the present disclosure, by calling the program or instruction stored in the memory 602, specifically, it can be the program or instruction stored in the application 6022, the processor 601 is used to execute the method steps provided by each method embodiment, for example, including: obtaining the target audio and the target facial image; for the audio frame in the above-mentioned target audio, inputting the audio frame sequence corresponding to the audio frame and the target area image in the above-mentioned target facial image into a pre-trained end-to-end model to generate a target image corresponding to the audio frame, wherein the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the above-mentioned target audio, the above-mentioned target area image is an area image of the above-mentioned target facial image excluding the mouth area image, and the target image corresponding to the audio frame is used to indicate that the person indicated by the above-mentioned target facial image emits the audio indicated by the audio frame; based on the generated target image, a digital human video is generated.

[0247] The method disclosed in the above embodiment of the present disclosure can be applied to the processor 601, or implemented by the processor 601. The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 601 or the instruction in the form of software. The above processor 601 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602 and completes the steps of the above method in combination with its hardware.

[0248] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the above-mentioned functions of the present application, or a combination thereof.

[0249] For software implementation, the technology described above in this article can be implemented by a unit that performs the functions described above in this article. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0250] The electronic device provided in this embodiment may be Figure 6 The electronic device shown in FIG. 1 may perform the following steps: Figure 2 All steps of the digital human video generation method in Figure 2 For details, please refer to the technical effect of the digital human video generation method shown in Figure 2 For the sake of brevity, the relevant description is not repeated here.

[0251] The embodiments of the present disclosure also provide a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0252] When one or more programs in the storage medium can be executed by one or more processors, the digital human video generation method executed on the electronic device side can be implemented.

[0253] The processor is used to execute the communication program stored in the memory to implement the following steps of the digital human video generation method performed on the electronic device side: obtaining target audio and target facial image; for the audio frame in the target audio, inputting the audio frame sequence corresponding to the audio frame and the target area image in the target facial image into a pre-trained end-to-end model to generate a target image corresponding to the audio frame, wherein the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the target audio, the target area image is an area image excluding the mouth area image in the target facial image, and the target image corresponding to the audio frame is used to indicate that the person indicated by the target facial image emits the audio indicated by the audio frame; based on the generated target image, a digital human video is generated.

[0254] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0255] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0256] The above-mentioned specific implementation methods further describe in detail the purpose, technical solutions and beneficial effects of the present disclosure. It should be understood that the above-mentioned are only specific implementation methods of the present disclosure and are not used to limit the protection scope of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A method for generating a digital human video, characterized in that: The method comprises: Acquire target audio and target face image; wherein the target face image is a frame of face image extracted from a video; For an audio frame in the target audio, an audio frame sequence corresponding to the audio frame and a target area image in the target facial image are input into a pre-trained end-to-end model to generate a target image corresponding to the audio frame, wherein the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the target audio, the target area image is an area image excluding a mouth area image in the target facial image, and the target image corresponding to the audio frame is used to indicate that the person indicated by the target facial image emits the audio indicated by the audio frame; generating a digital human video based on the generated target image; The end-to-end model is trained in the following way: Acquire video data; extract audio frames and face images corresponding to the audio frames from the video data, use the audio frame sequence corresponding to the extracted audio frames as sample audio, and use the extracted face images as sample face images; adopt a machine learning algorithm to acquire an initial generative adversarial network, wherein the initial generative adversarial network includes a first sub-model, a second sub-model, a third sub-model and a fourth sub-model, the input data of the fourth sub-model is a first hidden vector, and the output data of the fourth sub-model is a mouth key point; perform the following first training step: input the sample audio into the first sub-model included in the initial generative adversarial network to obtain the first hidden vector corresponding to the sample audio; input the first hidden vector corresponding to the sample audio into the fourth sub-model to obtain the predicted mouth key point corresponding to the sample audio; calculate the first function value of the first preset loss function based on the predicted mouth key point corresponding to the sample audio and the mouth key point extracted from the sample face image corresponding to the sample audio; if the calculated first function value is less than or equal to the first preset threshold, determine the model parameters of the first sub-model included in the current initial generative adversarial network as the model parameters of the first sub-model included in the trained end-to-end model.

2. The method according to claim 1, characterized in that The end-to-end model includes a first sub-model, a second sub-model and a third sub-model, the input data of the first sub-model is an audio frame sequence corresponding to an audio frame, the output data of the first sub-model is a first hidden vector, the input data of the second sub-model is a target area image in the target face image, the output data of the second sub-model is a second hidden vector, the input data of the third sub-model includes the first hidden vector and the second hidden vector, and the output data of the third sub-model includes a target image; as well as The step of inputting the audio frame sequence corresponding to the audio frame and the target area image in the target face image into a pre-trained end-to-end model to generate a target image corresponding to the audio frame includes: Inputting an audio frame sequence corresponding to the audio frame into the first sub-model to obtain a first hidden vector; Inputting the target area image in the target face image into the second sub-model to obtain a second hidden vector; Merging the first hidden vector and the second hidden vector to obtain a merged vector; The combined vector is input into the third sub-model to obtain a target image corresponding to the audio frame.

3. The method according to claim 2, characterized in that The method uses the sample audio as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio, and if the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, then uses the current generator as an end-to-end model, and further includes: If the calculated first function value is greater than the first preset threshold, the model parameters of the first sub-model and the model parameters of the fourth sub-model included in the current initial generative adversarial network are updated, and the first training step is continued based on the initial generative adversarial network after the model parameters are updated.

4. The method according to claim 2, characterized in that: The method uses the sample audio as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio, and if the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, then uses the current generator as an end-to-end model, and further includes: Perform the second training step as follows: Inputting the sample audio into a first sub-model included in the initial generative adversarial network to obtain a first hidden vector corresponding to the sample audio; Inputting the target area image in the sample face image corresponding to the sample audio into the second sub-model included in the initial generative adversarial network to obtain a second hidden vector corresponding to the sample audio; Merging a first hidden vector corresponding to the sample audio and a second hidden vector corresponding to the sample audio to obtain a merged vector corresponding to the sample audio; Inputting the merged vector corresponding to the sample audio into the third sub-model included in the initial generative adversarial network to obtain a predicted target image corresponding to the sample audio; Calculating a second function value of a second preset loss function based on a predicted target image corresponding to the sample audio and a target image extracted from a sample face image corresponding to the sample audio; If the calculated second function value is less than or equal to the second preset threshold, the model parameters of the second sub-model included in the current initial generative adversarial network are determined as the model parameters of the second sub-model included in the trained end-to-end model, and the model parameters of the third sub-model included in the current initial generative adversarial network are determined as the model parameters of the third sub-model included in the trained end-to-end model.

5. The method according to claim 4, characterized in that The method uses the sample audio as input data of a generator in a generative adversarial network to obtain a target image generated by the generator corresponding to the sample audio, and if the discriminator in the generative adversarial network determines that the target image generated by the generator meets a preset training end condition, then uses the current generator as an end-to-end model, and further includes: If the calculated second function value is greater than the second preset threshold, the model parameters of the second sub-model and the model parameters of the third sub-model included in the current initial generative adversarial network are updated, and the second training step is continued based on the initial generative adversarial network after the model parameters are updated.

6. The method according to any one of claims 3 to 5, characterized in that: The preset training end condition includes at least one of the following: A function value of a preset loss function calculated based on an audio frame sequence corresponding to the audio frame is less than or equal to a first preset value; A function value of the preset loss function calculated based on the audio frame sequence corresponding to the non-audio frame is greater than or equal to a second preset value.

7. The method according to any one of claims 2 to 5, characterized in that: The second sub-model is an encoder, and the third sub-model is a decoder corresponding to the encoder.

8. The method according to any one of claims 1 to 5, characterized in that: The audio frame sequence corresponding to the audio frame includes the audio frame and a preset number of consecutive audio frames before the audio frame in the target audio.

9. A digital human video generation device, characterized in that: The device comprises: An acquisition unit is configured to acquire a target audio and a target face image; wherein the target face image is a frame of face image extracted from a video; An input unit is configured to input, for an audio frame in the target audio, an audio frame sequence corresponding to the audio frame and a target area image in the target facial image into a pre-trained end-to-end model to generate a target image corresponding to the audio frame, wherein the audio frame sequence corresponding to the audio frame is a sequence of continuous audio frames containing the audio frame in the target audio, the target area image is an area image excluding a mouth area image in the target facial image, and the target image corresponding to the audio frame is used to indicate that the person indicated by the target facial image emits the audio indicated by the audio frame; A generating unit configured to generate a digital human video based on the generated target image; The end-to-end model is trained in the following way: Acquire video data; extract audio frames and face images corresponding to the audio frames from the video data, use the audio frame sequence corresponding to the extracted audio frames as sample audio, and use the extracted face images as sample face images; adopt a machine learning algorithm to acquire an initial generative adversarial network, wherein the initial generative adversarial network includes a first sub-model, a second sub-model, a third sub-model and a fourth sub-model, the input data of the fourth sub-model is a first hidden vector, and the output data of the fourth sub-model is a mouth key point; perform the following first training step: input the sample audio into the first sub-model included in the initial generative adversarial network to obtain the first hidden vector corresponding to the sample audio; input the first hidden vector corresponding to the sample audio into the fourth sub-model to obtain the predicted mouth key point corresponding to the sample audio; calculate the first function value of the first preset loss function based on the predicted mouth key point corresponding to the sample audio and the mouth key point extracted from the sample face image corresponding to the sample audio; if the calculated first function value is less than or equal to the first preset threshold, determine the model parameters of the first sub-model included in the current initial generative adversarial network as the model parameters of the first sub-model included in the trained end-to-end model.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute a computer program stored in the memory, and when the computer program is executed, implement the method described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method described in any one of claims 1 to 8 is implemented.

Images