Video generation method, device, server and storage medium

Abstract

The embodiment of the present application discloses a video generation method, device, server and storage medium, wherein the method includes: obtaining human voice audio data, and obtaining a face image of a target object; generating simulated facial expression parameters according to the human voice audio data, and generating three-dimensional face parameters of the target object according to the face image; generating an initial dynamic face video of the target object according to the simulated facial expression parameters and the three-dimensional face parameters; correcting the face lip shape in the initial dynamic face video according to the human voice audio data to obtain a target dynamic face video; the face lip shape in the target dynamic face video corresponds to the human voice content in the human voice audio data. A dynamic face video with realistic visual and auditory effects can be generated.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a video generation method, device, server and storage medium. Background Art

[0002] With the rapid development of computer technology, computer vision has been increasingly used in daily life, work, and study. For example, there are currently a large number of synthetic videos on the Internet. Such synthetic videos can be facial expression videos generated based on a voice and a face image. However, when synthesizing facial expression videos through voice and face images, people usually only focus on the mouth movement of the face, while the overall expression of the face and the synchronization of the voice content are poor. The face with only mouth movement is relatively stiff, which results in poor video quality of the synthesized facial expression video. Therefore, how to synthesize a more realistic facial expression video has become a hot research issue in current computer vision technology. Summary of the invention

[0003] The embodiments of the present application provide a video generation method, device, server and storage medium, which can make dynamic face videos achieve better effects in two dimensions of expression and lip shape, and produce dynamic face videos with realistic visual and auditory effects.

[0004] The first aspect of the embodiment of the present application discloses a video generation method, the method comprising:

[0005] Acquire human voice audio data and obtain the face image of the target object;

[0006] Generate simulated facial expression parameters according to the human voice audio data, and generate three-dimensional facial parameters of the target object according to the facial image;

[0007] Generate an initial dynamic face video of the target object according to the simulated facial expression parameters and the three-dimensional face parameters;

[0008] The face lip shape in the initial dynamic face video is corrected according to the human voice audio data to obtain a target dynamic face video; the face lip shape in the target dynamic face video corresponds to the human voice content in the human voice audio data.

[0009] A second aspect of an embodiment of the present application discloses a video generating device, the device comprising:

[0010] An acquisition unit, used to acquire human voice audio data and acquire a face image of a target object;

[0011] A first generating unit, configured to generate simulated facial expression parameters according to the human voice audio data, and to generate three-dimensional facial parameters of the target object according to the facial image;

[0012] A second generating unit, configured to generate an initial dynamic face video of the target object according to the simulated face expression parameters and the three-dimensional face parameters;

[0013] A correction unit is used to correct the face lip shape in the initial dynamic face video according to the human voice audio data to obtain a target dynamic face video; the face lip shape in the target dynamic face video corresponds to the human voice content in the human voice audio data.

[0014] A third aspect of an embodiment of the present application discloses a server, comprising a processor, a memory and a network interface, wherein the processor, the memory and the network interface are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method of the first aspect above.

[0015] The fourth aspect of an embodiment of the present application discloses a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method of the first aspect above.

[0016] The fifth aspect of the embodiment of the present application discloses a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a server reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the server executes the method of the first aspect above.

[0017] In an embodiment of the present application, the server can obtain human voice audio data and a facial image of a target object, and generate an initial dynamic facial video of the target object based on the human voice audio data and the facial image of the target object. When generating the initial dynamic facial video of the target object based on the human voice audio data and the facial image of the target object, simulated facial expression parameters can be generated according to the human voice audio data, and three-dimensional facial parameters of the target object can be generated according to the facial image, so that the initial dynamic facial video for the target object can be generated according to the simulated facial expression parameters and the three-dimensional facial parameters. Further, the lip shape of the face in the initial dynamic facial video can be corrected according to the human voice audio data to obtain a target dynamic facial video, and the lip shape of the face in the target dynamic facial video corresponds to the human voice content in the human voice audio data. By implementing the above method, a facial expression change video related to the human voice audio content can be generated by an image generated by the three-dimensional facial parameters and the simulated facial expression parameters generated by the human voice audio. And based on the lip shape generation method, the lip shape included in the face of the facial expression change video can be synchronized with the human voice audio content. As a result, the final dynamic face video can achieve better results in the two dimensions of expression and lip shape, and thus can produce a dynamic face video with realistic visual and auditory effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a schematic diagram of the architecture of a video generation system provided in an embodiment of the present application;

[0020] Figure 2 It is a flowchart of a video generation method provided in an embodiment of the present application;

[0021] Figure 3a This is a schematic diagram of a structure for obtaining human voice audio data and a personal face image provided by an embodiment of the present application;

[0022] Figure 3b This is a schematic diagram of a structure for determining an initial face image provided by an embodiment of the present application;

[0023] Figure 3c It is a structural schematic diagram of a lip shape generation adversarial model provided in an embodiment of the present application;

[0024] Figure 4 It is a flowchart of a video generation method provided in an embodiment of the present application;

[0025] Figure 5a This is a structural diagram of determining converted human voice audio data provided by an embodiment of the present application;

[0026] Figure 5b It is a structural schematic diagram of a timbre conversion model provided in an embodiment of the present application;

[0027] Figure 5c It is a structural schematic diagram of a face restoration model provided in an embodiment of the present application;

[0028] Figure 5d It is a structural diagram of a GAN priori network model provided in an embodiment of the present application;

[0029] Figure 5e It is a structural schematic diagram of a GAN block provided in an embodiment of the present application;

[0030] Figure 5f It is a structural schematic diagram of a face restoration adversarial model provided in an embodiment of the present application;

[0031] Figure 6 It is a structural diagram of a video recommendation scenario provided by an embodiment of the present application;

[0032] Figure 7 is a structural schematic diagram of a video generating device provided in an embodiment of the present application;

[0033] Figure 8 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0036] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0037] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further process images so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0038] The solution provided in the embodiments of the present application involves technologies such as video processing and image processing in computer vision technology of artificial intelligence, which is specifically described by the following embodiments:

[0039] In an embodiment of the present application, a video generation method is provided, and the general principle of the video generation method is as follows: first, human voice audio data and a facial image of a target object can be obtained, and an initial dynamic facial video for the target object can be generated based on the human voice audio data and the facial image of the target object. When generating the initial dynamic facial video for the target object based on the human voice audio data and the facial image of the target object, simulated facial expression parameters can be generated according to the human voice audio data, and three-dimensional facial parameters of the target object can be generated according to the facial image, so that the initial dynamic facial video of the target object can be generated according to the simulated facial expression parameters and the three-dimensional facial parameters. Further, the lip shape of the face in the initial dynamic facial video can be corrected according to the human voice audio data to obtain a target dynamic facial video, and the lip shape of the face in the target dynamic facial video corresponds to the human voice content in the human voice audio data. In the present application, a facial expression change video related to the human voice audio content can be generated by an image generated by the simulated facial expression parameters generated by the three-dimensional facial parameters and the human voice audio. And based on the lip shape generation method, the lip shape included in the face of the facial expression change video can be synchronized with the human voice audio content. As a result, the final dynamic face video can achieve better results in the two dimensions of expression and lip shape, and thus can produce a dynamic face video with realistic visual and auditory effects.

[0040] It should be noted that the video generation method provided in this application can be specifically applied to a video generation system, see Figure 1 , Figure 1 Schematic diagram of the architecture of a video generation system provided by an embodiment of the present application. The present application relates to a terminal and a server.

[0041] Taking the terminal as an example, the target object can input the target object's facial image and voice audio data on the terminal interface, and the terminal can obtain the facial image and voice audio data input by the target object. After the terminal obtains the facial image and voice audio data, the facial image and voice audio data can be uploaded to the server, so that the server obtains the facial image and voice audio data, and generates an initial dynamic facial video based on the facial image and voice audio data, and then generates a target dynamic facial video based on the initial dynamic facial video. After the target dynamic facial video is generated, the server can also return the target dynamic facial video to the terminal so that the target dynamic facial video is displayed on the terminal interface.

[0042] Figure 1The terminal shown can be a smart phone, tablet computer, laptop computer, desktop computer and other devices, and can also be a peripheral device such as a handle, touch screen and the like; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, etc. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and the embodiments of the present application are not limited here.

[0043] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0044] See also Figure 2 , Figure 2 is a flow chart of a video generation method provided in an embodiment of the present application. The method is applied to a server and can be executed by the server, such as Figure 2 As shown, the video generation method may include:

[0045] S201: Acquire human voice audio data and acquire a face image of a target object.

[0046] The human voice audio data may be the speech of a certain object, or a song, etc.

[0047] In one implementation, the object in the present application may refer to a user, and therefore, the target object may refer to any user. When the target object needs to use a human voice audio data and its own facial image to generate a dynamic facial video containing human voice audio data with a facial image, the target object may perform relevant operations through the user operation interface output on the terminal so that the computer device can obtain the human voice audio data and the facial image. The facial lip shape in the dynamic facial video corresponds to the vocal content in the vocal audio data. This correspondence can be understood as the facial lip shape changing in accordance with the vocal content, and the pronunciation of each word in the vocal content is consistent with the corresponding facial lip shape (mouth shape). See, for example, Figure 3aAs shown: the object terminal (or terminal) used by the target object can display a user operation interface in the terminal screen, and the user operation interface can at least include an input data setting area marked by 301, and a confirmation control marked by 302. Among them, the input data setting area includes an audio input setting item and an image input setting item, the audio input setting item is used for the target object to input human voice audio data, and the image input setting item is used for the target object to input a face image. If the target object wants to generate a dynamic face video containing human voice audio data of a face image, the target object can input a human voice audio data and a face image in the input data setting area 301. For example, the human voice audio data can be a speech of the target object, or a song sung by the target object, or a speech of other objects, or a song sung by other objects; the face image can be a photo containing the face of the target object. After the target object inputs the human voice audio data and facial image, a trigger operation (such as a click operation, a press operation, etc.) can be performed on the confirmation control 302, so that the terminal used by the target object can obtain the human voice audio data and facial image in the input data setting area 301, and send the human voice audio data and facial image to the server.

[0048] S202: Generate simulated facial expression parameters according to the human voice audio data, and generate three-dimensional facial parameters of the target object according to the facial image.

[0049] In one implementation, in order to make the final generated video a video based on changes in facial expressions, the human voice audio data and the facial image can be combined to determine the generation of an image with changes in facial expressions, so that a video based on changes in facial expressions can be generated based on the image with changes in facial expressions. Specifically, simulated facial expression parameters can be generated based on the human voice audio data, and three-dimensional facial parameters for the target object can be generated based on the facial image, so that subsequent processing can be performed based on the simulated facial expression parameters and the three-dimensional facial parameters.

[0050] In one implementation, in the process of generating simulated facial expression parameters according to human voice audio data, a facial expression mapping method can be introduced, by which facial expression parameters can be estimated from an audio, and the method can be implemented by an expression parameter extraction model. For example, human voice audio data can be input into the expression parameter extraction model, so as to process the human voice audio data using the expression parameter extraction model, thereby generating simulated facial expression parameters. Optionally, human voice audio data can be input into the expression parameter extraction model, so as to perform feature conversion on the human voice audio data based on the expression parameter extraction model, and obtain speech feature parameters of the human voice audio data, and the speech feature parameters can be Mel Frequency Cepstrum Coefficient (MFCC) features, and the MFCC features are cepstrum analyzed on the Mel (Mel) spectrum (for example, specifically, the logarithm of the Mel spectrum can be taken, and the inverse transformation can be performed, and the inverse transformation is generally implemented by discrete cosine transformation, and the 2nd to 13th coefficients after discrete cosine transformation can be taken as MFCC features), and the obtained Mel frequency cepstrum coefficient (MFCC), this MFCC is the feature of the human voice audio data. Then, MFCC can be used as speech features. In this application, MFCC features can be used as speech feature parameters. After obtaining the speech feature parameters, the speech feature parameters can be feature migrated based on the expression parameter extraction model to obtain the target audio features of the human voice audio data. The target audio features can be high-order features for the audio. It can be understood that compared with low-order features (low-order features are usually related features such as contours, edges, colors, and shapes, and the feature semantic information contained in low-order features is relatively small), the feature semantic information contained in high-order features is relatively rich, and the high-order features can be used to subsequently characterize a complete face image. From the above, it can be seen that the present application can extract high-order features, which can be effectively independent of any specific identity and contain sufficient feature semantic information for predicting the simulated facial expression parameters required subsequently. After obtaining the target audio features, the expression parameters can be mapped according to the target audio features to obtain simulated facial expression parameters.

[0051] For example, the above-mentioned expression parameter extraction model may include three networks, which may be a feature conversion network, a feature migration network and a mapping network. Among them, the feature conversion network can be used to convert the input human voice audio data into MFCC features (speech feature parameters); the feature migration network is used to convert the input MFCC features into target audio features, and the target audio features may be high-order features for audio. Among them, the feature migration network may be AT-Net, and the 256-dimensional output features of the penultimate layer in AT-Net may be used as target audio features, which output features may be effectively independent of any specific identity and contain sufficient feature semantic information for subsequent mapping. Optionally, a 256-dimensional output feature may be extracted for every 40 milliseconds of human voice audio, and the output feature may correspond to a video frame of 1 second and containing 25 frames of images. The mapping network may be an audio-facial expression mapping network, and the target audio features may be used as the input of the mapping network to predict the facial expression parameters according to the target audio features. The predicted facial expression parameters may be referred to as simulated facial expression parameters. Among them, the relevant data of the network structure of the mapping network may be shown in Table 1:

[0052] Table 1:

[0053] Type Convolution Kernel Stride Outputs Convolutional Layer 3 1 5×254 Convolutional Layer 3 1 3×252 Convolutional Layer 3 1 1×250 Fully connected layer - - 64

[0054] Among them, according to Table 1, it can be seen that the mapping network can include three one-dimensional convolutional layers and one fully connected layer. The mapping network uses three one-dimensional convolutional layers for convolution processing. The one-dimensional convolution kernel used in each convolutional layer is 3, the step size is 1, the output of the first convolutional layer is a 5×254 matrix, the output of the second convolutional layer is a 3×252 matrix, and the output of the third convolutional layer is a 1×250 matrix. The fully connected layer can include 64 nodes, that is, the fully connected layer can reduce the vector of length 250 (the output of the third convolutional layer) to a vector of length 64, and the vector of length 64 is the facial expression parameter. In other words, the fully connected layer can use a fully connected layer of 64 nodes to output the predicted facial expression parameters. Among them, the loss function corresponding to the mapping network can be the mean square error loss (MSE), which can be specifically expressed as L exp =MSE(H(F t )-δ t ), where H(·) represents the effect of the mapping network, and F t represents the target audio feature, H(F t ) represents the predicted facial expression parameters, δ t Represents the real facial expression parameters.

[0055] In one implementation, three-dimensional face parameters for a target object can be generated based on a face image, and the three-dimensional face parameters can include initial face expression parameters and face morphology parameters of the target object, and the face morphology parameters can include geometric shape parameters, face texture parameters, head posture parameters, and illumination parameters, wherein the geometric shape parameters can be used to characterize the face shape, the face texture parameters can be used to characterize the face texture, the head posture parameters can be used to characterize the head posture corresponding to the face, and the illumination parameters can be used to characterize the illumination on the face. Optionally, a three-dimensional face construction model can be used to obtain the three-dimensional face parameters for the target object. For example, a face image can be input into a three-dimensional face construction model, so that the face key points of the face of the target object in the face image can be extracted based on the three-dimensional face construction model, and then in the three-dimensional face construction model, the face of the target object can be reconstructed based on the face key points to obtain the three-dimensional face parameters of the target object.

[0056] For example, the 3D face construction model can be a Detailed Expression Capture and Animation (DECA) model. The DECA model can perform 3D face reconstruction based on a single image to reconstruct a 3D head model with head posture, shape, and detailed facial geometry parameters based on a single input image. The DECA model introduces a detail consistency loss, which can decompose details related to the identity and expression of the object. This decomposition can synthesize real wrinkles by controlling the expression parameters while keeping the specific details of the object unchanged. In other words, the DECA model can be used to obtain the detail parameters of a specific object. These details are specific to an object and can change with changes in expression.

[0057] S203: Generate an initial dynamic face video of the target object according to the simulated face expression parameters and the three-dimensional face parameters.

[0058] In one implementation, it can be seen from the above that the three-dimensional face parameters may include the initial face expression parameters and face morphology parameters of the target object; the human voice audio data may have multiple frames of audio data, and one frame of audio data corresponds to a set of simulated face expression parameters. The specific implementation method of generating the initial dynamic face video for the target object according to the simulated face expression parameters and the three-dimensional face parameters may be: first, the initial face expression parameters in the three-dimensional face parameters may be replaced with the simulated face expression parameters corresponding to each frame of audio data, and the target face parameters corresponding to each frame of audio data of the target object may be obtained. Then, according to the target face parameters corresponding to each frame of audio data, the initial face image of the target object corresponding to each frame of audio data may be generated. Thus, according to the initial face image corresponding to each frame of audio data, the initial dynamic face video may be generated. For example, the initial face images corresponding to each frame of audio data may be synthesized into an initial dynamic face video by the FFMPEG tool. The FFMPEG tool is a tool that can be used for audio and video processing, and a series of image frames may be synthesized into a video by using the FFMPEG tool.

[0059] In one implementation, after synthesizing the initial facial image through the target facial parameters, directly using the initial facial image to generate the initial dynamic facial video has a poor realistic effect. For example, the initial dynamic facial video may only contain a face, but not the hair or background image corresponding to the face, and the face may not be realistic enough. As can be seen from the above, it is possible to consider rendering the initial facial image to generate an initial dynamic facial video based on the rendered initial facial image. Optionally, the initial facial image corresponding to each frame of audio data can be rendered to obtain a rendered facial image corresponding to each frame of audio data. After obtaining the rendered facial image, the initial dynamic facial video can be generated based on the rendered facial image corresponding to each frame of audio data.

[0060] Optionally, the step of performing image rendering may include: first, the above-mentioned synthesized initial face image may be rendered using a rendering engine, wherein the rendering using the rendering engine mainly renders the face part. It can be understood that the initial face image rendered by the rendering engine does not have hair and background areas, and for the authenticity of a dynamic face video, the hair and background areas are usually necessary for a dynamic face video. For example, for a real singing video, the hair and background areas are indispensable. For the convenience of subsequent description, the initial face image rendered by the rendering engine can be referred to as a first rendered face image. Then, after the initial face image is rendered by the rendering engine, the face alignment can be realized by using a 3D dense face alignment (3D Dense Face Alignment, 3DDFA) method to render the first rendered face image to obtain a rendered first rendered face image, and the rendered first rendered face image can be referred to as a second rendered face image, in which the second rendered face image has hair and background matching the face. After the above rendering operation, the second rendered face image may still look like it is synthesized by a computer in terms of visual effect, that is, the second rendered face image is not very realistic. In order to make the face in the second rendered face image look more realistic and natural, a neural face renderer can be used to render the second rendered face image, and the synthesized frame in the second rendered face image can be converted into a real frame, so that the face image obtained by the neural face rendering model is more realistic, and the face image obtained by the neural face rendering model can be called a rendered face image. The neural face rendering model can be a generative adversarial rendering model, and the generative adversarial rendering model consists of a generative network and a discriminative network. It can be understood that when the neural face rendering model is trained, the model structure used is a generative adversarial model structure, and when the neural face rendering model is applied to the actual rendering scene, the generative network in the neural face rendering model is used. The generative network in the neural face rendering model may include a rendering encoding network and a rendering decoding network. Among them, the rendering encoding network can use a convolutional layer to downsample the input data of the rendering encoding network, and then use a normalization layer and an activation layer (Relu) to perform batch normalization and activation processing respectively. The rendering decoding network can synthesize high-quality output from low-dimensional potential representations through upsampling, transposed convolution, batch normalization, Dropout layer, and activation function (ReLU). From the above, we can see that after rendering processing, a more realistic rendered face image can be obtained.

[0061] For example, if Figure 3b The schematic diagram of the structure of obtaining the initial face image is shown in FIG. Figure 3bAs shown, human voice audio data can be input into the expression parameter extraction model to obtain simulated facial expression parameters. At the same time, the facial image can also be input into the three-dimensional face construction model to obtain the three-dimensional facial parameters of the target object, wherein the three-dimensional facial parameters can include the initial facial expression parameters and facial morphology parameters of the target object. Then, the initial facial expression parameters in the three-dimensional facial parameters can be replaced with the simulated facial expression parameters to obtain the target facial parameters corresponding to the target object. Then, the initial facial image of the target object can be generated according to the target facial parameters.

[0062] S204: Correcting the lip shape of the face in the initial dynamic face video according to the human voice audio data to obtain a target dynamic face video.

[0063] In one implementation, the lip shape of the face in the initial dynamic face video is relatively rough and lacks detail changes. In order to make the video more realistic, the lip shape of the face in the initial dynamic face video can be optimized to obtain a realistic video. For example, the lip shape of the face in the initial dynamic face video can be corrected according to the human voice audio data to obtain a target dynamic face video after the face lip shape correction, and the face lip shape in the target dynamic face video corresponds to the human voice content in the human voice audio data. Optionally, the face lip shape in the initial dynamic face video can be corrected according to the lip shape correction model to obtain the target dynamic face video. For example, the human voice audio data and the initial dynamic face video can be input into the lip shape correction model to extract the audio data features corresponding to each frame of audio data based on the lip shape correction model, and then according to the audio data features corresponding to each frame of audio data, the lip shape in the initial face image corresponding to each frame of audio data in the initial dynamic face video is corrected to obtain the target face image corresponding to each frame of audio data after the lip shape correction. Furthermore, a target dynamic face video after lip shape correction can be generated based on the target face image corresponding to each frame of audio data.

[0064] In one implementation, the above-mentioned lip shape correction model can be obtained by training a lip shape generation adversarial model, and the lip shape generation adversarial model can be a wav2lip network model. The lip shape generation adversarial model may include a lip shape generation network, a lip shape discrimination network, and a video quality discrimination network. Specifically, a first training sample set can be obtained, and the first training sample set may include at least one first training sample pair. After obtaining the first training sample set, the lip shape generation adversarial model can be trained according to the first training sample set. In describing the subsequent training process, any first training sample pair in the first training sample set is taken as an example. Then, the specific implementation method of training the lip shape generation adversarial model to obtain the lip shape correction model can be as follows:

[0065] A first training sample pair may be obtained, the first training sample pair may include sample audio data and sample video data, and a frame of audio data in the sample audio data corresponds to a frame of video data in the sample video data. Figure 3c As shown, the first training sample pair can be input into the lip shape generation network to obtain a predicted dynamic face video. After obtaining the predicted dynamic face video, the predicted dynamic face video and the sample audio data can be input into the lip shape discrimination network to obtain a lip shape discrimination result for the predicted dynamic face video. The predicted dynamic face video and the sample video data can be input into the video quality discrimination network to obtain a quality discrimination result for the predicted dynamic face video. After obtaining the predicted dynamic face video, sample video data, lip shape discrimination result and quality discrimination result, the network parameters of the lip shape generation network can be corrected according to the predicted dynamic face video, sample video data, lip shape discrimination result and quality discrimination result, thereby obtaining a target lip shape generation network, which can be determined as a lip shape correction model.

[0066] In one implementation, the first sample training pair may include multiple groups of data frames, and one group of data frames includes a frame of audio data in the corresponding sample audio data and a frame of video data in the sample video data. For the lip shape generation network, the specific implementation method of inputting the first training sample pair into the lip shape generation network to obtain the predicted dynamic face video may include: first, based on the lip shape generation network, the sample audio features of each frame of audio data in the sample audio data can be extracted respectively, and the sample video features of each frame of video data in the sample video data can be extracted respectively. Then, the sample audio features corresponding to the sample audio data contained in each group of data frames and the sample video features corresponding to the sample video data are respectively fused to obtain the fused sample features corresponding to each group of data frames. Further, the fused sample features corresponding to each group of data frames can be decoded to obtain the predicted face images corresponding to each data frame. Then, after obtaining the predicted face images corresponding to each group of data frames, the predicted dynamic face video can be generated according to the predicted face images corresponding to each group of data frames.

[0067] In one implementation, the lip shape generation adversarial model can use the loss function shown in Formula 1 to optimize the network parameters in the lip shape generation adversarial model. The loss function includes three types of losses, namely reconstruction loss L1, synchronization loss L2 and quality loss L3. The reconstruction loss L1 may refer to the loss associated with the lip shape generation network. The reconstruction loss L1 may be an L1 norm, which can be used to measure the difference between the predicted dynamic face video (the video output by the lip shape generation network) and the sample video data (which can be understood as the real video data). The synchronization loss L2 may refer to the loss associated with the lip shape discrimination network. The lip shape discrimination network is a discrimination network that has been pre-trained and whose network performance has been verified to be good. The lip shape discrimination network can be used to detect errors in lip synchronization in predicted dynamic face videos. The synchronization loss L2 can be shown in Formula 1.

[0068]

[0069] Where N represents the number of the first training sample pairs, P i represents the probability that the i-th pair of samples is a positive sample (which can be understood as sample video data (real video data)), P i The calculation formula can be shown as formula 2:

[0070]

[0071] Among them, v and s are sample video features and sample audio features respectively, and v·s represents the dot product of sample video features and sample audio features.

[0072] The quality loss L3 may refer to the loss associated with the video quality discrimination network. During the training process of the lip shape generation adversarial model, the lip shape generation network may generate very accurate lip shapes due to the overly precise network effect of the lip shape discrimination network. However, in this case, the image frames contained in the predicted dynamic face video generated by the lip shape generation network may be blurred or have slight ghosting, which affects the image quality of the image frames contained in the predicted dynamic face video, and further affects the video quality of the predicted dynamic face video. In order to reduce the loss of this image quality (or video quality), the video quality discrimination network in the present application may be introduced to solve the above problem. The video quality discrimination network may not perform any checks on the lip shapes generated in the predicted dynamic face video, but may punish the unreal (low-quality) faces generated in the predicted dynamic face video. Among them, the quality loss L3 may be a binary cross entropy loss function.

[0073] In an embodiment of the present application, human voice audio data and a facial image of a target object can be obtained, and an initial dynamic facial video for the target object can be generated based on the human voice audio data and the facial image of the target object. When generating the initial dynamic facial video for the target object based on the human voice audio data and the facial image of the target object, simulated facial expression parameters can be generated according to the human voice audio data, and three-dimensional facial parameters of the target object can be generated according to the facial image, so that the initial dynamic facial video of the target object can be generated according to the simulated facial expression parameters and the three-dimensional facial parameters. Further, the lip shape of the face in the initial dynamic facial video can be corrected according to the human voice audio data to obtain a target dynamic facial video, and the lip shape of the face in the target dynamic facial video corresponds to the human voice content in the human voice audio data. In the present application, a facial expression change video related to the human voice audio content can be generated by an image generated by the simulated facial expression parameters generated by the three-dimensional facial parameters and the human voice audio. And based on the lip shape generation method, the lip shape included in the face of the facial expression change video can be synchronized with the human voice audio content. As a result, the final dynamic face video can achieve better results in the two dimensions of expression and lip shape, and thus can produce a dynamic face video with realistic visual and auditory effects.

[0074] See also Figure 4 , Figure 4 FIG. 1 is a flow chart of another video generation method provided in an embodiment of the present application. The method is applied to a server and can be executed by the server. Figure 4 As shown, the video generation method may include:

[0075] S401: Acquire human voice audio data and acquire a face image of a target object.

[0076] In one implementation, target vocal audio data of the target object may also be obtained, and the target vocal audio data may be a speech audio or song audio of the target object, etc. The timbre conversion may be performed based on the vocal audio data and the target vocal audio data, and the original timbre of the vocal audio data may be converted into the timbre of the target vocal audio data, while the vocal content in the vocal audio data remains unchanged.

[0077] In one implementation, the specific implementation of timbre conversion may include: after obtaining the target human voice audio data, the human voice timbre features of the target object can be extracted according to the target human voice audio data, for example, it can be predicted using a pre-trained neural network model, and the neural network model can be an embedded model. The human voice audio data can also be converted into a human voice audio sequence, and the acoustic feature data of the human voice audio data can be obtained, and the acoustic feature data can include the fundamental frequency, root mean square error (RMSE) sequence and audio duration of the human voice audio data. Further, the human voice timbre features, the human voice audio sequence, and the acoustic feature data are input into the timbre conversion model to obtain a Mel spectrogram containing the human voice timbre of the target object, and the Mel spectrogram is a spectrogram that converts the frequency into a Mel (mel) scale. After obtaining the Mel spectrogram, the Mel spectrogram can be vocoded to obtain the converted human voice audio data of the target object. Among them, the converted human voice audio data has the human voice content of the human voice audio data, and has the human voice timbre of the target human voice audio data, that is, the converted human voice audio data replaces the human voice timbre in the human voice audio data.

[0078] For example, if Figure 5a The following is a schematic diagram of a timbre conversion structure. Figure 5aAs shown, assuming that the human voice audio data is a song, and the target human voice audio data is a speech of the target object, the purpose of the timbre conversion is to convert the timbre of the original singer in the song into the timbre of the target object while ensuring that the singing content in the song remains unchanged. In a specific implementation, after obtaining the song and the speech, the song and the speech can be processed respectively to obtain the data required for timbre conversion. For example, for a song, the acoustic feature data of the song can be obtained from the song, and the acoustic feature data can include the fundamental frequency, RMSE sequence and audio duration for the song. The acoustic feature data can be used as the conditions required in the timbre conversion process. Among them, the fundamental frequency can be obtained by processing the song through a world vocoder, and the song can also be converted into a text sequence to obtain the human voice audio sequence corresponding to the song according to the text sequence. After obtaining the human voice audio sequence, the human voice audio sequence and the song can also be aligned to obtain the audio duration. The alignment process may be, for example, a forced alignment, and the alignment process may refer to determining the duration of each audio frame in the vocal audio sequence from the song, for example, it may be measured by the duration of the speech unit corresponding to each audio frame. The audio duration may be understood as the duration corresponding to each audio frame included in the vocal audio sequence. For another example, for spoken speech, a pre-trained speech embedding network may be used to obtain the speech embedding of the target object, and the speech embedding may be the vocal timbre feature of the target object. After obtaining the vocal timbre feature, the vocal audio sequence, and the acoustic feature data (base frequency, RMSE sequence, and audio duration), the above-obtained data may be input into the timbre conversion model to obtain a Mel-spectrogram containing the vocal timbre of the target object. After obtaining the Mel-spectrogram, the Mel-spectrogram may be vocoded, wherein the Mel-spectrogram may be vocoded using a neural vocoder (e.g., WaveRNN) to generate a waveform, thereby obtaining the converted vocal audio data of the target object. The converted vocal audio data has the vocal content of the song and has the vocal timbre in the spoken speech. That is, through the above-mentioned timbre conversion, the timbre of the singing voice included in the song can be converted into the sound of the target object while keeping the content of the song unchanged.

[0079] In one implementation, the timbre conversion model may include an encoding network, an alignment network, and a decoding network, and the specific implementation method of obtaining a Mel-spectrogram through the timbre conversion model may include: a human voice audio sequence may be input into the encoding network for encoding processing to obtain an implicit feature sequence of the human voice audio sequence. After obtaining the implicit feature sequence, the implicit feature sequence, human voice timbre features, and acoustic feature data may be input into the alignment network, and the implicit feature sequence may be aligned in the alignment network based on the human voice timbre features and acoustic feature data to obtain an aligned audio sequence. Finally, the aligned audio sequence may be input into a decoding module for decoding processing to obtain a Mel-spectrogram.

[0080] For example, if Figure 5b The figure shows a schematic diagram of the structure of the timbre conversion model, which can be a Duration Informed Attention Network based Singing Voice Conversion System (DurIAN-SC) model. The timbre conversion model includes a context encoding network that can be used to encode an audio sequence, an alignment network for aligning an audio sequence (an audio sequence for vocal audio data) with a target audio sequence (an audio sequence for target vocal audio data), and an autoregressive decoding network for generating Mel-spectrogram features frame by frame. Among them, the encoding network (context encoding network) of the timbre conversion model can include an audio embedding layer, a fully connected layer, and a CBHG layer. The CBHG layer can also include a convolution layer, a high channel layer (Highway layer), and a bidirectional gated recurrent unit (Gated Recurrent Unit, GRU) layer. Specifically, assuming that the vocal audio sequence x 1:N , where N represents the length of the human voice audio sequence. The human voice audio sequence x 1:N Input the encoding network to obtain the implicit feature sequence of the human voice audio sequence, which can be expressed as h 1:N =encoder(x 1:N ), encoder(.) represents the effect of the encoding network. When the human voice audio sequence is input into the encoding network, the human voice audio sequence can first pass through the audio embedding layer and the fully connected layer to generate an implicit representation corresponding to the human voice audio sequence, and then pass through the CBHG layer to extract the features in the implicit representation, thereby obtaining an implicit feature sequence, which contains the order of audio frames in the human voice audio sequence.

[0081] The purpose of the alignment network is to generate an implicit state of audio frame alignment from the implicit feature sequence. This implicit state can be called an aligned audio sequence, and the implicit state is input into the decoding network. After the implicit feature sequence is input into the alignment network, first, the implicit feature sequence h 1:N The representation after the state expansion can be e according to the duration of the audio 1:T =state-expand(h 1:N , d 1:N ), where T is the total number of input audio frames, and state-expand(.) represents the effect of state expansion. State expansion is based on the audio duration d corresponding to each audio frame. 1:N Perform a copy operation. For the convenience of subsequent description, the sequence obtained after the implicit feature sequence is state-expanded can be referred to as the first implicit feature sequence. After the implicit feature sequence is state-expanded, the frame-aligned first implicit feature sequence can be connected with the fundamental frequency, RMSE sequence and human voice and chromatic features to obtain an aligned audio sequence. For example, the first implicit feature sequence obtained after state expansion can be connected (Concat) with the acoustic feature data (fundamental frequency and RMSE sequence) to obtain a second implicit feature sequence. Then, the second implicit feature sequence is connected with the human voice and chromatic features to obtain a third implicit feature sequence. Next, the third implicit feature sequence is passed through a fully connected layer, and the sequence obtained through the fully connected layer is the output of the alignment network, which is an aligned audio sequence.

[0082] The decoding network is an autoregressive decoding network, which is the same as the decoding network in the Duration Informed Attention Network (DurIAN) model, and is also composed of two autoregressive recurrent neural network (RNN) layers. The aligned audio sequence output by the alignment network can be autoregressively decoded through a decoding network composed of a fully connected layer and an RNN layer. For example, two frames can be decoded at each time point during the decoding process to obtain the output y of the decoding network 1:T ,y 1:T = decoder(v 1:T ), v 1:T represents the output of the alignment network, and decoder(.) represents the effect of the decoding network. The output of the decoding network can also be processed by the CBHG layer to improve the quality of the resulting mel-spectrogram.

[0083] From the above, it can be seen that after obtaining the converted vocal audio data, the generation of simulated facial expression parameters based on the vocal audio data mentioned in the subsequent step S402 can also be the generation of simulated facial expression parameters based on the converted vocal audio data, and the correction of the facial lip shape in the initial dynamic facial video based on the vocal audio data mentioned in step S404 can also be the correction of the facial lip shape in the initial dynamic facial video based on the converted vocal audio data, so that the facial lip shape in the final target dynamic facial video can correspond to the vocal content in the converted vocal audio data, and the target dynamic facial video has the vocal tonality of the target vocal audio data.

[0084] S402: Generate simulated facial expression parameters according to the human voice audio data, and generate three-dimensional facial parameters of the target object according to the facial image.

[0085] S403: Generate an initial dynamic face video of the target object according to the simulated face expression parameters and the three-dimensional face parameters.

[0086] S404: Correcting the lip shape of the face in the initial dynamic face video according to the human voice audio data to obtain a corrected dynamic face video.

[0087] Among them, correcting the dynamic face video may include multiple frames of corrected face images.

[0088] S405: Obtain the face position in each frame of the corrected face image, and perform image quality restoration on the face in each frame of the corrected face image according to the face position in each frame of the corrected face image.

[0089] In one implementation, the above process of generating an initial dynamic facial video for the target object and correcting the facial lip shape in the initial dynamic facial video to obtain a corrected dynamic facial video may cause loss in the quality of the generated video, resulting in the video quality of the corrected dynamic facial video being blurry. In order to make the final video present better picture quality, the corrected dynamic facial video obtained above can be further restored to obtain a video with better picture quality. The quality of the corrected dynamic facial video after the restoration is clearer.

[0090] In one implementation, when performing image quality restoration on a corrected dynamic face video, it may be considered to first extract a partial face image in the corrected dynamic face video to restore the face in the corrected dynamic face video. It is understandable that the corrected dynamic face video may include multiple frames of corrected face images. Then, the face position in each frame of the corrected face image may be first obtained, so as to restore the face in each frame of the corrected face image according to the face position in each frame of the corrected face image.

[0091] Optionally, the face position in each frame of the corrected face image can be obtained first, so that the partial face image in each frame of the corrected face image can be extracted according to the face position in each frame of the corrected face image. For example, face extraction can be performed based on the RetinaFace model. The RetinaFace model is a single-stage face detector with strong robustness. The RetinaFace model can combine supervision and self-supervision and perform multi-task learning. The RetinaFace model can use a feature pyramid to extract multi-scale features. The RetinaFace model can be used to locate faces of different scales, so that the face position in each frame of the corrected face image can be obtained more accurately. After obtaining the face position in each frame of the corrected face image, the corrected face image can be affine transformed and cropped according to the face position in each frame of the corrected face image to extract a valid face image in each frame of the corrected face image. The valid face image can be called a partial face image. It can be understood that one frame of the corrected face image corresponds to one frame of the partial face image.

[0092] After obtaining the partial face image corresponding to each corrected face image in the corrected dynamic face video, the image quality of each partial face image can be restored, so that the target dynamic face video can be obtained later based on the partial face image after the image quality restoration. For example, the image quality of each partial face image can be restored using a face restoration model. In a specific implementation, each partial face image can be input into a face restoration model to extract the coded face features of each partial face image based on the face restoration model. After obtaining the coded face features of each partial face image, the restored face features of each partial face image can be obtained according to the coded face feature mapping of each partial face image. Furthermore, the image quality of each partial face image can be restored according to the coded face features and the restored face features corresponding to each partial face image, so as to obtain the restored face partial images corresponding to each partial face image.

[0093] Among them, the face restoration model can be Figure 5c The face restoration model adopts the classic encoding-decoding structure. The face restoration model is formed by embedding the Generative Adversarial Networks (GAN) prior network as a decoding network into a U-shaped deep neural network. The network structure of the GAN prior network can be shown as follows Figure 5dAs shown, the relevant details in the GAN prior network can refer to the description of the face restoration model, which will not be repeated here. It can be understood that the face restoration model can include a decoding network and an encoding network, and the decoding network is the GAN prior network, wherein the decoding network can include multiple feature extraction layers, and the encoding network can include multiple GAN blocks. The number of feature extraction layers is equal to the number of GAN blocks. The GAN block can be as shown in FIG. Figure 5c The generation adversarial layer shown in FIG. 1 is the same as the GAN block in this application. The network structure of each GAN block can be as follows: Figure 5e As shown. Considering that the face restoration model is a low-quality face image processing model with incomplete convolution, before inputting the partial face image into the face restoration model, the low-resolution partial face image can be adjusted to the required resolution size (for example, the resolution of the low-resolution partial face image can be adjusted to 1024*1024), wherein a bilinear interpolator can be used to adjust the resolution corresponding to the partial face image. Then, the partial face image with adjusted resolution can be input into the face restoration model later.

[0094] For a frame of partial face image, first, the partial face image can be extracted to obtain the coded face features of the partial face image through the coding network in the face restoration model, and the coded face features include the coded face features corresponding to each feature extraction layer included in the decoding network. Among them, the coded face features can include shallow features and deep features. The shallow features can be understood as the coded face features corresponding to the shallower feature extraction layer in the coding network. The shallow features can usually include basic features about the partial face image, such as the edge or corner of the face, and the high-level features can usually be related features used to characterize a complete face. The deep features can be understood as the coded face features corresponding to the deeper feature extraction layer in the coding network. After obtaining the coded face features corresponding to each feature extraction layer, the coded face features corresponding to the last feature extraction layer in the coding network are extracted, and the coded face features are passed through the fully connected layer to obtain the latent face features, and then the latent face features are passed through the mapping network to obtain the less entangled restoration face features for the partial face image.

[0095] After obtaining the restored face features and the encoded face features corresponding to each feature extraction layer in the encoding network, the restored face features can be broadcast to each GAN block in the decoding network, and the encoded face features corresponding to each feature extraction layer in the encoding network can be used as a skip layer connection and input into each GAN block in the decoding network in a one-to-one manner in a noise manner to control the global face structure, local face details and background of the final restored face partial image. The output of the encoding network is the restored face partial image corresponding to the face partial image. Through the face restoration model, the restored face partial image corresponding to each frame of the face partial image can be obtained. After obtaining each frame of the restored face partial image, each frame of the corrected face image with the corresponding relationship can be synthesized with each frame of the restored face partial image to obtain each frame of the corrected face image after the image quality is restored.

[0096] In one implementation, the face restoration model can be obtained by training a face restoration adversarial model, which can be a GAN prior embedded network (GPEN) model. Similarly, the decoding network of the face restoration adversarial model can be a GAN prior network, which can generate high-quality face images and can be easily embedded in a U-shaped GPEN model as a decoding network. The face restoration adversarial model can include Figure 5f The encoding network, decoding network and restoration discriminant network. Specifically, a second training sample set can be obtained, and the second training sample set can include at least one second training sample pair. After the second training sample set is obtained, the face restoration adversarial model can be trained according to the second training sample set. In describing the subsequent training process, any second training sample pair in the second training sample set is used as an example. Then, the specific implementation method of training the face restoration adversarial model to obtain the face restoration model can be as follows:

[0097] A second training sample pair may be obtained, and the second training sample pair may include a first resolution image and a second resolution image, wherein the image resolution of the first resolution image is smaller than the image resolution of the second resolution image, and the first resolution image and the second resolution image have the same image content. Figure 5fAs shown, the second training sample pair can be input into the encoding network to obtain the sample encoded face features corresponding to the first resolution image, wherein the encoding network can include multiple feature extraction layers, and the sample encoded face features corresponding to the first resolution image include the sample encoded face features corresponding to each feature extraction layer. Then, the sample repair face features of the first resolution image can be obtained according to the sample encoded face feature mapping, wherein the specific determination method of the sample repair face features can refer to the above-mentioned method for determining the repair face features, which will not be repeated here. Further, the sample encoded face features and the sample repair face features can be input into the decoding network to obtain the predicted resolution image corresponding to the first resolution image. After obtaining the predicted resolution image, the second resolution image and the predicted resolution image can be input into the repair discrimination network to obtain the repair discrimination result of the predicted resolution image. The first image resolution feature of the first resolution image and the second image resolution feature of the second resolution image can also be obtained based on the repair discrimination network. Furthermore, the network parameters of the face repair adversarial model can be corrected according to the predicted resolution image, the second resolution image, the repair discrimination result, the first image resolution feature and the first image resolution feature. Therefore, the face restoration model can be determined based on the encoding network and decoding network in the modified face restoration adversarial model.

[0098] It should be noted that before training the face restoration adversarial model, the GAN prior network (decoding network) is first trained. After the GAN prior network training is completed, the pre-trained GAN prior network can be embedded into the U-shaped deep neural network as a decoding network to obtain the face restoration adversarial model. Subsequently, the second training sample set is used to fine-tune the face restoration adversarial model so that the encoding network and decoding network in the face restoration adversarial model can be better combined. Among them, a high-definition face image dataset can be used. For example, the face image dataset can be a FFHQ (Flickr-Faces-High-Quality) dataset, and the GAN prior network is trained according to the training strategy of StyleGAN.

[0099] In one implementation, in order to fine-tune (or train) the face restoration adversarial model, the face restoration adversarial model can use a loss function as shown in Formula 3 to optimize the network parameters in the face restoration adversarial model. The loss function includes three types of losses, which are: adversarial loss L A , content loss L C And feature matching loss L F .L A and L F They can be shown as Formula 4 and Formula 5 respectively.

[0100] L=L A +αLC +βL F Formula 3

[0101]

[0102]

[0103] Among them, the content loss L C The detailed features in the face image (first resolution image) are enhanced, the original color information is retained, and the feature matching loss L is introduced in the repair discriminant network. F Can better balance the adversarial loss L A , thus restoring a more realistic and vivid face image, that is, a face image with better image quality. α and β are balance parameters that can be set according to needs. Practice has shown that when α=1 and β=0.02, the training effect of the face restoration adversarial model is better. represents the first resolution image (or can be understood as a low-resolution image (blurred image)), X represents the second resolution image (or can be understood as a high-resolution image (high-definition image) corresponding to the low-resolution image). G represents the generative network when training the face restoration adversarial model, which may include an encoding network and a decoding network, and D represents the restoration discriminant network. G(·) and D(·) represent the effects of the generative network and the restoration discriminant network, respectively. Content loss L C It is the L1 norm, which can be used to measure the difference between the predicted resolution image (the image output by the decoding network) and the second resolution image (which can be understood as the real high-definition image). Feature matching loss L F is the perceptual loss based on the repair discriminant network, where T is the total number of feature extraction layers included in the repair discriminant network, and D i (X) represents the feature extracted by the i-th layer of the repair discriminant network. In this application, D i (X) can be understood as the second image resolution feature based on the above-mentioned restoration discriminant network for the second resolution image; It can be understood as the first image resolution feature of the first resolution image based on the above-mentioned restoration discriminant network.

[0104] S406: Generate a target dynamic face video according to each frame of the corrected face image after image quality restoration.

[0105] In one implementation, after obtaining each frame of the corrected face image after image quality restoration, the target dynamic face video can be synthesized according to each frame of the corrected face image after image quality restoration. The lip shape of the face in the target dynamic face video corresponds to the voice content in the voice audio data. If the audio data acquired by the server is voice audio data and target voice audio data, the voice audio data and the target voice audio data are converted into timbre, then the voice timbre in the target dynamic face video is the voice timbre of the target voice audio data. If the audio data acquired by the server is voice audio data, the voice timbre in the target dynamic face video is the voice timbre of the voice audio data.

[0106] The specific implementation of steps S401-S404 can refer to the specific description of steps S201-S204 in the above embodiment, which will not be repeated here.

[0107] In an embodiment of the present application, a video of facial expression changes related to vocal audio content can be generated by an image generated by simulating facial expression parameters generated by three-dimensional facial parameters and vocal audio. And based on the lip shape generation method, the lip shape included in the face of the facial expression change video can be synchronized with the vocal audio content, so that the expression and lip shape in the facial expression change video can achieve better effects. The facial expression change video can also be repaired based on the face repair model to achieve a high-definition face in the facial expression change video. The vocal audio can also be converted to timbre. For example, the target vocal audio of a certain object can be used to convert the original timbre in the vocal audio to the timbre in the target vocal audio, and the original audio content in the vocal audio can be preserved unchanged. Thereby, while ensuring that the video has realistic visual and auditory effects, the fun of the video is increased, thereby improving the user experience.

[0108] To better understand the video generation method provided in the embodiment of the present application, the following video recommendation scene is combined with Figure 6 The video recommendation scenario diagram shown in FIG. 1 is further described, wherein the solution can be deployed in the cloud, that is, the server executing the video generation method can be a cloud server. Figure 6 As shown in FIG. 1 , in the video recommendation scenario, the target object can input a song (which can be understood as the above-mentioned human voice audio data, where a song or a speech can be input) and a face photo of the target object (which can be understood as the above-mentioned face image) on the user operation interface of the target terminal corresponding to the target object. The user operation interface can be as follows: Figure 3aWhen the target object clicks the OK control, the target terminal can obtain the song and the face photo, and send the song and the face photo to the server, so that the server can generate a target dynamic face video based on the obtained song and face photo, and the face lip shape in the target dynamic face video corresponds to the vocal content in the song.

[0109] In a specific implementation, the server can drive the generation of facial expressions based on audio (taking songs as an example). For example, the song and the facial photo can be combined to determine the generation of an image with facial expression changes, so that a video based on facial expression changes can be generated based on the image with facial expression changes. Specifically, the song can be input into the expression parameter extraction model to generate simulated facial expression parameters of the target object, and the facial photo can be input into the three-dimensional face construction model to generate three-dimensional facial parameters for the target object. Then, the initial facial expression parameters in the three-dimensional facial parameters are replaced with simulated facial expression parameters to obtain the target facial parameters for the target object, so that the initial facial image of the target object can be synthesized based on the target facial parameters. For the authenticity of the subsequent video, the initial facial image can also be rendered to generate a realistic rendered facial image, so as to generate an initial dynamic facial video based on the rendered facial image. The initial dynamic face video is then input into the lip shape correction model, and the song also needs to be input into the lip shape correction model, so that the lip shape correction model can correct the face lip shape in the initial dynamic face video according to the song, and obtain the corrected target dynamic face video, wherein the face lip shape in the corrected dynamic face video corresponds to the vocal content in the song. Finally, the face in the corrected target dynamic face video is restored based on the face restoration model to generate a song video with realistic visual and auditory effects, which is the target dynamic face video mentioned above. When the vocal audio data is a spoken voice, the target dynamic face video can be a spoken video.

[0110] In one implementation, when the target object inputs a song and a face photo of the target object on the user operation interface of the object terminal, it can also input a speech voice (which can be understood as the above-mentioned target vocal audio data, and the speech voice here is different from the vocal timbre of the speech voice that can be included in the vocal audio data). Then, the object terminal can also send the acquired speech voice to the server, so that the server can perform song timbre conversion based on the speech voice and the song, so as to convert the vocal timbre of the original singer in the song into the vocal timbre of the target object when speaking, thereby generating a high-quality object song (which can be understood as the above-mentioned converted vocal audio data). Then, the facial lip shape in the target dynamic face video finally generated corresponds to the vocal content in the song, and the timbre of the original singer in the song is the speaking timbre of the target object.

[0111] See also Figure 7 , is a schematic diagram of the structure of a video generation device provided in an embodiment of the present application. The video generation device described in this embodiment includes:

[0112] The acquisition unit 701 is used to acquire human voice audio data and acquire a face image of a target object;

[0113] A first generating unit 702 is used to generate simulated facial expression parameters according to the human voice audio data, and to generate three-dimensional facial parameters of the target object according to the facial image;

[0114] A second generating unit 703 is used to generate an initial dynamic face video of the target object according to the simulated face expression parameters and the three-dimensional face parameters;

[0115] The correction unit 704 is used to correct the face lip shape in the initial dynamic face video according to the human voice audio data to obtain a target dynamic face video; the face lip shape in the target dynamic face video corresponds to the human voice content in the human voice audio data.

[0116] In one implementation, the first generating unit 702 is specifically configured to:

[0117] Inputting the human voice audio data into an expression parameter extraction model, performing feature conversion on the human voice audio data based on the expression parameter extraction model, and obtaining speech feature parameters of the human voice audio data;

[0118] Performing feature migration on the speech feature parameters based on the expression parameter extraction model to obtain target audio features of the human voice audio data;

[0119] Expression parameter mapping is performed according to the target audio feature to obtain the simulated facial expression parameters.

[0120] In one implementation, the first generating unit 702 is specifically configured to:

[0121] The face image is input into a three-dimensional face construction model so that the three-dimensional face construction model extracts the face key points of the target object in the face image, and uses the face key points to reconstruct the face of the target object to obtain the three-dimensional face parameters of the target object.

[0122] In one implementation, the three-dimensional face parameters include initial face expression parameters and face morphology parameters of the target object; the human voice audio data has multiple frames of audio data, and one frame of audio data corresponds to a set of simulated face expression parameters; the first generating unit 702 is specifically used to:

[0123] Replacing the initial facial expression parameters in the three-dimensional facial parameters with the simulated facial expression parameters corresponding to each frame of audio data, respectively, to obtain target facial parameters corresponding to each frame of audio data for the target object;

[0124] Generating an initial face image corresponding to each frame of audio data according to the target face parameters corresponding to each frame of audio data;

[0125] An initial dynamic face video of the target object is generated according to the initial face image corresponding to each frame of audio data.

[0126] In one implementation, the second generating unit 703 is specifically configured to:

[0127] Performing image rendering on the initial face image corresponding to each frame of audio data to obtain a rendered face image corresponding to each frame of audio data;

[0128] An initial dynamic face video of the target object is generated according to the rendered face images corresponding to each frame of audio data.

[0129] In one implementation, the correction unit 704 is specifically configured to:

[0130] Inputting the human voice audio data and the initial dynamic face video into a lip shape correction model;

[0131] Extracting audio data features corresponding to each frame of audio data based on the lip shape correction model;

[0132] According to the audio data features corresponding to each frame of audio data, the lip shapes in the initial face images corresponding to each frame of audio data in the initial dynamic face video are corrected to obtain the target face images corresponding to each frame of audio data;

[0133] The target dynamic face video after lip shape correction is generated according to the target face image corresponding to each frame of audio data.

[0134] In one implementation, the lip shape correction model is obtained by training a lip shape generation adversarial model, wherein the lip shape generation adversarial model includes a lip shape generation network, a lip shape discrimination network, and a video quality discrimination network; the correction unit 704 is further used to:

[0135] Acquire a first training sample pair; the first training sample pair comprises sample audio data and sample video data, and a frame of audio data in the sample audio data corresponds to a frame of video data in the sample video data;

[0136] Inputting the first training sample pair into the lip shape generation network to obtain a predicted dynamic face video;

[0137] Inputting the predicted dynamic face video and the sample audio data into the lip shape discrimination network to obtain a lip shape discrimination result for the predicted dynamic face video;

[0138] Inputting the predicted dynamic face video and the sample video data into the video quality judgment network to obtain a quality judgment result for the predicted dynamic face video;

[0139] According to the predicted dynamic face video, the sample video data, the lip shape discrimination result and the quality discrimination result, the network parameters of the lip shape generation network are modified to obtain a target lip shape generation network;

[0140] The target lip shape generation network is determined as the lip shape correction model.

[0141] In one implementation, the first sample training pair includes multiple groups of data frames, and a group of data frames includes a corresponding frame of audio data in the sample audio data and a frame of video data in the sample video data; the correction unit 704 is specifically used to:

[0142] Based on the lip shape generation network, respectively extracting sample audio features of each frame of audio data in the sample audio data, and respectively extracting sample video features of each frame of video data in the sample video data;

[0143] Performing feature fusion on the sample audio features of the sample audio data and the sample video features of the sample video data contained in each group of data frames to obtain fused sample features corresponding to each group of data frames;

[0144] Decoding the fused sample features corresponding to each group of data frames to obtain predicted face images corresponding to each group of data frames;

[0145] The predicted dynamic face video is generated according to the predicted face images corresponding to each group of data frames.

[0146] In one implementation, the acquiring unit 701 is further configured to:

[0147] Acquire target human voice audio data of the target object, and extract the human voice tonality characteristics of the target object according to the target human voice audio data;

[0148] Converting the human voice audio data into a human voice audio sequence, and obtaining acoustic feature data of the human voice audio data;

[0149] Inputting the human voice timbre feature, the human voice audio sequence, and the acoustic feature data into a timbre conversion model to obtain a mel-spectrogram containing the human voice timbre of the target object;

[0150] The mel-spectrogram is subjected to voice coding conversion to obtain converted human voice audio data of the target object; the converted human voice audio data has the human voice content of the human voice audio data and the human voice tonality of the target human voice audio data.

[0151] In one implementation, the timbre conversion model includes an encoding network, an alignment network, and a decoding network; the acquisition unit 701 is specifically used to:

[0152] The step of inputting the human voice timbre feature, the human voice audio sequence, and the acoustic feature data into a timbre conversion model to obtain a mel-spectrogram containing the human voice timbre of the target object includes:

[0153] Inputting the human voice audio sequence into the encoding network for encoding processing to obtain an implicit feature sequence of the human voice audio sequence;

[0154] Inputting the implicit feature sequence, the human voice tone feature and the acoustic feature data into the alignment network, and performing audio frame alignment on the implicit feature sequence based on the human voice tone feature and the acoustic feature data in the alignment network to obtain an aligned audio sequence;

[0155] The aligned audio sequence is input into the decoding module for decoding processing to obtain the Mel-spectrogram.

[0156] In one implementation, the correction unit 704 is specifically configured to:

[0157] Correcting the lip shape of the face in the initial dynamic face video according to the human voice audio data to obtain a corrected dynamic face video, wherein the corrected dynamic face video includes multiple frames of corrected face images;

[0158] Obtaining the face position in each frame of the corrected face image, and performing image quality restoration on the face in each frame of the corrected face image according to the face position in each frame of the corrected face image;

[0159] The target dynamic face video is generated according to each frame of the corrected face image after image quality restoration.

[0160] In one implementation, the correction unit 704 is specifically configured to:

[0161] Extracting a partial face image in each frame of the corrected face image according to the face position in each frame of the corrected face image; one frame of the corrected face image corresponds to one frame of the partial face image;

[0162] Inputting each frame of partial face image into a face restoration model, and extracting encoded face features of each frame of partial face image based on the face restoration model;

[0163] Obtaining restored facial features of each frame of partial face image according to the encoded facial feature mapping of each frame of partial face image;

[0164] Performing image quality restoration on each frame of the partial face image according to the encoded face features and the restored face features of each frame of the partial face image, to obtain a restored partial face image corresponding to each frame of the partial face image;

[0165] The corrected face images of each frame having a corresponding relationship are synthesized with the repaired face partial images of each frame to obtain the corrected face images of each frame after the image quality is restored.

[0166] In one implementation, the face restoration model is obtained by training a face restoration adversarial model, and the face restoration adversarial model includes an encoding network, a decoding network, and a restoration discriminant network; the correction unit 704 is further used to:

[0167] Acquire a second training sample pair, where the second training sample pair includes a first resolution image and a second resolution image; the image resolution of the first resolution image is smaller than the image resolution of the second resolution image, and the first resolution image and the second resolution image have the same image content;

[0168] Inputting the second training sample pair into the encoding network to obtain sample encoded facial features corresponding to the first resolution image, and obtaining sample restored facial features of the first resolution image according to the sample encoded facial feature mapping;

[0169] Inputting the sample encoded face features and the sample restored face features into the decoding network to obtain a predicted resolution image corresponding to the first resolution image;

[0170] Inputting the second resolution image and the predicted resolution image into the restoration discrimination network to obtain a restoration discrimination result of the predicted resolution image;

[0171] Acquire a first image resolution feature of the first resolution image and a second image resolution feature of the second resolution image based on the restoration discriminant network;

[0172] According to the predicted resolution image, the second resolution image, the restoration discrimination result, the first image resolution feature and the first image resolution feature, modifying the network parameters of the face restoration adversarial model;

[0173] The face restoration model is determined according to the modified encoding network and decoding network in the face restoration adversarial model.

[0174] In one implementation, the acquiring unit 701 is specifically configured to:

[0175] Acquire the human voice audio data and the human face image sent by the object terminal of the target object;

[0176] The device further includes an output unit 705, wherein the output unit 705 is specifically configured to:

[0177] The target dynamic face video is sent to the object terminal so that the object terminal outputs the target dynamic face video.

[0178] It is understood that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The functional units in the embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0179] See also Figure 8 , is a schematic diagram of a server structure provided in an embodiment of the present application. The server described in this embodiment includes: a processor 801, a memory 802, and a network interface 803. The processor 801, the memory 802, and the network interface 803 can exchange data.

[0180] The processor 801 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0181] The memory 802 may include a read-only memory and a random access memory, and provide program instructions and data to the processor 801. A portion of the memory 802 may also include a non-volatile random access memory. When the processor 801 calls the program instructions, it is used to execute:

[0182] Acquire human voice audio data and obtain the face image of the target object;

[0183] Generate simulated facial expression parameters according to the human voice audio data, and generate three-dimensional facial parameters of the target object according to the facial image;

[0184] Generate an initial dynamic face video of the target object according to the simulated facial expression parameters and the three-dimensional face parameters;

[0185] The face lip shape in the initial dynamic face video is corrected according to the human voice audio data to obtain a target dynamic face video; the face lip shape in the target dynamic face video corresponds to the human voice content in the human voice audio data.

[0186] In one implementation, the processor 801 is specifically configured to:

[0187] Inputting the human voice audio data into an expression parameter extraction model, performing feature conversion on the human voice audio data based on the expression parameter extraction model, and obtaining speech feature parameters of the human voice audio data;

[0188] Performing feature migration on the speech feature parameters based on the expression parameter extraction model to obtain target audio features of the human voice audio data;

[0189] Expression parameter mapping is performed according to the target audio feature to obtain the simulated facial expression parameters.

[0190] In one implementation, the processor 801 is specifically configured to:

[0191] The face image is input into a three-dimensional face construction model so that the three-dimensional face construction model extracts the face key points of the target object in the face image, and uses the face key points to reconstruct the face of the target object to obtain the three-dimensional face parameters of the target object.

[0192] In one implementation, the three-dimensional face parameters include initial face expression parameters and face morphology parameters of the target object; the human voice audio data has multiple frames of audio data, and one frame of audio data corresponds to a set of simulated face expression parameters; the processor 801 is specifically used to:

[0193] Replacing the initial facial expression parameters in the three-dimensional facial parameters with the simulated facial expression parameters corresponding to each frame of audio data, respectively, to obtain target facial parameters corresponding to each frame of audio data for the target object;

[0194] Generating an initial face image corresponding to each frame of audio data according to the target face parameters corresponding to each frame of audio data;

[0195] An initial dynamic face video of the target object is generated according to the initial face image corresponding to each frame of audio data.

[0196] In one implementation, the processor 801 is specifically configured to:

[0197] Performing image rendering on the initial face image corresponding to each frame of audio data to obtain a rendered face image corresponding to each frame of audio data;

[0198] An initial dynamic face video of the target object is generated according to the rendered face images corresponding to each frame of audio data.

[0199] In one implementation, the processor 801 is specifically configured to:

[0200] Inputting the human voice audio data and the initial dynamic face video into a lip shape correction model;

[0201] Extracting audio data features corresponding to each frame of audio data based on the lip shape correction model;

[0202] According to the audio data features corresponding to each frame of audio data, the lip shapes in the initial face images corresponding to each frame of audio data in the initial dynamic face video are corrected to obtain the target face images corresponding to each frame of audio data;

[0203] The target dynamic face video after lip shape correction is generated according to the target face image corresponding to each frame of audio data.

[0204] In one implementation, the lip shape correction model is obtained by training a lip shape generation adversarial model, wherein the lip shape generation adversarial model includes a lip shape generation network, a lip shape discrimination network, and a video quality discrimination network; the processor 801 is further used to:

[0205] Acquire a first training sample pair; the first training sample pair comprises sample audio data and sample video data, and a frame of audio data in the sample audio data corresponds to a frame of video data in the sample video data;

[0206] Inputting the first training sample pair into the lip shape generation network to obtain a predicted dynamic face video;

[0207] Inputting the predicted dynamic face video and the sample audio data into the lip shape discrimination network to obtain a lip shape discrimination result for the predicted dynamic face video;

[0208] Inputting the predicted dynamic face video and the sample video data into the video quality judgment network to obtain a quality judgment result for the predicted dynamic face video;

[0209] According to the predicted dynamic face video, the sample video data, the lip shape discrimination result and the quality discrimination result, the network parameters of the lip shape generation network are modified to obtain a target lip shape generation network;

[0210] The target lip shape generation network is determined as the lip shape correction model.

[0211] In one implementation, the first sample training pair includes multiple groups of data frames, and one group of data frames includes a corresponding frame of audio data in the sample audio data and a frame of video data in the sample video data; the processor 801 is specifically configured to:

[0212] Based on the lip shape generation network, respectively extracting sample audio features of each frame of audio data in the sample audio data, and respectively extracting sample video features of each frame of video data in the sample video data;

[0213] Performing feature fusion on the sample audio features of the sample audio data and the sample video features of the sample video data contained in each group of data frames to obtain fused sample features corresponding to each group of data frames;

[0214] Decoding the fused sample features corresponding to each group of data frames to obtain predicted face images corresponding to each group of data frames;

[0215] The predicted dynamic face video is generated according to the predicted face images corresponding to each group of data frames.

[0216] In one implementation, the processor 801 is further configured to:

[0217] Acquire target human voice audio data of the target object, and extract the human voice tonality characteristics of the target object according to the target human voice audio data;

[0218] Converting the human voice audio data into a human voice audio sequence, and obtaining acoustic feature data of the human voice audio data;

[0219] Inputting the human voice timbre feature, the human voice audio sequence, and the acoustic feature data into a timbre conversion model to obtain a mel-spectrogram containing the human voice timbre of the target object;

[0220] The mel-spectrogram is subjected to voice coding conversion to obtain converted human voice audio data of the target object; the converted human voice audio data has the human voice content of the human voice audio data and the human voice tonality of the target human voice audio data.

[0221] In one implementation, the timbre conversion model includes an encoding network, an alignment network, and a decoding network; the processor 801 is specifically configured to:

[0222] The step of inputting the human voice timbre feature, the human voice audio sequence, and the acoustic feature data into a timbre conversion model to obtain a mel-spectrogram containing the human voice timbre of the target object includes:

[0223] Inputting the human voice audio sequence into the encoding network for encoding processing to obtain an implicit feature sequence of the human voice audio sequence;

[0224] Inputting the implicit feature sequence, the human voice tone feature and the acoustic feature data into the alignment network, and performing audio frame alignment on the implicit feature sequence based on the human voice tone feature and the acoustic feature data in the alignment network to obtain an aligned audio sequence;

[0225] The aligned audio sequence is input into the decoding module for decoding processing to obtain the Mel-spectrogram.

[0226] In one implementation, the processor 801 is specifically configured to:

[0227] Correcting the lip shape of the face in the initial dynamic face video according to the human voice audio data to obtain a corrected dynamic face video, wherein the corrected dynamic face video includes multiple frames of corrected face images;

[0228] Obtaining the face position in each frame of the corrected face image, and performing image quality restoration on the face in each frame of the corrected face image according to the face position in each frame of the corrected face image;

[0229] The target dynamic face video is generated according to each frame of the corrected face image after image quality restoration.

[0230] In one implementation, the processor 801 is specifically configured to:

[0231] Extracting a partial face image in each frame of the corrected face image according to the face position in each frame of the corrected face image; one frame of the corrected face image corresponds to one frame of the partial face image;

[0232] Inputting each frame of partial face image into a face restoration model, and extracting encoded face features of each frame of partial face image based on the face restoration model;

[0233] Obtaining restored facial features of each frame of partial face image according to the encoded facial feature mapping of each frame of partial face image;

[0234] Performing image quality restoration on each frame of the partial face image according to the encoded face features and the restored face features of each frame of the partial face image, to obtain a restored partial face image corresponding to each frame of the partial face image;

[0235] The corrected face images of each frame having a corresponding relationship are synthesized with the repaired face partial images of each frame to obtain the corrected face images of each frame after the image quality is restored.

[0236] In one implementation, the face restoration model is obtained by training a face restoration adversarial model, and the face restoration adversarial model includes an encoding network, a decoding network, and a restoration discriminant network; the processor 801 is further used to:

[0237] Acquire a second training sample pair, where the second training sample pair includes a first resolution image and a second resolution image; the image resolution of the first resolution image is smaller than the image resolution of the second resolution image, and the first resolution image and the second resolution image have the same image content;

[0238] Inputting the second training sample pair into the encoding network to obtain sample encoded facial features corresponding to the first resolution image, and obtaining sample restored facial features of the first resolution image according to the sample encoded facial feature mapping;

[0239] Inputting the sample encoded face features and the sample restored face features into the decoding network to obtain a predicted resolution image corresponding to the first resolution image;

[0240] Inputting the second resolution image and the predicted resolution image into the restoration discrimination network to obtain a restoration discrimination result of the predicted resolution image;

[0241] Acquire a first image resolution feature of the first resolution image and a second image resolution feature of the second resolution image based on the restoration discriminant network;

[0242] According to the predicted resolution image, the second resolution image, the restoration discrimination result, the first image resolution feature and the first image resolution feature, modifying the network parameters of the face restoration adversarial model;

[0243] The face restoration model is determined according to the modified encoding network and decoding network in the face restoration adversarial model.

[0244] In one implementation, the processor 801 is specifically configured to:

[0245] Acquire the human voice audio data and the human face image sent by the object terminal of the target object;

[0246] The target dynamic face video is sent to the object terminal so that the object terminal outputs the target dynamic face video.

[0247] The present application also provides a computer storage medium in which program instructions are stored. When the program is executed, it may include: Figure 2 or Figure 4 Part or all of the steps of the video generation method in the corresponding embodiment.

[0248] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0249] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0250] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the server reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the server executes the steps performed in the embodiments of the above methods.

[0251] The above is a detailed introduction to a video generation method, device, server and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video generation method, characterized in that: The method comprises: Acquire human voice audio data and obtain the face image of the target object; Generate simulated facial expression parameters according to the human voice audio data, and generate three-dimensional facial parameters of the target object according to the facial image; Generate an initial dynamic face video of the target object according to the simulated facial expression parameters and the three-dimensional face parameters; Correcting the lip shape of the face in the initial dynamic face video according to the human voice audio data to obtain a target dynamic face video; the lip shape of the face in the target dynamic face video corresponds to the human voice content in the human voice audio data; Wherein, the three-dimensional facial parameters include initial facial expression parameters and facial morphology parameters of the target object; the human voice audio data has multiple frames of audio data, and one frame of audio data corresponds to a group of simulated facial expression parameters; the generating the initial dynamic facial video of the target object according to the simulated facial expression parameters and the three-dimensional facial parameters comprises: replacing the initial facial expression parameters in the three-dimensional facial parameters with the simulated facial expression parameters corresponding to each frame of audio data, respectively, to obtain the target facial parameters corresponding to each frame of audio data for the target object; generating the initial facial images corresponding to each frame of audio data according to the target facial parameters corresponding to each frame of audio data; performing image rendering on the initial facial images corresponding to each frame of audio data, to obtain the rendered facial images corresponding to each frame of audio data; generating the initial dynamic facial video of the target object according to the rendered facial images corresponding to each frame of audio data; The process of performing image rendering on any of the initial facial images includes: using a rendering engine to render any of the initial facial images to obtain a corresponding first rendered facial image; performing face alignment on the first rendered facial image to obtain a corresponding second rendered facial image; and performing realistic face enhancement rendering on the second rendered facial image to obtain a corresponding rendered facial image.

2. The method according to claim 1, characterized in that The step of generating simulated facial expression parameters according to the human voice audio data comprises: Inputting the human voice audio data into an expression parameter extraction model, performing feature conversion on the human voice audio data based on the expression parameter extraction model, and obtaining speech feature parameters of the human voice audio data; Performing feature migration on the speech feature parameters based on the expression parameter extraction model to obtain target audio features of the human voice audio data; Expression parameter mapping is performed according to the target audio feature to obtain the simulated facial expression parameters.

3. The method according to claim 1, characterized in that The step of generating the three-dimensional face parameters of the target object according to the face image comprises: The face image is input into a three-dimensional face construction model so that the three-dimensional face construction model extracts the face key points of the target object in the face image, and uses the face key points to reconstruct the face of the target object to obtain the three-dimensional face parameters of the target object.

4. The method according to claim 1, characterized in that: The step of correcting the lip shape of the face in the initial dynamic face video according to the human voice audio data to obtain a target dynamic face video includes: Inputting the human voice audio data and the initial dynamic face video into a lip shape correction model; Extracting audio data features corresponding to each frame of audio data based on the lip shape correction model; According to the audio data features corresponding to each frame of audio data, the lip shapes in the initial face images corresponding to each frame of audio data in the initial dynamic face video are corrected to obtain the target face images corresponding to each frame of audio data; The target dynamic face video after lip shape correction is generated according to the target face image corresponding to each frame of audio data.

5. The method according to claim 4, characterized in that The lip shape correction model is obtained by training a lip shape generation adversarial model, wherein the lip shape generation adversarial model includes a lip shape generation network, a lip shape discrimination network and a video quality discrimination network; the method further includes: Acquire a first training sample pair; the first training sample pair comprises sample audio data and sample video data, and a frame of audio data in the sample audio data corresponds to a frame of video data in the sample video data; Inputting the first training sample pair into the lip shape generation network to obtain a predicted dynamic face video; Inputting the predicted dynamic face video and the sample audio data into the lip shape discrimination network to obtain a lip shape discrimination result for the predicted dynamic face video; Inputting the predicted dynamic face video and the sample video data into the video quality judgment network to obtain a quality judgment result for the predicted dynamic face video; According to the predicted dynamic face video, the sample video data, the lip shape discrimination result and the quality discrimination result, the network parameters of the lip shape generation network are modified to obtain a target lip shape generation network; The target lip shape generation network is determined as the lip shape correction model.

6. The method according to claim 5, characterized in that The first sample training pair includes a plurality of groups of data frames, and a group of data frames includes a corresponding frame of audio data in the sample audio data and a frame of video data in the sample video data; The step of inputting the first training sample pair into the lip shape generation network to obtain a predicted dynamic face video includes: Based on the lip shape generation network, respectively extracting sample audio features of each frame of audio data in the sample audio data, and respectively extracting sample video features of each frame of video data in the sample video data; Performing feature fusion on the sample audio features of the sample audio data and the sample video features of the sample video data contained in each group of data frames to obtain fused sample features corresponding to each group of data frames; Decoding the fused sample features corresponding to each group of data frames to obtain predicted face images corresponding to each group of data frames; The predicted dynamic face video is generated according to the predicted face images corresponding to each group of data frames.

7. The method according to claim 1, characterized in that After obtaining the human voice audio data, the method further includes: Acquire target human voice audio data of the target object, and extract the human voice tonality characteristics of the target object according to the target human voice audio data; Converting the human voice audio data into a human voice audio sequence, and obtaining acoustic feature data of the human voice audio data; Inputting the human voice timbre feature, the human voice audio sequence, and the acoustic feature data into a timbre conversion model to obtain a mel-spectrogram containing the human voice timbre of the target object; The mel-spectrogram is subjected to voice coding conversion to obtain converted vocal audio data of the target object; the converted vocal audio data has the vocal content of the vocal audio data and the vocal tonality of the target vocal audio data.

8. The method according to claim 7, characterized in that The timbre conversion model includes an encoding network, an alignment network and a decoding network; The step of inputting the human voice timbre feature, the human voice audio sequence, and the acoustic feature data into a timbre conversion model to obtain a mel-spectrogram containing the human voice timbre of the target object includes: Inputting the human voice audio sequence into the encoding network for encoding processing to obtain an implicit feature sequence of the human voice audio sequence; Inputting the implicit feature sequence, the human voice chromatic feature and the acoustic feature data into the alignment network, and performing audio frame alignment on the implicit feature sequence based on the human voice chromatic feature and the acoustic feature data in the alignment network to obtain an aligned audio sequence; The aligned audio sequence is input into the decoding module for decoding processing to obtain the Mel-spectrogram.

9. The method according to claim 1, characterized in that: The step of correcting the lip shape of the face in the initial dynamic face video according to the human voice audio data to obtain a target dynamic face video after the lip shape of the face is corrected includes: Correcting the lip shape of the face in the initial dynamic face video according to the human voice audio data to obtain a corrected dynamic face video, wherein the corrected dynamic face video includes multiple frames of corrected face images; Obtaining the face position in each frame of the corrected face image, and performing image quality restoration on the face in each frame of the corrected face image according to the face position in each frame of the corrected face image; The target dynamic face video is generated according to each frame of the corrected face image after image quality restoration.

10. The method according to claim 9, characterized in that The performing image quality restoration on the face in each frame of the corrected face image according to the face position in each frame of the corrected face image comprises: Extracting a partial face image in each frame of the corrected face image according to the face position in each frame of the corrected face image; one frame of the corrected face image corresponds to one frame of the partial face image; Inputting each frame of partial face image into a face restoration model, and extracting encoded face features of each frame of partial face image based on the face restoration model; Obtaining restored facial features of each frame of partial face image according to the encoded facial feature mapping of each frame of partial face image; Performing image quality restoration on each frame of the partial face image according to the encoded face features and the restored face features of each frame of the partial face image, to obtain a restored partial face image corresponding to each frame of the partial face image; The corrected face images of each frame having a corresponding relationship are synthesized with the repaired face partial images of each frame to obtain the corrected face images of each frame after the image quality is restored.

11. The method according to claim 10, characterized in that The face restoration model is obtained by training a face restoration adversarial model, wherein the face restoration adversarial model includes an encoding network, a decoding network and a restoration discriminant network; The method further comprises: Acquire a second training sample pair, where the second training sample pair includes a first resolution image and a second resolution image; the image resolution of the first resolution image is smaller than the image resolution of the second resolution image, and the first resolution image and the second resolution image have the same image content; Inputting the second training sample pair into the encoding network to obtain sample encoded facial features corresponding to the first resolution image, and obtaining sample restored facial features of the first resolution image according to the sample encoded facial feature mapping; Inputting the sample encoded face features and the sample restored face features into the decoding network to obtain a predicted resolution image corresponding to the first resolution image; Inputting the second resolution image and the predicted resolution image into the restoration discrimination network to obtain a restoration discrimination result of the predicted resolution image; Acquire a first image resolution feature of the first resolution image and a second image resolution feature of the second resolution image based on the restoration discriminant network; According to the predicted resolution image, the second resolution image, the restoration discrimination result, the first image resolution feature and the first image resolution feature, modifying the network parameters of the face restoration adversarial model; The face restoration model is determined according to the modified encoding network and decoding network in the face restoration adversarial model.

12. The method according to any one of claims 1 to 11, characterized in that: The step of obtaining human voice audio data and obtaining a face image of a target object includes: Acquire the human voice audio data and the human face image sent by the object terminal of the target object; The method further comprises: The target dynamic face video is sent to the object terminal so that the object terminal outputs the target dynamic face video.

13. A server, characterized in that: The method comprises a processor, a memory and a network interface, wherein the processor, the memory and the network interface are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 12.

14. A computer storage medium, characterized in that: The computer storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, a computer device having the processor executes the method according to any one of claims 1 to 12.

Images