Method, device, computer equipment and storage medium for generating character image video

Abstract

The present application relates to a method, apparatus, computer device, and storage medium for generating a character image video. The method obtains a driving video file corresponding to a character image and a voice signal, inputs the character image and the driving video file into a video driving image model, extracts first facial key point information and first torso key point information from the driving image, and extracts second facial key point information and second torso key point information from the character image. Based on the first facial key point information, first torso key point information, second facial key point information, and second torso key point information, the character image is supplemented with posture information contained in the driving image to obtain several frames of new character image images. Finally, a character image video is generated based on the several frames of new character image images and the voice signal. This method reduces the time and labor costs consumed in the video generation process.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, apparatus, computer equipment, and storage medium for generating a human image video. Background Art

[0002] With the development of computer vision technology, portrait-driven and synthesis technologies have emerged. These technologies can be used to create customized 2D digital human avatars for each person, with applications in scenarios such as news broadcasting and training. These technologies use a voice or video clip to drive a specific person's image and generate a video of the person that lip-syncs with the voice or video, including facial expressions and body movements. Facial expressions include eyebrows, nose, eyes, and mouth shape. The lip shape in the video must be synchronized with the audio to create a complete video.

[0003] In traditional technologies, portrait driving and synthesis technologies that want to drive and synthesize portraits for unknown characters must re-collect portrait data for model training.

[0004] However, the process of generating character image videos in traditional technologies requires a lot of time and manpower costs. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, device, computer equipment and storage medium for generating character image videos that can reduce the time cost and manpower cost consumed in the video generation process in order to address the above technical problems.

[0006] A method for generating a character image video, the method comprising:

[0007] Acquire a driving video file corresponding to a character image and a voice signal, wherein the driving video file includes a plurality of frames of driving images, and the driving images include a face portion and a torso portion of the character;

[0008] Inputting the character image and the driving video file into a video driving image model, extracting first facial key point information and first torso key point information of the driving image; extracting second facial key point information and second torso key point information of the character image;

[0009] Adding the posture information contained in the driving image to the character image based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information to obtain a plurality of frames of new character image images;

[0010] A character image video is generated based on the several frames of new character image images and the voice signal.

[0011] In one embodiment, obtaining a driving video file corresponding to a voice signal includes:

[0012] Acquiring voice signal features of the voice signal, and mapping the voice signal features to facial expression parameters of a 3D face model;

[0013] Constructing a facial image according to the facial expression parameters to obtain a plurality of frames of 3D facial images corresponding to the facial expression parameters;

[0014] Rendering is performed on each of the 3D face images to obtain a 2D real face image corresponding to the 3D face image, and the driving video file is generated according to each of the 2D real face images.

[0015] In one embodiment, the step of adding the posture information contained in the driving image to the character image based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information to obtain a plurality of frames of new character image images includes:

[0016] performing a local affine transformation on the character image according to the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information to obtain a dense motion field image and an occlusion mask image;

[0017] The image features of the character image are extracted, and feature fusion and decoding are performed based on the image features, the dense motion field image and the occlusion mask image to obtain the several frames of new character image images.

[0018] In one embodiment, the video-driven image model is trained in the following manner, including:

[0019] Obtaining a training sample set, wherein the training sample set includes a plurality of sample images, wherein the sample images are annotated with sample face key points and sample torso key points;

[0020] The sample images and training character images are input into the video-driven image model for portrait driving and synthesis. When the perceptual loss meets the model training stop condition, the training of the video-driven image model is completed.

[0021] In one embodiment, the perceptual loss adopts a pyramid model.

[0022] In one embodiment, the drive image is obtained in the following manner, including:

[0023] Acquire text data and input the text data into a text-driven model, wherein the text-driven model includes a text-to-speech model, a lip movement model, a 3D face model, and a GAN renderer;

[0024] Converting the text data into the speech signal using the text-to-speech model;

[0025] Inputting the speech signal features of the speech signal into the lip movement model to obtain facial expression parameters of the 3D face model;

[0026] The 3D face model is controlled by using the facial expression parameters to obtain a plurality of frames of 3D face images;

[0027] The several frames of 3D face images are input into a GAN renderer for rendering to obtain the driving image.

[0028] In one embodiment, the text-driven model is trained in the following manner, including:

[0029] Obtain speech signals and several video frame character images from a training video file;

[0030] Extracting video frame expression parameters from the character images in each of the video frames;

[0031] Training the lip movement model according to the speech signal and the video frame expression parameters;

[0032] Reconstructing the 3D face model according to the video frame expression parameters of each of the video frame character images to obtain a plurality of 3D video frame face images;

[0033] The GAN renderer is trained according to the 3D video frame face image and the video frame character image.

[0034] A device for generating a character image video, comprising:

[0035] A first acquisition module is used to acquire a driving video file corresponding to a character image and a voice signal, wherein the driving video file includes a plurality of frames of driving images, and the driving images include a face portion and a torso portion of the character;

[0036] A key point extraction module is used to input the character image and the driving video file into the video driving image model, extract the first facial key point information and the first torso key point information of the driving image; and extract the second facial key point information and the second torso key point information of the character image;

[0037] a posture information adding module, configured to add the posture information contained in the driving image to the character image based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information, to obtain a plurality of frames of new character image images;

[0038] The video generation module is used to generate a character image video based on the several frames of new character image and the voice signal.

[0039] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0040] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0041] The above-mentioned character image video generation method, apparatus, computer device, and storage medium obtain a driving video file corresponding to a character image and a voice signal, wherein the driving image in the driving video file includes a face portion and a torso portion; input the character image and the driving video file into a video driving image model, extract first facial key point information and first torso key point information from the driving image, and extract second facial key point information and second torso key point information from the character image; thereby, based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information, append the posture information contained in the driving image to the character image, thereby obtaining several frames of new character image; and finally, generate a character image video based on the several frames of new character image and the voice signal. This method rapidly expands character image driving and synthesis capabilities to different character images based on the character image and video driving image model, reducing the time and labor costs consumed in the video generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A diagram showing an application environment of a method for generating a character image video in one embodiment;

[0043] Figure 2 1 is a flow chart of a method for generating a character image video according to an embodiment;

[0044] Figure 3 1 is a flow chart of a method for generating a character image video according to an embodiment;

[0045] Figure 4a 1 is a flow chart of a method for generating a character image video according to another embodiment;

[0046] Figure 4b A schematic diagram of FOMM model data processing in one embodiment;

[0047] Figure 5 1 is a flow chart of a video-driven image model training method according to an embodiment;

[0048] Figure 6 A schematic diagram of a process for obtaining a driving image in one embodiment;

[0049] Figure 7a Schematic diagram of a flow chart of a text-driven model training method in one embodiment;

[0050] Figure 7b Schematic diagram of training data flow for a text-driven model in one embodiment;

[0051] Figure 7c 1. A flowchart of a method for generating a character image video according to an embodiment;

[0052] Figure 8 is a structural block diagram of a device for generating a character image video in one embodiment;

[0053] Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] The character image video generation method provided in this application can be applied to Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. Server 104 constructs a text-to-speech model, a lip movement model, a 3D face model, a GAN renderer, and a video-driven image model, and trains these constructed models. When the model training stop condition is met, these models are published to terminal 102. Terminal 102 obtains a driving video file corresponding to a character image and a voice signal, the driving video file including several frames of driving images, each of which includes a face portion and a torso portion. Terminal 102 inputs the character image and the driving video file into the video-driven image model, extracts first facial key point information and first torso key point information from the driving image, and extracts second facial key point information and second torso key point information from the character image. Based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information, the character image is appended with posture information contained in the driving image to obtain several frames of new character image images. A character image video is generated based on the several frames of new character image images and the voice signal. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0056] Human-like driving and synthesis technology uses audio or video to drive a specific person's image and generate a video of that person's image that matches the voice or video. This technology includes facial expressions and body movements. Facial expressions include eyebrows, nose, eyes, and mouth shape. The lip shape and audio are synchronized to create a complete video. Human-like driving and synthesis technology can customize a unique 2D digital human image for each person, and is applicable to scenarios such as news broadcasting and training.

[0057] Portrait-driven driving can be based on voice and text. Voice-driven portrait driving involves first collecting video data of the character image; second, based on the facial image, using 3D Morphable Models (3DMM) to construct a 3D face model for the character image. The 3D face model has two sets of parameters: facial shape and texture. By controlling these two sets of parameters, the facial expression and movement of the character image's 3D face can be controlled; then, the portrait video data is decomposed into audio and image frames, and the facial shape and texture parameters corresponding to each frame are obtained. A deep neural network is used to build a model to achieve a mapping between the audio and the facial shape and facial texture parameters of the character image's 3D face model. Then, voice can be used to control the character image's 3D face to produce corresponding expressions and lip movements. Finally, a generative adversarial network (GAN) is used to build a model to achieve a mapping between the character image's 3D face and a real face. During portrait driving and synthesis, the sound is first mapped to two sets of control parameters for a 3D face model. These control parameters are then used to generate the character's 3D facial expressions. Finally, the 3D face is passed through a GAN model to generate a realistic face image. Text-based driving can involve first converting text into sound using Text-to-Speech (TTS) technology, and then performing portrait driving and video synthesis based on the voice drive.

[0058] In one embodiment, Figure 2 As shown, a method for generating a character image video is provided, which is applied to Figure 1 The following steps are used as an example to illustrate the terminal in the figure:

[0059] S210: Obtain a driving video file corresponding to the character image and voice signal.

[0060] The driving video file includes several frames of driving images, each of which includes a human face and torso. A human figure image, used as the driven image, may also include a human face and torso. In conventional technology, the driving image resolution is 256*256, and only includes an image of the head area. In this embodiment, the driving image resolution is increased from 256*256 to 512*512, and the driving range is expanded from the head to the upper body or the entire body.

[0061] Specifically, a character image can be obtained from a local terminal or a computer device connected to the terminal. A corresponding driving video file can be generated in advance based on the voice signal and stored in the terminal or a computer device connected to the terminal, so that the driving video file corresponding to the voice signal can be obtained from the local terminal or a computer device connected to the terminal. In some embodiments, the terminal can also obtain the voice signal and generate a corresponding driving video file based on the voice signal.

[0062] S220. Input the character image and the driving video file into the video driving image model, extract the first facial key point information and the first torso key point information of the driving image, and extract the second facial key point information and the second torso key point information of the character image.

[0063] Among them, the video-driven image model can adopt the first-order motion model (FOMM). In this embodiment, the character image is expanded to the face part and the upper body part, and the resolution is increased from 256*256 to 512*512. For this purpose, a series of video data of 512*512 size (including the upper body) are collected, and the video-driven image model is trained based on FOMM. Specifically, the character image and the driving video file are input into the video-driven image model to extract key points, and the first face key point information and the first torso key point information of the driving image are obtained, as well as the second face key point information and the second torso key point information of the character image are obtained.

[0064] S230: Add the posture information contained in the driving image to the character image based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information to obtain several frames of new character image images.

[0065] Specifically, the distance between key points is evaluated based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information. The intermediate portion between the key points of the character image and the key points of the driving image is obtained to generate a reference frame. Motion information is obtained between the reference frame and each driving image and the character image. The posture information contained in each driving image is determined and appended to the character image to generate a plurality of new character image frames.

[0066] S240: Generate a character image video based on several frames of new character image and voice signals.

[0067] Specifically, a preliminary character image video without a voice signal can be generated by using several frames of new character image images, and the voice signal is synthesized with the preliminary character image video to obtain the character image video.

[0068] In the above-mentioned character image video generation method, a driving video file corresponding to a character image and a voice signal is obtained, wherein the driving image in the driving video file includes a face portion and a torso portion; the character image image and the driving video file are input into a video driving image model, and first facial key point information and first torso key point information of the driving image are extracted, as well as second facial key point information and second torso key point information of the character image image; thereby, posture information contained in the driving image is added to the character image based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information, to obtain several frames of new character image images; and finally, a character image video is generated based on the several frames of new character image images and the voice signal. This method achieves rapid expansion of character image driving and synthesis capabilities to different character images based on the character image and video driving image model, reducing the time and labor costs consumed in the video generation process. Furthermore, by rapidly expanding to unknown character images, it solves the technical problems of traditional text- or voice-based character driving and synthesis technologies, which have long time periods, high labor costs, and huge computing resource requirements for character driving and synthesis for unknown characters.

[0069] In one embodiment, Figure 3 As shown, obtaining the driving video file corresponding to the voice signal includes the following steps:

[0070] S310: Acquire speech signal features of the speech signal, and map the speech signal features to facial expression parameters of the 3D face model.

[0071] S320: Construct a facial image according to the facial expression parameters to obtain several frames of 3D facial images corresponding to the facial expression parameters.

[0072] S330 , rendering each 3D face image to obtain a 2D real face image corresponding to the 3D face image, and generating a driving video file according to each 2D real face image.

[0073] Among them, text can be converted into speech through TTS to obtain a speech signal. Specifically, first, the feature extraction of the speech signal is performed to obtain the speech signal features of the speech signal; then, the facial expression parameters of the 3D face model are predicted using the audio through the lip movement model; secondly, the 3D face model uses the facial expression parameters to reconstruct a 3D face image; finally, the reconstructed 3D face image is input into the GAN renderer to obtain a 2D real face image, and the 2D real face images are synthesized to generate a driving video file.

[0074] In one embodiment, Figure 4aAs shown, based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information, the posture information contained in the driving image is added to the character image to obtain several frames of new character image images, including:

[0075] S410: Perform a local affine transformation on the character image according to the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information to obtain a dense motion field image and an occlusion mask image.

[0076] S420 , extracting image features of the character image, performing feature fusion and decoding based on the image features, the dense motion field image, and the occlusion mask image, to obtain several frames of new character image images.

[0077] Specifically, if Figure 4b As shown, the video driven image model based on the first-order motion model can be divided into two parts: motion estimation and image generation. Among them, the motion estimation part has two outputs, one is the dense motion field (DenseMotion Field), which characterizes the mapping relationship between each key point in the driving image and the driven image. The other is the occlusion mask (Occlusion Mask), which shows which part of the driven image can be obtained by warping the driven image and which part can be obtained by image completion in the final generated image. For a picture input to the motion estimation module, the key point detector (Kepoint Detector) will output key point information and parameter information of the local affine transformation. These parameters enable the posture information near the key point to be obtained through affine transformation, and then the mapping between the target frame and the reference frame can be obtained through Taylor expansion. In the image generation module, the driven picture is input, and in the feature layer obtained by the encoder (Encoder), the dense motion field and occlusion mask are used for deformation, and then the final output is obtained by decoding (Decoder). It can be understood that Figure 4b The d layer in can represent a feature fusion layer.

[0078] In one embodiment, Figure 5 As shown, the video-driven image model is trained in the following way, including:

[0079] S510: Obtain a training sample set.

[0080] S520: Input the sample image and the training person image into the video-driven image model for portrait driving and synthesis. When the perceptual loss meets the model training stop condition, the training of the video-driven image model is completed.

[0081] The training sample set includes several sample images annotated with sample facial and torso keypoints. A perceptual loss employs a pyramidal model. The sample images serve as driving images, and the training person images serve as driven images. Specifically, during the training of the first-order motion model, the inputs are the driving image and the driven image. The pose information (including motion information) in the driving image is appended to the driven image, and the driven image becomes the result image. The output is a new image containing the pose of the driving image while preserving the object information of the driven image. If the two input images come from the same video, i.e., the same object information, the entire training process is a reconstruction of the driving image. Since the training process involves image reconstruction, a reconstruction loss is introduced, specifically a perceptual loss. Two images are processed separately through VGG19, and the feature values ​​of five layers are taken, followed by the difference and average, and then the sum is calculated. The perceptual loss employs a pyramidal model and is calculated at four different scales (e.g., the original image, 1 / 2 downsampled, 1 / 4 downsampled, and 1 / 8 downsampled). At the same time, in some implementations, equivariance constraints commonly used in unsupervised keypoint prediction are also introduced.

[0082] In one embodiment, Figure 6 As shown, the drive image is obtained in the following manner, including:

[0083] S610: Acquire text data, and input the text data into a text-driven model.

[0084] S620: Convert the text data into a speech signal through a text-to-speech conversion model.

[0085] S630: Input the speech signal features of the speech signal into the lip movement model to obtain facial expression parameters of the 3D face model.

[0086] S640: Control the 3D face model using facial expression parameters to obtain several frames of 3D face images.

[0087] S650: Input several frames of 3D face images into the GAN renderer for rendering to obtain a driving image.

[0088] Among them, the text-driven model includes a text-to-speech conversion model, a lip movement model, a 3D face model, and a GAN renderer. Specifically, the terminal can obtain text data and input the text data into the text-to-speech conversion model. The text data is converted into a speech signal through the text-to-speech conversion model. The speech signal features of the speech signal are extracted, and the speech signal features are input into the lip movement model. The speech signal features are processed by the lip movement model to obtain the facial expression parameters of the 3D face model. The facial expression parameters are used to control the 3D face model to obtain several frames of 3D face images. Several frames of 3D face images are input into the GAN renderer for rendering to obtain driving images.

[0089] In some embodiments, the 3D face model is constructed based on the Basel Face Model (BFM). The Basel Face Model is an open-source face database based on the 3DMM. Currently, there are two versions of the database (2009 and 2017).

[0090] In some embodiments, the lip movement model represents the input source audio as a high-dimensional feature, and then realizes the mapping between the high-dimensional feature of the audio and the facial expression parameters of the 3D face model. The model input is audio data and outputs the facial expression parameters of the 3D face model. The model first uses a long short-term memory (LSTM) neural network (ATVGnet) to extract the audio as a high-dimensional feature, and then uses a convolutional neural network to map the high-dimensional feature into a 64-dimensional vector, and the vector is represented as the facial expression parameter of the 3D face model. LSTM is a special recurrent neural network (RNN) that is mainly used to solve the problems of gradient vanishing and gradient exploding during long sequence training. Compared with ordinary RNN, LSTM can perform better in longer sequences and is very suitable for extracting information features in audio.

[0091] In some embodiments, the GAN renderer uses a pix2pixHD model, whose input data is a 3D face reconstructed using facial expression parameters and a real face, to achieve a mapping between the 3D face and the real face. In the inference stage, the input of the pix2pixHD model is a 3D face, and the output is a real face. pix2pixHD, as an improved version of pix2pix, is used to generate high-resolution images. The generator of pix2pixHD consists of two parts, G1 and G2, where G2 is further divided into two parts. The generators of G1 and pix2pix are no different, and are an end2end U-Net structure. The left half of G2 extracts features, adds and fuses the information with the features of the previous layer of the output layer of G1, and sends the fused information to the second half of G2 to output a high-resolution image. Compared with pix2pix, pix2pixHD uses multi-scale generators and discriminators to generate high-resolution images. The discriminator uses a multi-scale discriminator to perform discrimination at three different scales and average the results. The three scales used for discrimination are: original image, 1 / 2 downsampled image, and 1 / 4 downsampled image. Finer scales have smaller receptive fields and focus more on details. Coarser scales have larger receptive fields and focus more on global consistency.

[0092] In one embodiment, Figure 7a As shown, the text-driven model is trained in the following ways, including:

[0093] S710: Obtain a speech signal and several video frame character images in a training video file.

[0094] S720: Extracting video frame expression parameters from the character images in each video frame.

[0095] S730: Train a lip movement model based on the speech signal and the expression parameters of the video frame.

[0096] S740 , reconstructing a 3D face model according to the video frame expression parameters of the character image of each video frame to obtain several frames of 3D video frame face images.

[0097] S750: Train a GAN renderer based on the 3D video frame face image and the video frame character image.

[0098] Specifically, if Figure 7bAs shown, a training video file (such as video data) is obtained, and a voice signal and several video frame character images (such as video frame-face) are obtained from the training video file. The video frame expression parameters in each video frame character image are obtained based on the BFM model. The voice signal and the video frame expression parameters are input into the lip movement model, and the lip movement model is trained using the voice signal and the video frame expression parameters. The video frame expression parameters of each video frame character image are input into the 3D face model, and the video frame expression parameters of each video frame character image are reconstructed to obtain several frames of 3D video frame face images. The GAN renderer is trained based on the 3D video frame face images and the video frame character images.

[0099] In one embodiment, Figure 7c As shown, this method consists of two main modules. The first module is a text-driven 2D portrait driving and video synthesis module. This module can synthesize corresponding portrait videos based on text-driven driving. Its input is text and its output is a portrait video of the person announcing the text. The second module is a video-driven image module based on the first-order motion model (FOMM). This module inputs the portrait video generated by the first module as the driving video and any photo of a person, and outputs a video of the person being driven and synthesized. Unlike traditional methods, the same driving video can drive different person photos, enabling rapid driving and synthesis of multiple-image videos. Furthermore, the entire multi-image video driving and synthesis process requires only a piece of text and multiple person photos to quickly synthesize corresponding portrait videos for these characters. The specific workflow of the multi-image video driving and synthesis method is shown in the figure below. First, through the text-driven module, use text to drive and synthesize the required driving video; then, use the video generated by the above module as the driving video, select different character image photos as the driven pictures, and use the video-driven picture module to generate the corresponding character image video (without sound); finally, add audio to the driving result video to obtain the final video.

[0100] In one embodiment, a method for generating a character image video is provided. Figure 1 The following steps are used as an example to illustrate the terminal in the figure:

[0101] S802: Obtain a training sample set.

[0102] The training sample set includes several sample images, and the sample images are marked with sample face key points and sample torso key points.

[0103] S804: Input the sample image and the training person image into the video-driven image model for portrait driving and synthesis. When the perceptual loss meets the model training stop condition, the training of the video-driven image model is completed.

[0104] Among them, the perceptual loss adopts a pyramid model.

[0105] S806: Acquire text data, and input the text data into the text-driven model.

[0106] The text-driven model includes a text-to-speech model, a lip movement model, a 3D face model, and a GAN renderer. Specifically, the training process involves obtaining speech signals and several video frame character images from a video file; extracting expression parameters from each video frame character image; training the lip movement model based on the speech signals and expression parameters; reconstructing the 3D face model based on the expression parameters of each video frame character image to obtain several 3D video frame face images; and training the GAN renderer based on the 3D video frame face images and the video frame character images.

[0107] S808: Convert the text data into a speech signal through a text-to-speech conversion model.

[0108] S810: Input the speech signal features of the speech signal into the lip movement model to obtain facial expression parameters of the 3D face model.

[0109] S812: Control the 3D face model using facial expression parameters to obtain several frames of 3D face images.

[0110] S814: Input several frames of 3D face images into a GAN renderer for rendering to obtain several frames of driving images.

[0111] S816: Obtain the driving video file corresponding to the character image and voice signal.

[0112] The driving video file includes several frames of driving images, and the driving images include a face part and a torso part.

[0113] S818. Input the character image and the driving video file into the video driving image model, extract the first facial key point information and the first torso key point information of the driving image, and extract the second facial key point information and the second torso key point information of the character image.

[0114] S820: Perform a local affine transformation on the character image according to the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information to obtain a dense motion field image and an occlusion mask image.

[0115] S822: Extract image features of the character image, perform feature fusion and decoding based on the image features, the dense motion field image, and the occlusion mask image, and obtain several frames of new character image images.

[0116] S824: Generate a character image video based on several frames of new character image and voice signals.

[0117] It should be understood that, although the various steps in the above flow chart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above flow chart may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0118] In one embodiment, Figure 8 As shown, a device 800 for generating a character image video is provided, comprising: a first acquisition module 810, a key point extraction module 820, a posture information addition module 830 and a video generation module 840, wherein:

[0119] A first acquisition module 810 is configured to acquire a driving video file corresponding to a character image and a voice signal, wherein the driving video file includes a plurality of frames of driving images, each of which includes a face portion and a torso portion of the character;

[0120] A key point extraction module 820 is configured to input the character image and the driving video file into a video driving image model, extract first facial key point information and first torso key point information of the driving image, and extract second facial key point information and second torso key point information of the character image;

[0121] a posture information adding module 830 for adding the posture information contained in the driving image to the character image based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information, to obtain a plurality of frames of new character image images;

[0122] The video generation module 840 is used to generate a character image video based on the several frames of new character image and the voice signal.

[0123] In one embodiment, the first acquisition module 810 is also used to obtain the voice signal features of the voice signal, map the voice signal features to the facial expression parameters of the 3D face model; construct a facial image according to the facial expression parameters to obtain several frames of 3D face images corresponding to the facial expression parameters; render each of the 3D face images to obtain a 2D real face image corresponding to the 3D face image, and generate the driving video file according to each of the 2D real face images.

[0124] In one embodiment, the posture information addition module 830 is also used to perform a local affine transformation on the character image based on the first facial key point information, the first torso key point information, the second facial key point information and the second torso key point information to obtain a dense motion field image and an occlusion mask image; extract image features of the character image, and perform feature fusion and decoding based on the image features, the dense motion field image and the occlusion mask image to obtain the several frames of new character image images.

[0125] In one embodiment, the device also includes a driving image model training module for obtaining a training sample set, wherein the training sample set includes several sample images, and the sample images are marked with sample face key points and sample torso key points; the sample images and training character images are input into the video-driven image model for portrait driving and synthesis, and when the perceptual loss meets the model training stopping condition, the training of the video-driven image model is completed.

[0126] In one embodiment, the perceptual loss adopts a pyramid model.

[0127] In one embodiment, the first acquisition module 810 is also used to acquire text data and input the text data into a text-driven model, which includes a text-to-speech conversion model, a lip movement model, a 3D face model, and a GAN renderer; converting the text data into the speech signal through the text-to-speech conversion model; inputting the speech signal features of the speech signal into the lip movement model to obtain facial expression parameters of the 3D face model; using the facial expression parameters to control the 3D face model to obtain several frames of 3D face images; inputting the several frames of 3D face images into the GAN renderer for rendering to obtain the driven image.

[0128] In one embodiment, the device also includes a text-driven model training module, which is used to obtain a voice signal and several video frame character images in a training video file; extract video frame expression parameters in each of the video frame character images; train the lip movement model based on the voice signal and the video frame expression parameters; reconstruct the 3D face model based on the video frame expression parameters of each of the video frame character images to obtain several frames of 3D video frame face images; and train the GAN renderer based on the 3D video frame face images and the video frame character images.

[0129] The specific limitations of the character image video generation device can be found in the limitations of the character image video generation method described above and will not be repeated here. The various modules in the above-mentioned character image video generation device can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above-mentioned modules.

[0130] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for generating a character image video is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0131] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0132] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the method steps in the above embodiment when executing the computer program.

[0133] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method steps in the above embodiment are implemented.

[0134] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0135] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for generating a character image video, characterized in that: The method comprises: Acquire a driving video file corresponding to a character image and a voice signal, wherein the driving video file includes a plurality of frames of driving images, and the driving images include a face portion and a torso portion of the character; Inputting the character image and the driving video file into a video driving image model, extracting first facial key point information and first torso key point information of the driving image, and extracting second facial key point information and second torso key point information of the character image; performing distance evaluation between key points based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information, obtaining intermediate portions between key points of the character image and key points of the driving image to generate a reference frame, obtaining motion information between the reference frame and each of the driving images and the character image, determining posture information contained in each of the driving images, and appending the posture information contained in the driving image to the character image to obtain a plurality of frames of new character image images; A character image video is generated based on the several frames of new character image images and the voice signal.

2. The method according to claim 1, characterized in that Get the driving video file corresponding to the voice signal, including: Acquiring voice signal features of the voice signal, and mapping the voice signal features to facial expression parameters of a 3D face model; Constructing a facial image according to the facial expression parameters to obtain a plurality of frames of 3D facial images corresponding to the facial expression parameters; Rendering is performed on each of the 3D face images to obtain a 2D real face image corresponding to the 3D face image, and the driving video file is generated according to each of the 2D real face images.

3. The method according to claim 1, characterized in that The method includes: performing distance evaluation between key points based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information, obtaining an intermediate portion between the key points of the character image and the key points of the driving image to generate a reference frame, obtaining motion information between the reference frame and each of the driving images and the character image, determining posture information contained in each of the driving images, and appending the posture information contained in the driving image to the character image to obtain a plurality of frames of new character image images, including: performing a local affine transformation on the character image according to the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information to obtain a dense motion field image and an occlusion mask image; The image features of the character image are extracted, and feature fusion and decoding are performed based on the image features, the dense motion field image and the occlusion mask image to obtain the several frames of new character image images.

4. The method according to claim 1, wherein The video-driven image model is trained in the following manner, including: Obtaining a training sample set, wherein the training sample set includes a plurality of sample images, wherein the sample images are annotated with sample face key points and sample torso key points; The sample images and training character images are input into the video-driven image model for portrait driving and synthesis. When the perceptual loss meets the model training stop condition, the training of the video-driven image model is completed.

5. The method according to claim 4, characterized in that The perceptual loss adopts a pyramid model.

6. The method according to claim 1, characterized in that The drive image is obtained in the following manner, including: Acquire text data and input the text data into a text-driven model, wherein the text-driven model includes a text-to-speech model, a lip movement model, a 3D face model, and a GAN renderer; Converting the text data into the speech signal using the text-to-speech model; Inputting the speech signal features of the speech signal into the lip movement model to obtain facial expression parameters of the 3D face model; The 3D face model is controlled by using the facial expression parameters to obtain a plurality of frames of 3D face images; The several frames of 3D face images are input into a GAN renderer for rendering to obtain the driving image.

7. The method according to claim 6, characterized in that The text-driven model is trained in the following manner, including: Obtain speech signals and several video frame character images from a training video file; Extracting video frame expression parameters from the character images in each of the video frames; Training the lip movement model according to the speech signal and the video frame expression parameters; Reconstructing the 3D face model according to the video frame expression parameters of each of the video frame character images to obtain a plurality of 3D video frame face images; The GAN renderer is trained according to the 3D video frame face image and the video frame character image.

8. A device for generating a character image video, characterized in that: The device comprises: A first acquisition module is used to acquire a driving video file corresponding to a character image and a voice signal, wherein the driving video file includes a plurality of frames of driving images, and the driving images include a face portion and a torso portion of the character; A key point extraction module is used to input the character image and the driving video file into the video driving image model, extract the first facial key point information and the first torso key point information of the driving image; and extract the second facial key point information and the second torso key point information of the character image; a posture information addition module, configured to evaluate the distance between key points based on the first facial key point information, the first torso key point information, the second facial key point information, and the second torso key point information, obtain the intermediate portion between the key points of the character image and the key points of the driving image to generate a reference frame, obtain motion information between the reference frame and each of the driving images and the character image, determine the posture information contained in each of the driving images, and append the posture information contained in the driving image to the character image to obtain a plurality of frames of new character image images; The video generation module is used to generate a character image video based on the several frames of new character image and the voice signal.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images