Voice-driven face video generation method, system, storage medium, and electronic device

Abstract

The present application provides a method for generating a voice-driven face video, comprising: obtaining voice data and extracting audio features of the voice data; inputting the audio features, background image, and sampling points into a face generation model, and using the face generation model to generate a face speaking image corresponding to each frame of voice; splicing the face speaking images into a video, and superimposing the voice data to obtain a voice-driven face video generation video. The present application does not require independent three-dimensional modeling of the background image, and can greatly reduce the amount of calculation and the hardware cost of rendering, thereby meeting the real-time and low-cost requirements in the application scenario. The present application also provides a voice-driven face video generation system, storage medium, and electronic device, which have the above-mentioned beneficial effects.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, system, storage medium and electronic device for generating voice-driven face videos. Background Art

[0002] With the development of computer vision and natural language processing technologies, artificial intelligence has been widely used in the field of facial generation. Voice-driven facial video generation is a technology that generates realistic facial images corresponding to a given voice input. This technology eliminates the need for real-person presence and uses pre-trained models of specific individuals. Simply accepting voice input, it generates a corresponding speaking video of that person. This technology not only enhances entertainment experiences and artistic creation, but also improves human-computer interaction and educational training. It also has widespread application in the field of virtual livestreaming.

[0003] Thanks to its powerful generative capabilities, Generative Adversarial Networks (GANs) have been applied to the field of speech-driven speakers. GANs can learn the mapping from speech to facial images. Once trained, they can generate facial images end-to-end based on speech input. Because this two-dimensional method directly outputs the RGB values ​​of the target image based on speech, it only considers the results of a single frame and therefore lacks constraints on the three-dimensional consistency of the face. Summary of the Invention

[0004] The purpose of this application is to provide a voice-driven facial video generation method, system, storage medium and electronic device, which do not require three-dimensional modeling of the background, control image generation through style variables, and can generate high-quality images while retaining three-dimensional modeling information.

[0005] To solve the above technical problems, this application provides a method for generating voice-driven face videos. The specific technical solutions are as follows:

[0006] Acquiring voice data and extracting audio features of the voice data;

[0007] The audio features, background image, and sampling points are input into a face generation model, and the face generation model is used to generate a speaking face image corresponding to each frame of speech; the face generation model is composed of a rendering model and a generative adversarial model, and is used to downsample and encode the background image and combine it with the three-dimensional facial features extracted by the rendering model to obtain style variables; the generative adversarial model is used to generate the speaking face image based on the style variables;

[0008] The face speaking images are spliced ​​into a video, and the voice data is superimposed to obtain a voice-driven face video generation video.

[0009] Optionally, the process of generating the background image includes:

[0010] Acquire a training video, and extract the training video to obtain training data; the training data includes speech features and image features contained in the training video;

[0011] Cutting out a portion of a human face image from the training data; the cutout portion includes a human face area, a neck area, and a shoulder area;

[0012] Performing face segmentation on the partial face image using a face analysis model to obtain a face mask and a background mask; the face mask and the background mask have sizes consistent with those of the partial face image;

[0013] A background image after removing the face is obtained according to the background mask and the partial face image.

[0014] Optionally, intercepting a partial facial image from the training data includes:

[0015] Extracting original image frames from the training video at a set frame rate; the number of the original image frames is the same as the number of the audio frames;

[0016] Determining a facial image position in the original image frame;

[0017] A partial face image is obtained according to the position of the face image.

[0018] Optionally, after obtaining the face mask and background mask, the following steps are also included:

[0019] A foreground image containing only the face is obtained according to the face mask and the partial face image, and the foreground image is reduced to obtain a true value of the rendering model.

[0020] Optionally, after reducing the foreground image to obtain the true value of the rendering model, the method further includes:

[0021] Converting pixel coordinates in the partial face image into world coordinates in a world coordinate system;

[0022] Each pixel point is regarded as being rendered by a ray passing through the pixel point in space, and each ray is discretized to obtain the sampling point.

[0023] Optionally, converting pixel coordinates in the partial facial image into world coordinates in a world coordinate system includes:

[0024] Extracting two-dimensional key points from the partial face image;

[0025] Matching the two-dimensional key points with the three-dimensional key points of a three-dimensional standard face model to determine the face posture of each frame image;

[0026] Calculate the camera extrinsic parameter matrix corresponding to each frame of image according to the facial posture;

[0027] Assuming that the image observed by the camera is the true value, obtain the camera intrinsic parameter matrix;

[0028] The camera extrinsic parameter matrix and the camera intrinsic parameter matrix are applied to convert pixel coordinates in the partial face image into world coordinates in a world coordinate system.

[0029] Optionally, the rendering model includes:

[0030] The speech coding network consists of several convolutional blocks that extract audio features from the input audio and obtain speech embedding features;

[0031] A position encoding network is used to encode the three-dimensional position information of the sampling points to obtain position features;

[0032] A direction encoding network, configured to encode the directions of the sampling points to obtain direction codes; wherein the directions and codes of the sampling points on the same ray are the same;

[0033] a transparency prediction network, configured to output the color transparency of all the sampling points, and to concatenate the speech embedding feature and the position feature to obtain a first combined feature;

[0034] An RGB prediction network is used to output red, green and blue color values ​​for all the sampling points, and to concatenate the first combined feature and the direction code to obtain a second combined feature;

[0035] A rendering module is used to perform cumulative rendering on the sampling points on each ray to obtain the RGB color value corresponding to the two-dimensional pixel point corresponding to the ray; and to perform cumulative calculation on the three pixel values ​​of several sampling points on the ray in the ray direction to obtain an RGB predicted image;

[0036] A background encoding network is used to downsample and encode the background image to obtain two-dimensional background features;

[0037] A style encoding network is used to combine the two-dimensional background feature and the second combined feature to encode the style variable.

[0038] This application also provides a voice-driven face video generation system, including:

[0039] An audio feature extraction module, configured to obtain speech data and extract audio features of the speech data;

[0040] A face image generation module is configured to input the audio features, background image, and sampling points into a face generation model, and utilize the face generation model to generate a speaking face image corresponding to each frame of speech; the face generation model is based on a rendering model and a generative adversarial model, and is configured to downsample and encode the background image and combine it with the three-dimensional facial features extracted by the rendering model to obtain style variables; the generative adversarial model is configured to generate the speaking face image based on the style variables;

[0041] The video generation module is used to splice the face speaking images into a video and superimpose the voice data to obtain a voice-driven face video generation video.

[0042] The present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-described method when executed by a processor.

[0043] The present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned method when calling the computer program in the memory.

[0044] The present application provides a method for generating a speech-driven face video, comprising: obtaining speech data and extracting audio features of the speech data; inputting the audio features, a background image, and sampling points into a face generation model, and using the face generation model to generate a speaking face image corresponding to each frame of speech; the face generation model is composed of a rendering model and a generative adversarial model, and is used to downsample and encode the background image, and combine it with the three-dimensional facial features extracted by the rendering model to obtain style variables; the generative adversarial model is used to generate the speaking face image based on the style variables; the speaking face images are spliced ​​into a video, and the speech data is superimposed on it to obtain a speech-driven face video generation video.

[0045] This application uses a rendering model to extract three-dimensional features of the face and combines them with the encoded features of the background image. There is no need to independently perform three-dimensional modeling on the background image. At the same time, the three-dimensional features of the face are combined with the background features as style variables to control the image generation of the generative adversarial model. This not only retains the three-dimensional modeling information, but also utilizes the high-quality image generation capability of the generative adversarial model. At the same time, it can greatly reduce the amount of calculation and reduce the hardware cost of rendering, thereby meeting the real-time and low-cost requirements in the application scenario.

[0046] The present application also provides a voice-driven facial video generation system, storage medium and electronic device, which have the above-mentioned beneficial effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0048] Figure 1 A flowchart of a method for generating a voice-driven face video provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of the structure of a voice-driven face video generation system provided in an embodiment of the present application;

[0050] Figure 3 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0052] Please refer to Figure 1 , Figure 1 This is a flow chart of a method for generating a voice-driven face video provided in an embodiment of the present application, the method comprising:

[0053] S101: Acquire voice data and extract audio features of the voice data;

[0054] S102: Inputting the audio features, background image, and sampling points into a face generation model, and using the face generation model to generate a speaking face image corresponding to each frame of speech; the face generation model is composed of a rendering model and a generative adversarial model, and is used to downsample and encode the background image and combine it with the three-dimensional facial features extracted by the rendering model to obtain style variables; the generative adversarial model is used to generate the speaking face image based on the style variables;

[0055] S103: Splicing the human face speaking images into a video, and superimposing the voice data to obtain a voice-driven human face video generation video.

[0056] When generating a voice-driven face video, simply input the voice data into the face generation model to obtain a face speaking image. After splicing and combining the voice data, the voice-driven face video generation video is obtained. This embodiment assumes that the face generation model has been acquired or trained before executing step S102.

[0057] The following first describes the process of generating the background image provided by this application:

[0058] The first step is to obtain a training video and extract the training video to obtain training data; the training data includes voice features and image features contained in the training video;

[0059] Step 2: intercepting a portion of the face image from the training data; the intercepted portion includes the face area, the neck area, and the shoulder area;

[0060] Step 3: Use a face analysis model to perform face segmentation on the partial face image to obtain a face mask and a background mask; the sizes of the face mask and the background mask are consistent with those of the partial face image;

[0061] Step 4: obtaining a background image after removing the face according to the background mask and the partial face image.

[0062] Get speech features and image features from the pre-set training video as training data. First, extract the corresponding audio from the training video. You can use the pre-trained speech recognition model Wav2vector or DeepSpeech to extract features at 25 frames per second. The dimension is t. In order to make the speech features have a contextual correlation, take a window of w in the time dimension. The final extracted feature d a ∈R t×w , w represents the w consecutive frames before and after the current frame. When the front and back boundaries are encountered, they are filled with the value 0.

[0063] After that, image feature extraction is performed. First, the video is extracted at a frame rate of 25 to obtain the original image frames, which is consistent with the frame rate of the audio features. That is, the number of image frames and the number of audio frames are consistent. Then, the position of the face in the image is determined, and the upper body face image is cut from a fixed position in each original image. The image includes the face area, neck area, and shoulder area. The image size is then resized to 512 to obtain the partial face image I∈R 512×512×3 Use the face parsing model to segment I and obtain the face mask F and background mask B, which are the same size as I. According to the face mask F, the foreground image I containing only the face is obtained. c , the pixel values ​​in other areas are 0. According to the background mask B, the background image I with the face and neck removed is obtained. b , the pixel values ​​in other areas are 0.

[0064]

[0065] in Represents the multiplication of corresponding elements. For the foreground image I c Zoom out to get I m ∈R c×c×3 ,c<512, is used as the true value of the rendering model.,There is no limitation on which rendering model to use.,The following uses NeRF as an example for explanation.

[0066] Afterwards, 2D key points are extracted from the partial facial images and matched with the 3D key points of a standard 3D facial model to determine the facial pose for each frame. Based on the facial pose, the camera extrinsic parameter matrix corresponding to each frame is calculated. Assuming the image observed by the camera is the true value, the camera intrinsic parameter matrix is ​​obtained. The camera extrinsic and intrinsic parameter matrices are used to convert the pixel coordinates in the partial facial images into world coordinates in the world coordinate system.

[0067] Extract the 2D key points of the face from the partial face image I, match the 2D key points with the 3D key points of the 3D standard face model, and estimate the face posture of each frame image, thereby calculating the camera extrinsic parameter matrix corresponding to each frame image, and assuming that the image observed by the camera is the reduced image I m , in order to obtain the camera intrinsic parameter matrix. The extrinsic matrix and intrinsic matrix can transform the pixel coordinates in the image into the world coordinate system.

[0068] According to the NeRF rendering model, each pixel is rendered by a ray passing through the point in space, and each ray is discretized to obtain n points. Then I m The number of points in the space corresponding to all pixels in the image is nc 2 .

[0069] Construct a face generation model based on NeRF and GAN. b Downsampling and encoding are performed and combined with the 3D facial features rendered with NeRF, which serve as style variables to control the generative adversarial model to generate facial images. The NeRF rendering model consists of a speech encoding network, a position encoding network, a direction encoding network, a transparency prediction network, an RGB prediction network, and a rendering module. Taking the GAN network as an example, the generative adversarial model can include a background encoding network, a style encoding network, a decoding generation network, and a discriminator network.

[0070] The speech coding network consists of multiple convolution blocks in sequence, each of which consists of basic network structures such as one-dimensional convolution and nonlinear activation function. The step size of each convolution is set to 2, so that the audio feature d a ∈R t×wThe last dimension is halved until it is 1. After multiple convolution blocks, the first dimension of the input audio feature is changed from t to e. The final output speech embedding feature f a ∈R e .

[0071] Encode the three-dimensional positions of all sampling points. Use sine and cosine functions to encode the three coordinates of each point, using a total of n p Group frequency. An example of encoding the x-coordinate is as follows:

[0072]

[0073] in is the sine-cosine encoding feature, calculated as follows:

[0074]

[0075] The encoding method for y and z coordinates is the same as above, and the features f are obtained respectively. y , f z The final position encoding feature is the concatenation of the three:

[0076]

[0077] The directions of all sampling points are encoded. The directions of sampling points on the same ray are consistent and the encoding is also consistent. Similar to position encoding, the three coordinates of each ray direction are encoded using sine and cosine functions and spliced ​​with the original direction coordinates. A total of n d Group frequency, then the direction encoding feature is obtained:

[0078]

[0079] The transparency prediction network outputs color transparency for all sampling points: the speech embedding feature and the position feature are concatenated together to form the first combined feature [f a , f p The transparency prediction network consists of multiple MLP modules in sequence, and each MLP consists of basic network structures such as fully connected layers and nonlinear activation layers. Before the last MLP, the feature f is obtained. t The last MLP output module contains only fully connected layers, which t Mapped to transparency σ, dimension is 1.

[0080] The RGB prediction network outputs three RGB color values ​​for all sampling points: t and direction code f d Spliced ​​together, we get the second combined feature [f t , f dThe RGB prediction network is composed of multiple MLP modules in sequence, and each module consists of basic network structures such as fully connected layers and nonlinear activation layers. Before the last MLP, the feature f is obtained. c The last MLP output module contains only fully connected layers, which c Mapped to color value RGB, dimension is 3, recorded as C o .

[0081] The rendering module performs cumulative rendering on the sampling points on each ray to obtain the RGB color value C of the two-dimensional pixel point corresponding to the ray. m Each two-dimensional pixel on the image is passed by a ray emitted from the camera center O, which is represented as follows:

[0082] r(t)=O+td;

[0083] Where d∈R 3 is the direction of the ray, and t is the length of the ray. The color C of the two-dimensional pixel is determined by the near end t on the ray. n and remote t f Point C between o The rendering formula is as follows:

[0084]

[0085] Among them, σ(·) and C o (·) are the transparency prediction network and the RGB prediction network respectively. T(t) is the transparency accumulation in the ray direction, which is calculated as follows:

[0086]

[0087] Discretize the integral in the rendering formula, that is, at the n sampling points in the ray mentioned above, o The three pixel values ​​are accumulated and calculated in the ray direction, and finally the RGB prediction image I is obtained. n ∈R c×c×3 .

[0088] Background encoding network for background image I b Downsampling encoding: It consists of multiple convolution blocks, each of which includes basic network structures such as convolution, batch normalization function and nonlinear activation function. By setting the number of convolution kernels and the convolution step size, the number of feature maps is changed and the size of the feature maps is reduced. Finally, the feature where n b is the number of channels, height and width are both c, and the same as the rendering image I n The same size.

[0089] The style encoding network transforms the two-dimensional background feature f b and rendering feature fc Combined together, the style variables required by StyleGAN are encoded: the f obtained by a single sampling point is recorded c The number of feature channels is n c , similar to the above mentioned RGB three-channel rendering, in f c Feature channels are rendered separately to obtain features Put f b and F c Spliced ​​together to form new features as network input:

[0090]

[0091] The style encoding network consists of two parts. The first part contains multiple consecutive convolution blocks. Each convolution block includes basic network structures such as convolution, batch normalization function and nonlinear activation function. By setting the convolution step size to continuously reduce F m The size is from c to 4. Only in the first convolution block the number of channels is changed to 512, and the rest remain unchanged. Therefore, after the first part, the feature f is finally obtained. s ∈R 4×4×512 , and then f s The flattened one-dimensional features are 8192 and used as the input of the second part of the network. The second part of the network consists of multiple MLP modules, each of which consists of a fully connected layer and a nonlinear activation layer, and the features are downsampled by 16 times. After the second part, the style variable F is finally obtained. s ∈R 512 .

[0092] The decoding generation network uses the pre-trained StyleGAN decoder with F s As input, the final RGB image I is generated g ∈R 512×512×3 .

[0093] The discriminator network uses the StyleGAN discriminator network, inputs a 512-size RGB image, and outputs the probability value of the image being a real image.

[0094] Set the network loss function and train the network to update the network parameters.

[0095] For 3D rendering, supervision is required. n Generated, its rendering loss function is:

[0096] L n =Per(I m , I n )+L1(I m , I n );

[0097] Where Per(·) is the perceptual loss function, L1(·) is the mean absolute error loss function, and I n Generated for rendering, I m is a true value.

[0098] For the generation network of the GAN network, supervision is required. g Generate, its generation loss function is:

[0099] L g =Per(I,I g )+L1(I,I g );

[0100] Among them I g The result generated by the StyleGAN network, I is the true value.

[0101] Put I and I g Input the discriminant network, and its output is D I , D g .

[0102] For the generative network, the total loss is the generation loss plus the discrimination loss, as follows:

[0103] LG=L g +BCE(1,D g );

[0104] Among them, BCE(·) is the cross entropy function.

[0105] For the discriminant network of the GAN network, its loss function is:

[0106] L D =BCE(0, D g )+BCE(1,D I );

[0107] The network optimizer can use the Adam optimizer to continuously optimize network parameters based on the loss function. The decoder uses pre-trained parameters and is not updated during the entire training process. Training stops when the loss function no longer significantly decreases.

[0108] Generate a speaking video using the trained model. Get a speech as the driving signal, first extract the audio features of the input speech, and compare it with the pre-processed background image I b The training model then generates a speaker image for each frame of speech. The images are then stitched together into a video and combined with the speech data to create the final speech-driven face video.

[0109] This application uses a rendering model to extract three-dimensional facial features and combines them with the encoded features of the background image, eliminating the need for independent three-dimensional modeling of the background image. At the same time, the three-dimensional facial features are combined with the background features as style variables to control the image generation of the generative adversarial model. This not only retains the three-dimensional modeling information, but also utilizes the high-quality image generation capabilities of the generative adversarial model. When using low-resolution NeRF to obtain three-dimensional features and then using the GAN network for super-resolution, the amount of computation can be greatly reduced, reducing the hardware cost of rendering, thereby meeting the real-time and low-cost requirements of the application scenario.

[0110] The following is an introduction to the voice-driven face video generation system provided by the embodiment of the present application. The voice-driven face video generation system described below and the voice-driven face video generation method described above can be referenced to each other. Figure 2 , Figure 2 This is a schematic diagram of the structure of a voice-driven face video generation system provided in an embodiment of the present application, the system comprising:

[0111] An audio feature extraction module, configured to obtain speech data and extract audio features of the speech data;

[0112] A face image generation module is configured to input the audio features, background image, and sampling points into a face generation model, and utilize the face generation model to generate a speaking face image corresponding to each frame of speech; the face generation model is based on a rendering model and a generative adversarial model, and is configured to downsample and encode the background image and combine it with the three-dimensional facial features extracted by the rendering model to obtain style variables; the generative adversarial model is configured to generate the speaking face image based on the style variables;

[0113] The video generation module is used to splice the face speaking images into a video and superimpose the voice data to obtain a voice-driven face video generation video.

[0114] Based on the above embodiment, as a preferred embodiment, the system includes:

[0115] A background image generation module is used to obtain a training video and extract the training video to obtain training data; the training data includes the voice features and image features contained in the training video; a partial face image is captured from the training data; the captured area includes the face area, neck area and shoulder area; the partial face image is segmented using a face analysis model to obtain a face mask and a background mask; the size of the face mask and the background mask are consistent with the size of the partial face image; and the background image after removing the face is obtained based on the background mask and the partial face image.

[0116] Based on the above embodiment, as a preferred embodiment, the background image generation module includes:

[0117] A face capture unit is used to extract original image frames from the training video at a set frame rate; the number of frames of the original image frames is the same as the number of audio frames; determine the position of the face image in the original image frame; and capture a partial face image based on the face image position.

[0118] Based on the above embodiment, as a preferred embodiment, the background image generation module may further include:

[0119] A true value data generating unit is used to obtain a foreground image containing only the face according to the face mask and the partial face image, and to reduce the foreground image to obtain the true value of the rendering model.

[0120] Based on the above embodiment, as a preferred embodiment, the background image generation module may further include:

[0121] A world coordinate generation unit, configured to convert pixel coordinates in the partial face image into world coordinates in a world coordinate system;

[0122] The sampling point generating unit is configured to regard each pixel point as being rendered by a ray passing through the pixel point in space, and discretize each ray to obtain the sampling point.

[0123] Based on the above embodiment, as a preferred embodiment, the world coordinate generation unit is a unit for performing the following steps:

[0124] Extract two-dimensional key points from the partial facial image; match the two-dimensional key points with three-dimensional key points of a three-dimensional standard facial model to determine the facial posture of each frame; calculate a camera extrinsic parameter matrix corresponding to each frame according to the facial posture; assume that the image observed by the camera is the true value, and obtain a camera intrinsic parameter matrix; apply the camera extrinsic parameter matrix and the camera intrinsic parameter matrix to convert pixel coordinates in the partial facial image into world coordinates in a world coordinate system.

[0125] Based on the above embodiment, as a preferred embodiment, the rendering model includes:

[0126] The speech coding network consists of several convolutional blocks that extract audio features from the input audio and obtain speech embedding features;

[0127] A position encoding network is used to encode the three-dimensional position information of the sampling points to obtain position features;

[0128] A direction encoding network, configured to encode the directions of the sampling points to obtain direction codes; wherein the directions and codes of the sampling points on the same ray are the same;

[0129] a transparency prediction network, configured to output the color transparency of all the sampling points, and to concatenate the speech embedding feature and the position feature to obtain a first combined feature;

[0130] An RGB prediction network is used to output red, green and blue color values ​​for all the sampling points, and to concatenate the first combined feature and the direction code to obtain a second combined feature;

[0131] A rendering module is used to perform cumulative rendering on the sampling points on each ray to obtain the RGB color value corresponding to the two-dimensional pixel point corresponding to the ray; and to perform cumulative calculation on the three pixel values ​​of several sampling points on the ray in the ray direction to obtain an RGB predicted image;

[0132] A background encoding network is used to downsample and encode the background image to obtain two-dimensional background features;

[0133] A style encoding network is used to combine the two-dimensional background feature and the second combined feature to encode the style variable.

[0134] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, can implement the steps provided in the above embodiments. The storage medium may include: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0135] This application also provides an electronic device, see Figure 3 , a structural diagram of an electronic device provided in an embodiment of the present application, such as Figure 3 As shown, a processor 1410 and a memory 1420 may be included.

[0136] The processor 1410 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1410 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1410 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1410 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1410 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0137] The memory 1420 may include one or more computer-readable storage media, which may be non-transitory. The memory 1420 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 1420 is at least used to store the following computer program 1421, wherein, after the computer program is loaded and executed by the processor 1410, it can implement the relevant steps in the method performed by the electronic device side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 1420 may also include an operating system 1422 and data 1423, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 1422 may include Windows, Linux, Android, etc.

[0138] In some embodiments, the electronic device may further include a display screen 1430 , an input / output interface 1440 , a communication interface 1450 , a sensor 1460 , a power supply 1470 , and a communication bus 1480 .

[0139] certainly, Figure 3 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include Figure 3 More or fewer components than shown, or combinations of certain components.

[0140] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems provided in the embodiments, since they correspond to the methods provided in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0141] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core ideas of this application. It should be noted that for those skilled in the art, without departing from the principles of this application, various improvements and modifications can be made to this application, and such improvements and modifications also fall within the scope of protection of this application.

[0142] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A method for generating a voice-driven face video, characterized in that: include: Acquiring voice data and extracting audio features of the voice data; Inputting the audio features, background image and sampling points into a face generation model, and using the face generation model to generate a speaking face image corresponding to each frame of speech; The face generation model is composed of a rendering model and a generative adversarial model, and is used to downsample and encode the background image and combine it with the three-dimensional facial features extracted by the rendering model to obtain style variables; the generative adversarial model is used to generate the speaking face image based on the style variables; The face speaking images are spliced ​​into a video, and the voice data is superimposed to obtain a voice-driven face video generation video.

2. The method for generating a voice-driven face video according to claim 1, wherein: The process of generating the background image includes: Acquire a training video, and extract the training video to obtain training data; the training data includes speech features and image features contained in the training video; Cutting out a portion of a human face image from the training data; the cutout portion includes a human face area, a neck area, and a shoulder area; Performing face segmentation on the partial face image using a face analysis model to obtain a face mask and a background mask; the face mask and the background mask have sizes consistent with those of the partial face image; A background image after removing the face is obtained according to the background mask and the partial face image.

3. The method for generating a voice-driven face video according to claim 2, wherein: The partial facial images intercepted from the training data include: Extracting original image frames from the training video at a set frame rate; the number of the original image frames is the same as the number of the audio frames; Determining a facial image position in the original image frame; A partial face image is obtained according to the position of the face image.

4. The method for generating a voice-driven face video according to claim 2, wherein: After obtaining the face mask and background mask, it also includes: A foreground image containing only the face is obtained according to the face mask and the partial face image, and the foreground image is reduced to obtain a true value of the rendering model.

5. The method for generating a voice-driven face video according to claim 4, wherein: After reducing the foreground image to obtain the true value of the rendering model, the method further includes: Converting pixel coordinates in the partial face image into world coordinates in a world coordinate system; Each pixel point is regarded as being rendered by a ray passing through the pixel point in space, and each ray is discretized to obtain the sampling point.

6. The method for generating a voice-driven face video according to claim 5, wherein: Converting pixel coordinates in the partial face image into world coordinates in a world coordinate system includes: Extracting two-dimensional key points from the partial face image; Matching the two-dimensional key points with the three-dimensional key points of a three-dimensional standard face model to determine the face posture of each frame image; Calculate the camera extrinsic parameter matrix corresponding to each frame of image according to the facial posture; Assuming that the image observed by the camera is the true value, obtain the camera intrinsic parameter matrix; The camera extrinsic parameter matrix and the camera intrinsic parameter matrix are applied to convert pixel coordinates in the partial face image into world coordinates in a world coordinate system.

7. The method for generating a voice-driven face video according to claim 5, wherein: The rendering model includes: The speech coding network consists of several convolutional blocks that extract audio features from the input audio and obtain speech embedding features; A position encoding network is used to encode the three-dimensional position information of the sampling points to obtain position features; A direction encoding network, configured to encode the directions of the sampling points to obtain direction codes; wherein the directions and codes of the sampling points on the same ray are the same; a transparency prediction network, configured to output the color transparency of all the sampling points, and to concatenate the speech embedding feature and the position feature to obtain a first combined feature; An RGB prediction network is used to output red, green and blue color values ​​for all the sampling points, and to concatenate the first combined feature and the direction code to obtain a second combined feature; A rendering module is used to perform cumulative rendering on the sampling points on each ray to obtain the RGB color value corresponding to the two-dimensional pixel point corresponding to the ray; and to perform cumulative calculation on the three pixel values ​​of several sampling points on the ray in the ray direction to obtain an RGB predicted image; A background encoding network is used to downsample and encode the background image to obtain two-dimensional background features; A style encoding network is used to combine the two-dimensional background feature and the second combined feature to encode the style variable.

8. A voice-driven face video generation system, characterized in that: include: An audio feature extraction module, configured to obtain speech data and extract audio features of the speech data; A face image generation module, configured to input the audio features, background image, and sampling points into a face generation model, and generate a speaking face image corresponding to each frame of speech using the face generation model; The face generation model is composed of a rendering model and a generative adversarial model, and is used to downsample and encode the background image and combine it with the three-dimensional facial features extracted by the rendering model to obtain style variables; the generative adversarial model is used to generate the speaking face image based on the style variables; The video generation module is used to splice the face speaking images into a video and superimpose the voice data to obtain a voice-driven face video generation video.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the voice-driven facial video generation method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the voice-driven face video generation method according to any one of claims 1 to 7 are implemented.

Images