A method for speech-driven digital human face video generation based on style vector

Through the combination of style vectors and audio features, the use of StyleGAN generators has solved the problem of information loss and large amount of calculations generated by audio-driven videos in the prior art, and achieved high-quality and diverse speaker video generation, which is suitable for a variety of Application scenario.

CN118632086BActive Publication Date: 2024-11-12HUBEI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410775608.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-17
Publication Date
2024-11-12
Estimated Expiration
2044-06-17

AI Technical Summary

Technical Problem

The existing audio-driven speaker video generation methods have problems such as information loss, information semantic mismatch, large calculation amount, inability to generalize, and cumbersome perspective training requirements, making it difficult to generate natural and high-quality speaker videos.

Method used

Using a speech-driven digital human facial video generation method based on style vectors, the style vector and audio features are extracted through the image and audio encoding module, combined with the image generation and video encoding module, generate high-quality speaker head video, and use StyleGAN as the generator , combining deep learning and computer vision technology.

Benefits of technology

It realizes high-quality and realistic speaking video generation, enhances the expressiveness and diversity of digital human technology, and is suitable for virtual education, virtual actors and other fields, reducing computing costs and perspective training needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118632086B_ABST
    Figure CN118632086B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for generating a digital human face video based on a voice-driven style vector, and the method comprises the following steps: step S1: collecting a data set; step S2: algorithm design, the overall network structure consists of four parts, including a video decoupling module, an image and audio encoding module, a feature coupling module, and an image generation and video encoding module; step S3: training a data set; step S4: testing a data set. The present invention obtains a style vector and an audio feature through an image and audio encoding module, combines an image generation and video encoding module, and couples the style vector and the audio feature to generate a high-quality speaker face video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for generating a voice-driven digital human face video based on a style vector. Background Art

[0002] Audio-driven speaking portrait synthesis is an important and challenging problem with multiple potential applications, such as digital characters, virtual avatars, filmmaking, and video conferencing. In recent years, many researchers have used deep generative models to tackle this task.

[0003] There are still many challenges in generating a natural-looking talking head video, mainly because different pronunciations correspond to different movements. Lip movements are the most important for pronunciation, followed by head movements and blinking movements.

[0004] Most existing methods use some intermediate face representation, including reconstructing explicit 3D face shape, regressing expression coefficients or 2D key points. Since the intermediate representation causes information loss, it may cause semantic mismatch between the original audio signal and the face deformation to be learned. Moreover, existing audio-driven methods have several limitations, such as only rendering the mouth area and fixing static head posture, which makes them unsuitable for advanced speaking face editing tasks.

[0005] When generating images based on NeRF (dynamic radiation field), each pixel requires nearly 200 forward predictions from the MLP deep model. Although the scale of a single calculation is not large, the amount of calculation required to render the entire image is still considerable, and NeRF takes a long time to train for each scene. NeRF cannot be directly extended to scenes that have not been seen before, which is obviously contrary to the goal of pursuing generalization. Although the NeRF method can achieve excellent perspective synthesis effects, it requires a large number (hundreds) of perspectives for training, which limits its application in frontal videos only.

[0006] Therefore, it is urgent to design a voice-driven digital human facial video generation method based on style vector to solve the problems existing in the above-mentioned prior art. Summary of the invention

[0007] In view of the above defects or improvement needs of the prior art, the present invention provides a voice-driven digital human facial video generation method based on style vector, the purpose of which is to obtain style vectors and audio features through image and audio encoding modules, combine image generation and video encoding modules, couple the style vectors and audio features to generate high-quality speaking head video.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] The first aspect of the present invention provides a method for generating a voice-driven digital human face video based on a style vector, the method comprising the following steps:

[0010] Step S1: Collecting data sets;

[0011] Step S2: Algorithm design, the overall network structure consists of four parts, including video decoupling module, image and audio encoding module, feature coupling module, image generation and video encoding module;

[0012] Step S3: training the data set; specifically comprising the following steps:

[0013] S31: extracting information from the collected data set through a video decoupling module to generate an image sequence and an audio sequence;

[0014] S32: Processing the generated image sequence and audio sequence through the image and audio encoding module, and outputting the image style vector and audio features;

[0015] S33: splicing the style vector and the audio feature through a feature coupling module to obtain a joint feature representation including audio and image style information;

[0016] S34: processing the joint feature representation including the audio and image style information through the image generation and video encoding module, and finally integrating the image sequence and the audio information to generate a final video;

[0017] Step S4: testing the data set.

[0018] As an embodiment of the present application, the data set collected in step S1 is the LRS2 data set, which is a lip reading video from the BBC and contains more than 40,000 spoken sentences. It is divided into training set, validation set and test set according to the broadcast date, and the CMLR data set is used for generalization training.

[0019] As an embodiment of the present application, the data set is collected in step S1 by collecting various video data, setting the video frame rate to 25fps, the audio sampling rate to 16k, and the length of the audio block corresponding to one frame of the video is 16.

[0020] As an embodiment of the present application, the video decoupling module in step S31 extracts an image sequence containing a face through face recognition and cropping algorithms, and extracts audio information through ffmpeg to generate an audio sequence.

[0021] As an embodiment of the present application, the image and audio encoding module in step S32 includes an image encoder and an audio encoder. The image encoder can directly encode the real image into the latent space, extract image features at different levels through the feature pyramid and encode them into a special style vector. The input is an image sequence and the output is the style vector of the image.

[0022] As an embodiment of the present application, the audio encoder is composed of several network blocks consisting of two-dimensional convolution blocks, batch normalization and Relu activation functions. A speech segment S of size Ta×D is input into the audio encoder, where Ta is the time step of the audio, D is the number of audio slices, and finally the audio features are output.

[0023] As an embodiment of the present application, in step S33, the feature coupling module concatenates the audio features with the 18 style vectors respectively to obtain a joint feature representation that includes audio and image style information.

[0024] As an embodiment of the present application, the image generation and video encoding module in step S34 includes an image generation module and a video encoding module. The image generation module uses StyleGAN as a generator, and the video encoding uses ffmpeg for video synthesis. Using the above-mentioned comprehensive feature representation, the decoder generates an image sequence, and the video encoding integrates the image sequence and audio information to generate the final video.

[0025] As an embodiment of the present application, the loss function designed in the process of training the data set in step S3 includes: Loss function, Loss function, Loss function, Loss function, Loss function, the loss function calculation formula is:

[0026]

[0027] in, The loss function is used to calculate the error between the real image and the predicted image. The loss function calculates the perceptual loss between the true value and the predicted value; in order to ensure that the input and output are human faces, The loss function uses the ArcFace network to measure the cosine similarity between the input and output; The loss function regularizes the decoder. The loss function uses the syncnet network to calculate the matching degree between the lip shape and the speech; , , , , are the weights of the corresponding loss functions respectively.

[0028] As an embodiment of the present application, the step S4 of testing the data set specifically includes: obtaining the lip synchronization accuracy through the syncnet network for the input speaker face and voice; combining The loss function obtains the perceptual loss to judge the face realism and finally outputs the speaking video.

[0029] The beneficial effects of the present invention are:

[0030] 1. The present invention can directly encode real images into latent space through an image encoder without the need for additional online optimization steps, obtain the style vector of the image, combine the image generation and video encoding modules, use StyleGAN as a generator, and generate high-quality speaking face videos after coupling the style vector and audio features, thereby enhancing the expressiveness and realism of digital human technology to better support the application of digital humans in virtual education, virtual actors, games and other fields; in addition, this method combines technologies such as deep learning, computer vision and signal processing, and provides new ideas and methods for interdisciplinary research.

[0031] 2. The present invention uses StyleGAN as a generator in the image generation and video encoding modules. StyleGAN has a good decoupling property for the style vector space and can be used to control and edit images. It is very suitable for solving multimodal problems. Therefore, it has a good effect on voice-driven digital human generation and can generate high-resolution and realistic face images, which makes the generated digital human's performance in the video more realistic and can accurately reflect the emotions conveyed by the voice.

[0032] 3. The style vector obtained by the image encoder of the present invention can capture the personality characteristics and style of the character, so the generated digital human is no longer a simple template image; the adjustment of the style vector can realize flexible control of the appearance and expression style of the generated digital human, so that the generated video can have greater diversity and customizability; such flexibility enables the generated digital human to adapt to different scenarios and needs, improving its practicality and scope of application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flow chart of a method for generating a voice-driven digital human face video based on a style vector provided in an embodiment of the present invention;

[0034] Figure 2 An algorithm framework diagram of a method for generating digital human facial videos driven by voice based on style vectors provided in an embodiment of the present invention;

[0035] Figure 3A structural diagram of an image encoding module of a method for generating a digital human face video driven by a voice based on a style vector provided in an embodiment of the present invention;

[0036] Figure 4 A structural diagram of StyleGAN in an image generation and video encoding module of a voice-driven digital human face video generation method based on a style vector provided in an embodiment of the present invention;

[0037] Figure 5 An LRS2 data set diagram of a method for generating digital human facial videos driven by voice based on style vectors provided in an embodiment of the present invention;

[0038] Figure 6 This is a CMLR data set diagram of a style vector-based voice-driven digital human facial video generation method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0041] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0042] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing in the full text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or a scheme that satisfies both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0043] Reference Figure 1-Figure 6 The first aspect of the present invention provides a method for generating a voice-driven digital human face video based on a style vector, the method comprising the following steps:

[0044] Step S1: Collecting data sets;

[0045] Step S2: Algorithm design, the overall network structure consists of four parts, including video decoupling module, image and audio encoding module, feature coupling module, image generation and video encoding module;

[0046] Step S3: training the data set; specifically comprising the following steps:

[0047] S31: extracting information from the collected data set through a video decoupling module to generate an image sequence and an audio sequence;

[0048] S32: Processing the generated image sequence and audio sequence through the image and audio encoding module, and outputting the image style vector and audio features;

[0049] S33: splicing the style vector and the audio feature through a feature coupling module to obtain a joint feature representation including audio and image style information;

[0050] S34: processing the joint feature representation including the audio and image style information through the image generation and video encoding module, and finally integrating the image sequence and the audio information to generate a final video;

[0051] Step S4: testing the data set.

[0052] As an embodiment of the present application, Figure 5 and Figure 6As shown, the data set collected in step S1 is the LRS2 data set, which is a lip reading video from the BBC and contains more than 40,000 spoken sentences. The training set, validation set and test set are divided according to the broadcast date. In addition, in order to solve the accuracy of Chinese lip reading synthesis, the CMLR data set is used for generalization training.

[0053] As another embodiment of the present application, the data set collected in step S1 can also be collected by the user himself to collect various video data, the video frame rate is set to 25fps, the audio sampling rate is 16k, and the length of the audio block corresponding to one frame of the video is 16.

[0054] As an embodiment of the present application, the video decoupling module in step S31 extracts an image sequence containing a face through face recognition and cropping algorithms, extracts audio information through ffmpeg to generate an audio sequence, slices each image sequence into five consecutive images, and splices the full face and the image without the lower half of the face by channel to obtain an image with a size of (6*96*96).

[0055] As an embodiment of the present application, Figure 3 As shown, the image and audio encoding module in step S32 includes an image encoder (Image_Encoder) and an audio encoder (Audio_Encoder). The image encoder (Image_Encoder) is an encoder that can directly encode a real image into a latent space, extract image features at different levels through a feature pyramid and encode them into a special style vector. The input is an image sequence, and the output is the image style vector Image_Features (image features).

[0056] Specifically, the traditional image editing method follows the principle of "invert first, edit later", that is, first obtain the latent code of the image in the W space, then edit the latent code, and then generate a new image through Gan; however, encoding the real image into the W space cannot accurately reconstruct the input image, so there is a method of encoding the real image into the W+ space. Although the reconstruction result is more accurate, the optimization time required is longer; the image encoder (Image_Encoder) used in this application can also directly encode the input image into the W+ space without the need for additional online optimization steps.

[0057] In addition, the style vector obtained by the image encoder of the present invention can capture the personality characteristics and style of the character, so the generated digital human is no longer a simple template image; the adjustment of the style vector can realize flexible control of the appearance and expression style of the generated digital human, so that the generated video can be made more diverse and customizable later; such flexibility enables the generated digital human to adapt to different scenarios and needs, improving its practicality and scope of application.

[0058] As an embodiment of the present application, the audio encoder (Audio_Encoder) is composed of several network blocks (blocks) composed of two-dimensional convolution blocks, batch normalization and Relu activation functions. A speech segment S of size Ta×D is input into the audio encoder, where Ta is the time step of the audio, D is the number of audio slices, and finally audio features (Audio_Features) are output.

[0059] As an embodiment of the present application, in step S33, the feature coupling module concatenates and convolves the audio features (Audio_Features) with the 18 style vectors Image_Features (image features) to obtain a joint feature representation that includes audio and image style information.

[0060] As an embodiment of the present application, Figure 4 As shown, the image generation and video encoding module in step S34 includes an image generation module and a video encoding module. The image generation module uses stylegan as a generator, and the video encoding uses ffmpeg for video synthesis. Using the above-mentioned comprehensive feature representation, the decoder generates an image sequence, and the video encoding integrates the image sequence and audio information to generate the final video.

[0061] Specifically, the present invention uses StyleGAN as a generator in the image generation and video encoding modules. StyleGAN has a good decoupling property for the style vector space, can be used to control and edit images, and is very suitable for solving multimodal problems. Compared with the NeRF-based method, the method based on StyleGAN inversion has greatly reduced space and time consumption, so it has a better effect on voice-driven digital human generation, and can generate high-resolution, realistic facial images, which makes the generated digital human's performance in the video more realistic and can accurately reflect the emotions conveyed by the voice.

[0062] As an embodiment of the present application, the loss function designed in the process of training the data set in step S3 includes: Loss function, Loss function, Loss function, Loss function, Loss function, the loss function calculation formula is:

[0063]

[0064] in, The loss function is used to calculate the error between the real image and the predicted image. The loss function calculates the perceptual loss between the true value and the predicted value; in order to ensure that the input and output are as close to human faces as possible, The loss function uses the ArcFace network to measure the cosine similarity between the input and output; Regularize the decoder. The loss function uses the syncnet network to calculate the matching degree between the lip shape and the speech; , , , , are the weights of the corresponding loss functions, and the parameters can be adjusted in actual operation.

[0065] As an embodiment of the present application, the step S4 of testing the data set specifically includes: obtaining the lip synchronization accuracy through the syncnet network for the input speaker face and voice; combining The loss function obtains the perceptual loss to judge the face realism and finally outputs the speaking video.

[0066] The present invention can generate a speaker video that is highly consistent with the audio and portrait by inputting the portrait and audio, and super-resolution the output video to generate a higher quality video. Its main purpose is to enhance the expressiveness and realism of digital human technology to better support the application of digital humans in virtual education, virtual actors, games and other fields; in addition, this method combines deep learning, computer vision and signal processing technologies, providing new ideas and methods for interdisciplinary research.

[0067] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for generating digital human face videos driven by speech based on style vectors, It is characterized in that The method comprises the following steps: Step S1: Collecting data sets; Step S2: Algorithm design, the overall network structure consists of four parts, including video decoupling module, image and audio encoding module, feature coupling module, image generation and video encoding module; Step S3: training the network; specifically comprising the following steps: S31: extracting information from the collected data set through a video decoupling module to generate an image sequence and an audio sequence; S32: Processing the generated image sequence and audio sequence through the image and audio encoding module, and outputting the image style vector and audio features; S33: splicing the style vector and the audio feature through a feature coupling module to obtain a joint feature representation including audio and image style information; S34: processing the joint feature representation including the audio and image style information through the image generation and video encoding module, and finally integrating the image sequence and the audio sequence to generate a final video; Step S4: testing the network; The image and audio encoding module in step S32 includes an image encoder and an audio encoder. The image encoder can directly encode the real image into the latent space, extract image features at different levels through the feature pyramid and encode them into a special style vector. The input is an image sequence and the output is the style vector of the image. The image generation and video encoding module in step S34 includes an image generation module and a video encoding module. The image generation module uses StyleGAN as a generator, and the video encoding module uses ffmpeg for video synthesis. Using the above-mentioned joint feature representation, the generator generates an image sequence, and the video encoding module integrates the image sequence and the audio sequence to generate the final video.

2. A method for generating digital human face video driven by voice based on style vector according to claim 1, It is characterized in that The data set collected in step S1 is the LRS2 data set, which is a lip reading video from the BBC and contains more than 40,000 spoken sentences. The training set, validation set and test set are divided according to the broadcast date, and the CMLR data set is used for generalization training.

3. The method for generating digital human face video driven by voice based on style vector according to claim 1, It is characterized in that The data set is collected in step S1 by collecting various video data, setting the video frame rate to 25fps, the audio sampling rate to 16KHz, and the length of the audio block corresponding to one frame of the video to 16.

4. The method for generating digital human face video driven by voice based on style vector according to claim 1, It is characterized in that The video decoupling module in step S31 extracts an image sequence containing a face through face recognition and cropping algorithms, and extracts audio information through ffmpeg to generate an audio sequence.

5. The method for generating digital human face video based on voice-driven style vector according to claim 1, It is characterized in that The audio encoder consists of several network blocks composed of two-dimensional convolution blocks, batch normalization and Relu activation functions. A speech segment S of size Ta×D is input into the audio encoder, where Ta is the time step of the audio and D is the number of audio slices, and finally the audio features are output.

6. The method for generating digital human face video based on voice-driven style vector according to claim 1, It is characterized in that In step S33, the feature coupling module concatenates the audio features with the 18 style vectors respectively to obtain a joint feature representation that includes audio and image style information.

7. The method for generating digital human face video driven by voice based on style vector according to claim 1, It is characterized in that The loss function designed in the process of training the network in step S3 includes: Loss function, Loss function, Loss function, Loss function, Loss function, the loss function calculation formula is: in, The loss function is used to calculate the error between the real image and the predicted image. The loss function calculates the perceptual loss between the true value and the predicted value; in order to ensure that the input and output are human faces, The loss function uses the ArcFace network to measure the cosine similarity between the input and output; The loss function regularizes the decoder. The loss function uses the syncnet network to calculate the matching degree between the lip shape and the speech; , , , , are the weights of the corresponding loss functions respectively.

8. The method for generating digital human face video based on voice-driven style vector according to claim 1, It is characterized in that The testing of the network in step S4 specifically includes: obtaining the lip synchronization accuracy through the syncnet network for the input speaker face and voice; combining The loss function obtains the perceptual loss to judge the face realism and finally outputs the speaking video.