Text-driven lip-sync digital human generation method, device, equipment and medium

Abstract

The present application relates to a text-driven lip-sync digital human generation method, device, equipment and medium, including: obtaining input text and a single object video, preprocessing the input text to obtain a phoneme sequence and semantic features; preprocessing the single object video to obtain a looped video segment, a masked facial image and a reference facial image; generating voice features based on the phoneme sequence and semantic features; generating an audio sequence based on the voice features; obtaining facial deformation parameters based on the masked facial image, the reference facial image and the voice features and deforming the reference facial image to generate a target facial image; embedding the target facial image into the video frame of the video segment to generate a video sequence and splicing it with the audio sequence to obtain the target digital human video. Generating digital human videos based on text-driven can instantly generate voice and lip shape videos that match the text, eliminating the time delay of waiting for text to be converted to voice processing, and greatly improving the efficiency of digital human video production.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a text-driven lip-sync digital human generation method, device, equipment and medium. Background Art

[0002] With the rapid development of artificial intelligence technology, digital human generation technology has become a key bridge connecting the boundaries of virtuality and reality, showing unprecedented application potential in entertainment, education, remote communications and other fields. In particular, in the pursuit of highly immersive film production, interactive games and increasingly mature virtual reality experiences, the importance of lip synchronization technology has become increasingly prominent. This technology greatly enhances the realism and interactivity of the user experience by accurately matching the lip movements of virtual characters with audio content.

[0003] However, the current development of lip-sync digital human technology still faces significant challenges. Traditional methods are highly dependent on pre-recorded audio materials and cumbersome post-processing processes, which are not only inefficient, but also accompanied by high production costs and long project cycles, which seriously restrict the widespread application of the technology. Although in recent years, the text-to-speech (TTS) system combined with voice cloning technology has been introduced to simplify the process of generating digital humans and improve the flexibility of generation, the non-instantaneous defect of this technology is still a major bottleneck. Users need to wait for the TTS system to complete audio generation before they can perform subsequent processing, which cannot meet the requirements of real-time interactive scenarios for instant response, thereby limiting the scope of application of the technology and reducing the user experience.

[0004] Therefore, how to achieve more efficient, flexible and highly instant lip-sync digital human technology has become the focus of current research. Summary of the invention

[0005] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art and provide a text-driven lip-synced digital human generation method, device, equipment and medium to solve the problem in the prior art that digital human generation is not timely enough to meet real-time interaction scenarios.

[0006] According to a first aspect of the present application, a text-driven lip-sync digital human generation method is provided, the method comprising:

[0007] Obtaining input text and a single-object video, preprocessing the input text to obtain a phoneme sequence and semantic features; preprocessing the single-object video to obtain a video segment that can be played in a loop, a masked facial image, and a reference facial image;

[0008] generating a speech feature based on the phoneme sequence and the semantic feature;

[0009] generating an audio sequence based on the speech features;

[0010] Acquire facial deformation parameters according to the masked facial image, the reference facial image and the voice features, deform the reference facial image based on the facial deformation parameters to generate a target facial image; embed the target facial image into a video frame of the video segment to generate a video sequence;

[0011] The video sequence and the audio sequence are combined to obtain a target digital human video.

[0012] Text-driven generation of digital human videos can instantly generate voice and lip-sync videos that match the text, eliminating the time delay of waiting for text to be converted into voice processing, achieving end-to-end automated generation from text to video, and greatly improving the efficiency of digital human video production.

[0013] Optionally, generating an audio sequence based on the speech feature includes:

[0014] Quantizing the speech features to obtain quantized speech features;

[0015] Encoding the quantized speech features to generate mean and variance parameters describing the distribution of the speech features;

[0016] Using the mean and variance parameters of the speech feature distribution, sampling from the speech feature distribution to obtain a speech intermediate representation, and generating an audio sequence according to the speech intermediate representation;

[0017] By quantizing speech features, data redundancy can be effectively reduced, reducing the complexity of subsequent processing. At the same time, the mean and variance parameters generated by the encoding process concisely describe the distribution characteristics of speech features, making the sampling process faster and more accurate, thereby improving the efficiency of audio sequence generation.

[0018] Optionally, acquiring facial deformation parameters according to the masked facial image, the reference facial image and the voice feature includes:

[0019] Encoding the masked facial image, the reference facial image and the speech features to obtain facial features, reference facial features and lip shape features respectively;

[0020] The facial features and the reference facial features are combined and compressed into one-dimensional image transformation parameters, and the image transformation parameters are combined with the lip shape features to obtain facial deformation parameters.

[0021] By integrating masked facial images, reference facial images, and voice features, precise control of facial expressions and lip shapes is achieved, thereby improving the accuracy and naturalness of facial deformation.

[0022] Optionally, deforming the reference facial image based on the facial deformation parameters to generate a target facial image specifically includes:

[0023] Processing the reference facial features using a residual convolution module of a convolutional neural network to obtain deep semantic features of the reference face;

[0024] Performing dimension transformation and feature embedding on the facial deformation parameters to obtain embedded facial deformation parameters;

[0025] Processing the embedded facial deformation parameters using a fully convolutional layer of a convolutional neural network to obtain a scaling matrix, a rotation matrix, and a translation matrix respectively;

[0026] Deforming the reference facial deep semantic features using the scaling matrix, the rotation matrix, and the translation matrix to obtain deformed reference facial features;

[0027] combining the facial features with the deformed reference facial features to obtain combined features;

[0028] The stitched features are gradually upsampled and restored, and the spatial resolution of the features is enhanced to generate a target facial image.

[0029] Through the residual convolution module and full convolution layer of the convolutional neural network, deep semantic understanding of the reference facial features is achieved, and natural and smooth deformation adjustments are made according to the lip shape changes driven by speech, ultimately generating high-quality and high-fidelity target facial images, making the generated digital human video more realistic and vivid.

[0030] Optionally, generating a speech feature based on the phoneme sequence and the semantic feature includes:

[0031] Phoneme feature embedding: Map the phoneme sequence to a high-dimensional continuous vector space to obtain phoneme features;

[0032] Semantic feature dimension transformation and fusion: perform dimension transformation on the semantic feature to match the dimension of the semantic feature with the phoneme feature, and then combine the dimension-transformed semantic feature and the phoneme feature to form a fusion feature;

[0033] Fusion feature position encoding: Apply position encoding to the fusion features;

[0034] Autoregressive decoding: Autoregressive feature interaction and decoding are performed on the fusion features after position encoding in a cyclic iterative manner to generate speech features.

[0035] Through phoneme feature embedding, semantic feature dimensionality transformation and fusion, fused feature position encoding and autoregressive decoding, text information is effectively converted into rich speech features; through position encoding and autoregressive decoding, the speech features can maintain temporal order and coherence, and generate speech features that are consistent with the text content and have natural speech characteristics, providing a solid foundation for subsequent digital human video generation and significantly improving the realism and expressiveness of speech synthesis.

[0036] Optionally, performing autoregressive feature interaction and decoding on the fused features after applying position encoding in a cyclic iteration manner to generate speech features specifically includes:

[0037] Initialization: using the preset startup identifier feature STA as the initial voice feature y;

[0038] Loop iteration: For each iteration, perform the following steps:

[0039] Embedding and position encoding the speech feature y;

[0040] Combine the fusion feature x with the speech feature y that has been embedded and positionally encoded to form a combined feature xy;

[0041] Constructing an attention mask according to the length of the fused feature x and the speech feature y;

[0042] Use the decoder to decode the combined feature xy to obtain the decoded output;

[0043] The decoded output is processed using the fully connected layer of the Transformer model to obtain the speech feature y_now corresponding to the current cycle;

[0044] Combine the speech feature y_now corresponding to the previous cycle with the speech feature y, and update the speech feature y with the combined result;

[0045] Determine whether the speech feature y_now corresponding to the current loop is the preset stop feature EOS, if not, continue to the next iteration, if yes, end the loop and output the speech feature y.

[0046] The speech feature sequence is generated through a loop iteration method. Each step in the loop generates a new speech feature sequence based on the previous information. It not only reflects the speech characteristics of the target text but also integrates the semantics of the target text. The generated speech features can more accurately reflect the speech rhythm and emotional expression of the target text, laying a solid foundation for subsequent audio and video generation.

[0047] Optionally, the preprocessing of the single object video to obtain a video segment that can be played in a loop, a masked facial image, and a reference facial image specifically includes:

[0048] Obtaining a reverse playback video of the single object video, and splicing the single object video with the reverse playback video to form a video segment that can be played in a loop;

[0049] Performing facial recognition on each frame of the video segment and extracting facial key points of each frame; smoothing the facial key points using a filter, cropping each frame according to the smoothed facial key points, and obtaining a facial image, the coordinates of the facial key points and a cropping frame position.

[0050] Randomly selecting a number of frames of facial images from all the facial images for splicing to form a reference facial image;

[0051] Based on the facial key points, the mouth area of ​​the facial image is masked to obtain a masked facial image that only shows the mouth movement and has a transparent background.

[0052] By acquiring and looping video templates, the smooth continuation of video content can be achieved. Regardless of the length of the generated video, visual continuity can be guaranteed without worrying about sudden interruptions or unnatural repetitions. Smoothing effectively reduces noise and jitter, laying a solid foundation for subsequent face cropping and masking. Randomly splicing reference facial images helps to better capture and simulate the overall features and changes of the face in subsequent processing, improving the generalization ability of the processing results. By masking the mouth area, the effect of only retaining the mouth movement and making the background transparent is achieved, making the mouth movement the visual focus, which helps to more accurately analyze and apply mouth information.

[0053] According to a second aspect of the present application, a text-driven lip-sync digital human generation device is provided, the device comprising:

[0054] A preprocessing module is used to obtain input text and a single object video, preprocess the input text to obtain a phoneme sequence and semantic features; preprocess the single object video to obtain a video segment that can be played in a loop, a masked facial image, and a reference facial image;

[0055] A speech feature generation module, used for generating speech features based on the phoneme sequence and the semantic features;

[0056] An audio generation module, used for generating an audio sequence based on the speech features;

[0057] A video sequence generation module, configured to obtain facial deformation parameters according to the masked facial image, the reference facial image and the voice features, deform the reference facial image based on the facial deformation parameters to generate a target facial image; and embed the target facial image into a video frame of the video segment to generate a video sequence;

[0058] Digital human video generation module: used for combining the video sequence and the audio sequence to obtain the target digital human video.

[0059] According to a third aspect of the present application, an electronic device is provided, including:

[0060] a memory for storing one or more computer programs;

[0061] The processor, when the one or more computer programs are executed by the processor, implements the text-driven lip-sync digital human generation method described in the first aspect.

[0062] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text-driven lip-sync digital human generation method described in the first aspect when executed.

[0063] Based on any of the above aspects, the text-driven lip-sync digital human generation method, device, electronic device and computer storage medium provided by the embodiments of the present application have the following beneficial effects:

[0064] 1. By directly starting from the input text, it can instantly generate matching voice and lip animation, effectively eliminating the time delay problem in traditional text-to-speech processing, improving the fluency and naturalness of communication in fast-paced instant interactive application scenarios, and significantly enhancing the real-time interactive quality of the user experience.

[0065] 2. By directly generating sound and images based on the deep semantic understanding of the input text, the synchronization accuracy of the sound and lip shape of the digital human video is improved. The sound characteristics and lip shape changes are rooted in the deep semantic understanding of the input text. The use of rich audio feature mapping improves the realism of the digital human expression, the perfect coordination and natural expression of the sound characteristics and lip shape changes.

[0066] 3. The present invention uses input text as the driving source and integrates audio generation and digital human driving into the same efficient network, which significantly reduces the demand for computing resources, making the technology more lightweight and efficient, and able to run efficiently on resource-constrained platforms, greatly broadening the application scope of digital human technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 A schematic application scenario diagram of the text-driven lip-sync digital human generation method provided in this embodiment.

[0069] Figure 2 This is a flow chart of the text-driven lip-sync digital human generation method provided in this embodiment.

[0070] Figure 3 This embodiment provides a schematic diagram of the steps of preprocessing the single object video.

[0071] Figure 4 A schematic diagram of the speech feature generation process provided in this embodiment.

[0072] Figure 5 A schematic diagram of a flow chart of the autoregressive decoding sub-steps provided in this embodiment.

[0073] Figure 6 This embodiment provides a flowchart of generating an audio sequence based on the speech features.

[0074] Figure 7 This embodiment provides a schematic diagram of a process of deforming the reference facial image to generate a target facial image.

[0075] Figure 8 A schematic diagram of the structure of a text-driven lip-sync digital human generation device provided in this embodiment.

[0076] Fig. 9 This embodiment provides a schematic diagram of the overall flow of the text-driven lip-sync digital human generation method.

[0077] Fig.10 A schematic diagram of the structure of an electronic device provided by this embodiment. DETAILED DESCRIPTION

[0078] The drawings of this application are only used for illustrative purposes and should not be construed as limiting the present application. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0079] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0080] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0081] Although the current lip-sync digital human technology has shown great potential in the fields of entertainment, education, and communications, it still faces multiple technical bottlenecks. Traditional methods rely on high-cost, long-term audio recording and post-processing, and the emerging text-to-speech (TTS) system based on voice cloning has improved flexibility, but is still limited by the immediacy of audio generation and cannot meet the needs of seamless lip synchronization in real-time interactive scenarios.

[0082] This embodiment provides a technical solution that can solve the above-mentioned problem. The specific implementation methods of this application are described in detail below in conjunction with the accompanying drawings.

[0083] For example, a text-driven lip-sync digital human generation method provided in an embodiment of the present application is shown in the following application scenario diagram. Figure 1 As shown, the application scenario at least includes a server 100 and a terminal 200 that can communicate with the server 100, the server 100 has image and voice processing functions; the terminal device 200 has a streaming media playback function and also has image and voice processing functions.

[0084] It is understandable that the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a car terminal, etc., but is not limited thereto.

[0085] In one practicable manner, the server 100 and the terminal 200 can respectively execute the text-driven lip-synced digital human generation method provided in the embodiment of the present application, or, optionally, the text-driven lip-synced digital human generation method provided in the embodiment of the present application is partially executed in the server 100 and partially executed in the terminal 200.

[0086] like Figure 2 and Fig. 9 As shown, this embodiment provides a text-driven lip-sync digital human generation method, which may include the following steps:

[0087] S1. Obtain input text and a single object video, preprocess the input text to obtain a phoneme sequence and semantic features; preprocess the single object video to obtain a video segment that can be played in a loop, a masked facial image, and a reference facial image.

[0088] In this embodiment, the single-object video may be a speech video of a single person, may be a real-time recorded video, or may be a speech video of a single person obtained through the network.

[0089] Preprocessing the input text to obtain the phoneme sequence and semantic features includes two steps: text segmentation and semantic feature extraction. Text segmentation is specifically: segmenting the input text into sentences, and then converting each segmented sentence into a corresponding phoneme sequence. Semantic feature extraction is specifically: using the BERT network to deeply analyze each segmented input text and extract the semantic features of each input text;

[0090] In the specific implementation process, TTS technology can be used to convert each segmented sentence into a corresponding phoneme sequence, or a statistical model or deep learning method can be used to implement it, which is not limited in this embodiment.

[0091] like Figure 3 As shown, the single object video is preprocessed, specifically including the following steps:

[0092] S102, video template acquisition and loop construction: generating a reverse video of the single object video, splicing the original video with the reverse video, and forming a video segment that can be played in an infinite loop.

[0093] By constructing a video loop, a video segment that can be played in a seamless loop is obtained. Since the video segment can be looped infinitely, a lip-synced digital human video of any length can be generated based on the video segment as needed. Regardless of the length of the generated digital human video, its visual coherence can be guaranteed, and there is no need to worry about the digital human video being suddenly interrupted or repeated unnaturally.

[0094] S104, key point detection and filtering: performing facial recognition on each frame of the single object video and extracting facial key points of each frame; smoothing the facial key points using a filter, cropping each frame according to the smoothed facial key points, and obtaining a facial image, coordinates of the facial key points, and a cropping frame position.

[0095] Specifically, in this embodiment, the filter may be a Euro filter, which is a low-pass filter for time series data, and can effectively reduce the impact of noise on the key point sequence, making the key point trajectory smoother and more natural.

[0096] S106, reference face extraction: randomly selecting a number of frames of face images from all the face images and splicing them to form a reference face image.

[0097] For example, in this embodiment, five frames of facial images are randomly selected and spliced ​​together as reference facial images for subsequent guidance of learning and generation of digital human facial features. By randomly selecting facial images for splicing, the diversity of reference facial images can be increased, which helps to improve the realism and naturalness of digital human facial features.

[0098] S108, obtaining a masked facial image: Based on the facial key points, masking is performed on the mouth areas of all corresponding facial images to obtain a masked facial image that only shows the mouth movement and has a transparent background.

[0099] The mouth area is separated from the entire face to form a masked face image that only shows the mouth movement and the background is transparent. Since the masking process removes the interference of the non-mouth area, the lip shape generation algorithm can focus more on the movement changes of the mouth area, thereby improving the accuracy and naturalness of lip synchronization.

[0100] S2. Generate speech features based on the phoneme sequence and the semantic features.

[0101] As an optional embodiment, this step realizes the generation of speech features by adopting a generative Transformer model. The Transformer model has a powerful sequence modeling capability and is particularly suitable for long-distance dependent learning tasks. By constructing a speech feature generation module using the Transformer model, speech features can be generated based on the phoneme sequence and the semantic features of the target text for subsequent audio and lip shape generation.

[0102] In this embodiment, if Figure 4 As shown, generating speech features based on the phoneme sequence and the semantic features specifically includes the following steps:

[0103] S202, phoneme feature embedding: map the phoneme sequence to a high-dimensional continuous vector space to obtain phoneme features.

[0104] By mapping the phoneme sequence into a high-dimensional continuous vector space, an easy-to-process data format is subsequently provided.

[0105] S204, semantic feature dimension transformation and fusion: perform dimension transformation on the semantic feature to make the dimension of the semantic feature match that of the phoneme feature, and then combine the dimension-transformed semantic feature and the phoneme feature to form a fusion feature.

[0106] The fusion of semantic feature dimension transformation can ensure that the generated speech features not only contain correct pronunciation information, but also reflect the semantics of the text itself.

[0107] S206, fusion feature position coding: applying position coding to the fusion feature.

[0108] The position information of each feature vector in the fusion feature is crucial to understanding the language order and contextual relationship. The fusion feature position encoding can help the model distinguish the phoneme features and semantic features at different positions, thereby generating speech features more accurately.

[0109] S208, autoregressive decoding: performing autoregressive feature interaction and decoding on the fusion features in a cyclic iteration manner to generate speech features.

[0110] In this embodiment, autoregressive decoding is implemented by a decoder composed of multiple self-attention layers, and the autoregressive feature interaction and decoding process is performed in a loop iteration. Preferably, in this embodiment, the self-attention layer is 6 layers. In the loop iteration process, each step depends on the previous information to generate new speech features. The specific execution process of autoregressive decoding is as follows:

[0111] Initialization: using the preset startup identifier feature STA as the initial voice feature y;

[0112] Loop iteration: For each iteration, Figure 5 As shown, perform the following steps:

[0113] A1. Embed and position encode the current speech feature y;

[0114] A2, combining the fusion feature x with the current speech feature y to form a combined feature xy;

[0115] A3. Construct an attention mask based on the length of the fused feature x and the output speech feature y;

[0116] A4. Use a decoder to decode the combined feature xy to obtain a decoded output;

[0117] A5. Use the fully connected layer of the Transformer model to process the decoded output and obtain the speech feature y_now corresponding to the current cycle.

[0118] A6. Combine the speech feature y_now corresponding to the previous cycle with the speech feature y to form a new speech feature y.

[0119] A7. Determine whether the current speech feature y_now is the preset stop feature EOS. If not, continue to the next iteration. If yes, end the loop and output the speech feature y.

[0120] After the cycle ends, the model outputs the final generated speech feature y. The complete speech feature sequence is gradually generated by autoregression, so that the generated speech features not only conform to the phoneme sequence, but also integrate the semantics of the input text, while maintaining the natural fluency of the language, laying a solid foundation for subsequent audio and video generation.

[0121] S3. Generate an audio sequence based on the speech features.

[0122] In this embodiment, an audio sequence generation module is optimized and constructed based on the VITS variational autoencoder (VAE) architecture to achieve accurate conversion from predefined speech features to high-fidelity audio signals, thereby achieving deep integration and unification of content, emotional expression and user personalized needs in the process of synthesizing speech.

[0123] Figure 6 A flowchart for generating an audio sequence based on the speech features, such as Figure 6 As shown, generating an audio sequence based on the speech feature specifically includes the following steps:

[0124] S302, speech feature quantization processing: quantizing the speech features to obtain quantized speech features.

[0125] Quantizing speech features can reduce the complexity of speech feature representation and obtain quantized speech features. Quantization can reduce the volume of data while retaining enough information for subsequent steps, which is crucial for the efficient processing and storage of speech features.

[0126] S304, deep feature coding and latent space mapping: encoding the quantized speech features to generate mean and variance parameters describing the distribution of the speech features.

[0127] In this embodiment, the quantized speech features are encoded by a variational autoencoder (VAE) and mapped to a latent space to generate mean and variance parameters describing the distribution of the speech features, thereby constructing a transformation from speech features to speech feature distribution.

[0128] S306, sampling, decoding and waveform reconstruction: sampling the speech feature distribution to obtain a speech intermediate representation, and generating an audio sequence based on the speech intermediate representation.

[0129] In this embodiment, the intermediate representation of speech is decoded by a decoder of a variational autoencoder to generate an audio sequence of a natural speech waveform.

[0130] S4. Acquire facial deformation parameters according to the masked facial image, the reference facial image and the voice features, deform the reference facial image based on the facial deformation parameters to generate a target facial image; embed the target facial image into a video frame of the video segment to generate a video sequence.

[0131] In this step, a deep learning convolutional neural network is used to implement the process, including input encoding, feature alignment and conversion, adaptive appearance deformation, and feature decoding and output.

[0132] Acquiring facial deformation parameters according to the masked facial image, the reference facial image and the voice feature specifically includes:

[0133] S402, encoding the masked facial image, the reference facial image and the voice features to obtain facial features, reference facial features and lip shape features respectively;

[0134] This step mainly realizes the preliminary encoding of three inputs, including facial image encoding, reference facial image encoding, and speech feature encoding, in preparation for the subsequent generation of a facial image with correct lip shape.

[0135] Facial image encoding specifically includes: using convolutional neural networks to gradually downsample features, while increasing the channel depth of the feature map, and finally extracting a 256-channel facial feature map. Among them, facial image encoding includes several sequentially connected convolutional layers, each of which is followed by a batch normalization layer and a ReLU activation function layer.

[0136] The reference face encoding uses the same network structure as the face image encoding, and is processed by the same number of convolutional layers as the face image encoding, each of which is followed by a batch normalization layer and a ReLU activation function layer. The mask face image and the reference face image are processed by the same number of convolutional layers, so that the converted reference face image and the face features have the same dimension, which is convenient for subsequent feature fusion and deformation.

[0137] The speech feature encoding specifically includes: further refining the speech features and integrating the lip shape related clues in the audio through a one-dimensional convolution layer and a residual connection to obtain a lip shape feature of size 1x128. Preferably, the number of one-dimensional convolution layers is 8 and the number of residuals is 2.

[0138] S404: The facial features and the reference facial features are combined and compressed into one-dimensional image transformation parameters, and the image transformation parameters are combined with the lip shape features to obtain facial deformation parameters.

[0139] Step S404 is mainly used for feature alignment and conversion, by aligning the facial features with the reference facial features in the latent space, and then combining the lip shape features to obtain the facial deformation parameters, as follows:

[0140] Combining the facial features with the reference facial features, and compressing the combined facial features into one-dimensional image transformation parameters using a convolution block;

[0141] The image transformation parameters and the lip shape features are combined to obtain facial deformation parameters.

[0142] Preferably, the convolution block has 11 layers, wherein each convolution block includes a convolution layer, a batch normalization layer and a ReLU activation function layer.

[0143] like Figure 7 As shown, deforming the reference facial image based on the facial deformation parameters to generate a target facial image specifically includes:

[0144] S411, using a residual convolution module of a convolutional neural network to process the reference facial features to obtain deep semantic features of the reference face;

[0145] S412, performing dimension transformation and feature embedding on the facial deformation parameters to obtain embedded facial deformation parameters;

[0146] In a specific implementation process, the dimension transformation and feature embedding of the facial deformation parameters can be achieved through the first fully connected layer of the convolutional neural network.

[0147] S413, using a full convolution layer of a convolutional neural network to process the embedded facial deformation parameters to obtain a scaling matrix, a rotation matrix, and a translation matrix respectively;

[0148] In a specific implementation process, multiple second full convolutional layers of the convolutional neural network can be used to further process the embedded facial deformation parameters respectively, so as to obtain a scaling matrix, a rotation matrix and a translation matrix.

[0149] S414: deform the reference facial deep semantic features using the scaling matrix, the rotation matrix, and the translation matrix to obtain deformed reference facial features.

[0150] Steps S411 to S414 implement adaptive appearance deformation by processing the reference facial features using several layers of residual convolution modules to obtain deep semantic features of the reference face.

[0151] S415, combining the input facial features with the deformed reference facial features to obtain combined features;

[0152] S416, gradually upsampling the stitched features to restore them and enhance the spatial resolution of the features to generate a target facial image.

[0153] Step S415 and step S416 implement feature decoding and output, and combine the input facial features with the deformed reference facial features and then process them to compensate for the detailed features lost in the reference facial features after deformation, so that the generated face is consistent with the real face.

[0154] Exemplarily, the feature decoding and output stage of this embodiment includes 4 convolutional layers and 2 deconvolutional layers. Each convolutional layer and each deconvolutional layer except the last convolutional layer are connected to a batch normalization layer and a ReLU activation function layer to ensure stable transformation of features. Finally, the range of the output value is adjusted to the (0,1) interval through the Sigmoid activation layer to generate and output the target facial image for subsequent processing and display.

[0155] S5. Combine the video sequence and the audio sequence to obtain a target digital human video.

[0156] In the specific implementation process, the generated target facial image is accurately embedded into the corresponding position of the video frame of the video segment, so that each frame can transition naturally. By processing the video frame by frame, a coherent digital human video sequence can be generated. After the video processing is completed, the generated audio sequence is spliced ​​with the video sequence to achieve natural and accurate synchronization of the digital human's lip shape and audio, and finally obtain the target digital human video with natural synchronization of lip shape and audio.

[0157] The technical solution of the text-driven lip-sync digital human generation method provided by the present application is described below in conjunction with a specific embodiment.

[0158] like Figure 8 As shown, the embodiment of the present application further provides a text-driven lip-sync digital human generation device 610. Optionally, the text-driven lip-sync digital human generation device 610 may include:

[0159] The preprocessing module 611 is used to obtain input text and a single object video, preprocess the input text to obtain a phoneme sequence and semantic features; preprocess the single object video to obtain a video segment that can be played in a loop, a masked facial image, and a reference facial image;

[0160] A speech feature generation module 612, configured to generate speech features based on the phoneme sequence and the semantic features;

[0161] An audio generation module 613, configured to generate an audio sequence based on the speech features;

[0162] The video sequence generation module 614 is used to obtain facial deformation parameters according to the masked facial image, the reference facial image and the voice feature, deform the reference facial image based on the facial deformation parameters to generate a target facial image; and embed the target facial image into the video frame of the video segment to generate a video sequence;

[0163] The digital human video generation module 615 is used to combine the video sequence and the audio sequence to obtain the target digital human video.

[0164] It can be understood that the above-mentioned device embodiment and the above-mentioned method embodiment can correspond to each other, and similar descriptions of the device embodiment can refer to the method embodiment. To avoid repetition, it will not be repeated here. A text-driven lip-sync digital human generation device provided in an embodiment of the present application can execute a text-driven lip-sync digital human generation method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. The functional modules of the text-driven lip-sync digital human generation device can be implemented in the form of hardware, can be implemented by instructions in the form of software, and can also be implemented by a combination of hardware and software modules.

[0165] Specifically, each step of the method embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor. The steps of the text-driven lip-sync digital human generation method in combination with the embodiment of the present application can be directly embodied as a hardware encoding processor, or a combination of hardware and software modules in the encoding processor. Optionally, the software module can be located in a random access memory, a read-only memory, a programmable read-only memory, a flash memory, an electrically erasable programmable memory, a register, or other storage media. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.

[0166] The present application embodiment provides an electronic device 710, whose structure is as follows: Fig. 9 The electronic device 710 may be Figure 1 The server 100 or the terminal 200 is shown.

[0167] like Fig. 9As shown, the electronic device 710 includes a memory 711, a processor 712, a communication module 713 and an input / output interface 714, etc. Optionally, the memory 711, the processor 712, the communication module 713 and the input / output interface 714 can be connected and communicated through a bus 715.

[0168] The memory 711 is used to store one or more computer programs and transmit the code of the computer program to the processor 712; when the one or more computer programs are executed by the processor 712, the text-driven lip-sync digital human generation method in the embodiment of the present application is implemented.

[0169] Optionally, the electronic device 710 can be connected to a network via a communication module 713 to communicate with other devices, such as a terminal or a server, through the network to achieve data interaction. The electronic device 710 can be various forms of digital computers, such as desktop computers, servers, workstations, mainframe computers or other types of computers. The electronic device 710 can also be various forms of mobile terminals, such as smart phones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.) and other similar mobile terminals.

[0170] Optionally, the electronic device 710 can be connected to the required input / output devices, such as a keyboard, a display device, etc., through the input / output interface 714. The electronic device 710 itself can have a display device, and can also be connected to other display devices through the input / output interface 714. Optionally, a storage device, such as a hard disk, can also be connected through the input / output interface 714, so that data in the electronic device 710 can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 711. It can be understood that the input / output interface 714 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 714 can be a component of the electronic device 710, or it can be an external device connected to the electronic device 710 when needed.

[0171] Optionally, the memory 711 may be a volatile memory and / or a non-volatile memory, the volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory or a flash memory, etc.

[0172] Optionally, the computer program stored in the memory 711 may be divided into one or more modules, which are stored in the memory 711 and executed by the processor 712 to complete the method provided in the embodiment. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the computer program instruction segments are used to describe the execution process of the computer program in the electronic device 710.

[0173] Optionally, the processor 712 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 712 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special artificial intelligence computing chips, various processors running machine learning model algorithms, and may also be any appropriate controller, microcontroller, processor, etc. The processor 712 executes the various methods and processes of this embodiment, illustratively, such as a text-driven lip-sync digital human generation method in an embodiment of the present application.

[0174] Optionally, the bus 715 may include a path for transmitting information. The bus 715 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 715 may be divided into an address bus, a data bus, a control bus, etc.

[0175] In an optional implementation, the embodiment of the present application further provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. Part or all of the computer program can be loaded and / or installed on the memory 711 of the electronic device 710. When the computer program is executed by the processor 712, one or more steps of a text-driven lip-sync digital human generation method of the embodiment of the present application can be performed.

[0176] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or the like.

[0177] Obviously, the above embodiments of the present application are merely examples for clearly illustrating the technical solution of the present application, and are not intended to limit the specific implementation methods of the present application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A text-driven lip-sync digital human generation method, characterized in that: The method comprises: Obtaining input text and a single-object video, preprocessing the input text to obtain a phoneme sequence and semantic features; preprocessing the single-object video to obtain a video segment that can be played in an infinite loop, a masked facial image, and a reference facial image; generating a speech feature based on the phoneme sequence and the semantic feature; generating an audio sequence based on the speech features; Acquire facial deformation parameters according to the masked facial image, the reference facial image and the voice features, deform the reference facial image based on the facial deformation parameters to generate a target facial image; embed the target facial image into the video frame of the video segment to generate a video sequence; wherein, the facial deformation parameters acquired by the masked facial image, the reference facial image and the voice features include: encoding the masked facial image, the reference facial image and the voice features to obtain facial features, reference facial features and lip features respectively; splicing the facial features and the reference facial features and compressing them into one-dimensional image transformation parameters, splicing the image transformation parameters and the lip features to obtain facial deformation parameters; wherein, the facial deformation parameters based on the facial deformation parameters are used to generate a video sequence; The reference facial image is deformed to generate a target facial image, including: using a residual convolution module of a convolutional neural network to process the reference facial features to obtain deep semantic features of the reference face; performing dimension transformation and feature embedding on the facial deformation parameters to obtain embedded facial deformation parameters; using a full convolution layer of a convolutional neural network to process the embedded facial deformation parameters to obtain a scaling matrix, a rotation matrix and a translation matrix respectively; using the scaling matrix, the rotation matrix and the translation matrix to deform the deep semantic features of the reference face to obtain deformed reference facial features; splicing the facial features with the deformed reference facial features to obtain spliced ​​features; performing step-by-step upsampling on the spliced ​​features to restore and enhance the spatial resolution of the features to generate a target facial image; The video sequence and the audio sequence are combined to obtain a target digital human video.

2. The method for generating a text-driven lip-sync digital human according to claim 1, characterized in that: Generating an audio sequence based on the speech feature comprises: Quantizing the speech features to obtain quantized speech features; Encoding the quantized speech features to generate mean and variance parameters describing the distribution of the speech features; The mean and variance parameters of the speech feature distribution are used to sample from the speech feature distribution to obtain a speech intermediate representation, and an audio sequence is generated according to the speech intermediate representation.

3. A text-driven lip-sync digital human generation method according to any one of claims 1 to 2, characterized in that: The generating of speech features based on the phoneme sequence and the semantic features comprises: Phoneme feature embedding: Map the phoneme sequence to a high-dimensional continuous vector space to obtain phoneme features; Semantic feature dimension transformation and fusion: perform dimension transformation on the semantic feature to match the dimension of the semantic feature with the phoneme feature, and then combine the dimension-transformed semantic feature and the phoneme feature to form a fusion feature; Fusion feature position encoding: Apply position encoding to the fusion features; Autoregressive decoding: Autoregressive feature interaction and decoding are performed on the fusion features after position encoding in a cyclic iterative manner to generate speech features.

4. The method for generating a text-driven lip-sync digital human according to claim 3, characterized in that: The fusion features after applying the position encoding are subjected to autoregressive feature interaction and decoding in a cyclic iteration manner to generate speech features, specifically including: Initialization: using the preset startup identifier feature STA as the initial voice feature y; Loop iteration: For each iteration, perform the following steps: Embedding and position encoding the speech feature y; Combine the fusion feature x with the speech feature y that has been embedded and positionally encoded to form a combined feature xy; Constructing an attention mask according to the length of the fused feature x and the speech feature y; Use the decoder to decode the combined feature xy to obtain the decoded output; The decoded output is processed using the fully connected layer of the Transformer model to obtain the speech feature y_now corresponding to the current cycle; Combine the speech feature y_now corresponding to the current cycle with the speech feature y, and update the speech feature y with the combined result; Determine whether the speech feature y_now corresponding to the current loop is the preset stop feature EOS, if not, continue to the next iteration, if yes, end the loop and output the speech feature y.

5. The method for generating a text-driven lip-sync digital human according to any one of claims 1 to 2, characterized in that: The preprocessing of the single object video to obtain a video segment that can be played in a loop, a masked facial image, and a reference facial image specifically includes: Obtaining a reverse playback video of the single object video, and splicing the single object video with the reverse playback video to form a video segment that can be played in a loop; Performing facial recognition on each frame of the video segment and extracting facial key points of each frame; smoothing the facial key points using a filter, and cropping each frame according to the smoothed facial key points to obtain a facial image, coordinates of the facial key points, and a cropping frame position; Randomly selecting a number of frames of facial images from all the facial images for splicing to form a reference facial image; Based on the facial key points, the mouth area of ​​the facial image is masked to obtain a masked facial image that only shows the mouth movement and has a transparent background.

6. A text-driven lip-sync digital human generation device, characterized in that: The device comprises: A preprocessing module is used to obtain input text and a single object video, preprocess the input text to obtain a phoneme sequence and semantic features; preprocess the single object video to obtain a video segment that can be played in a loop, a masked facial image, and a reference facial image; A speech feature generation module, used for generating speech features based on the phoneme sequence and the semantic features; An audio generation module, used for generating an audio sequence based on the speech features; The video sequence generation module is used to obtain facial deformation parameters according to the masked facial image, the reference facial image and the voice features, deform the reference facial image based on the facial deformation parameters to generate a target facial image; embed the target facial image into the video frame of the video segment to generate a video sequence; wherein the facial deformation parameters obtained by the masked facial image, the reference facial image and the voice features include: encoding the masked facial image, the reference facial image and the voice features to obtain facial features, reference facial features and lip features respectively; splicing the facial features and the reference facial features and compressing them into one-dimensional image transformation parameters, splicing the image transformation parameters and the lip features to obtain facial deformation parameters; wherein the facial deformation parameters are obtained based on the facial image. The method comprises: deforming the reference facial image using the residual convolution module of the convolutional neural network to obtain the deep semantic features of the reference face; performing dimension transformation and feature embedding on the facial deformation parameters to obtain embedded facial deformation parameters; processing the embedded facial deformation parameters using the full convolution layer of the convolutional neural network to obtain a scaling matrix, a rotation matrix and a translation matrix respectively; deforming the deep semantic features of the reference face using the scaling matrix, the rotation matrix and the translation matrix to obtain deformed reference facial features; splicing the facial features with the deformed reference facial features to obtain spliced ​​features; performing step-by-step upsampling on the spliced ​​features to restore and enhance the spatial resolution of the features to generate a target facial image; Digital human video generation module: used for combining the video sequence and the audio sequence to obtain the target digital human video.

7. An electronic device, characterized in that: include: a memory for storing one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements the text-driven lip-sync digital human generation method as described in any one of claims 1-5.

8. A computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to implement the text-driven lip-sync digital human generation method according to any one of claims 1 to 5 when executed.

Images