A speech-driven lip generation method and device and a medium

Abstract

The application provides a speech-driven lip shape generation method and device, and a medium, and relates to the field of computer vision.The method comprises the following steps: obtaining an audio signal of an original target video; constructing an audio-expression coefficient mapping network; inputting the audio signal into the audio-expression coefficient mapping network for training to obtain mixed deformation coefficients of a target person; the audio-expression coefficient mapping network comprises a feature extraction module, a self-attention enhancement module, and a personalized expression conversion module; constructing a 3D rendered face image of the target person; performing neural texture mapping on the 3D rendered face image to generate a feature map; rendering a mouth region of the 3D rendered face image through the feature map to obtain a high-resolution mouth rendered image; and combining a background synthesis network through the high-resolution mouth rendered image to obtain a speech-driven lip shape generation video.The technical scheme of the application can significantly reduce the training cost and effectively solve the lip shape jitter during silent sound.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a speech-driven lip shape generation method, device and medium. BACKGROUND

[0002] Speech-driven lip shape generation technology: refers to the technology of generating corresponding lip movement by analyzing and processing speech signals. This technology uses the correlation between the audio features of speech (such as pitch, volume, audio spectrum, etc.) and lip shape changes, and realizes accurate mapping from speech signals to lip movements through modeling and algorithm analysis. This technology is commonly used in animation production, virtual character interaction, spoken language recognition, etc., and can generate realistic lip shape changes that conform to the content of the speech, improving the expressiveness and interactivity of virtual characters.

[0003] Traditional lip synchronization methods usually focus on generating general lip shape sequences, relying on training on a large amount of speaking video data, expecting to generate applicable general lip shapes for all speaking videos. However, this method can still achieve certain effects when generating 1080P or even 2K resolution videos, but with the rapid development of media technology, the generation of 4K videos involves more complex pixel details and higher data storage costs. Continuing to use traditional technical solutions may result in loss of lip shape details, reduction of synchronization, and lip shape jitter when generating closed sounds, etc. In addition, the high computing power cost required to process 4K data also poses a serious challenge to the application of existing technology. SUMMARY

[0004] The purpose of the present application is to solve the problems of blurred lip shape details, loss of personal image, high training cost, and lip shape jitter in the process of generating 4K high-definition videos in the prior art, and to provide a speech-driven lip shape generation method, device and medium.

[0005] The above-mentioned purpose of the present application is achieved by the following technical solutions:

[0006] S1: obtaining the audio signal of the original target video;

[0007] S2: constructing an audio-expression coefficient mapping network; inputting the audio signal into the audio-expression coefficient mapping network for training to obtain the mixed deformation coefficient of the target character;

[0008] The audio-expression coefficient mapping network includes a feature extraction module, a self-attention enhancement module, and a personalized expression conversion module;

[0009] S3: based on the mixed deformation coefficient, constructing a 3D rendered face image of the target character;

[0010] S4: performing neural texture mapping on the 3D rendered face image to generate a feature map;

[0011] S5: rendering a mouth region of the 3D rendered face image through the feature map to obtain a high-resolution mouth rendered image;

[0012] S6: obtaining a speech-driven lip generation video through the high-resolution mouth rendered image and a background synthesis network.

[0013] Optionally, the audio-expression coefficient mapping network comprises four 2D convolution layers and three fully connected networks; each 2D convolution layer uses a 3x1 convolution kernel and has a time dimension step of 2.

[0014] Optionally, step S2 comprises:

[0015] S21: extracting audio features of the audio signal through a feature extraction module and a self-attention enhancement module; the audio features comprise time sequence information and pronunciation content; the feature extraction module is a speech recognition framework based on a recurrent neural network and uses a multi-scale convolution kernel;

[0016] S22: mapping the audio features to a facial expression space of the target person to obtain facial expression coefficients;

[0017] S23: mapping the facial expression coefficients to a personalized expression space of the target person in the facial expression space through a personalized expression conversion module to obtain mixed deformation coefficients of the target person;

[0018] S24: smoothing the facial expression space of the current frame through a time filter.

[0019] Optionally, step S23 comprises:

[0020] S231: learning a linear mapping matrix from an original target video of the target person;

[0021] S232: converting the facial expression coefficients to the mixed deformation coefficients of the target person through the personalized expression conversion module in combination with the linear mapping matrix.

[0022] Optionally, step S3 comprises:

[0023] S31: constructing a personalized 3D face model of the target person, specifically comprising: constructing a statistical 3D face model, adjusting a lip shape and an expression of the statistical 3D face model using the mixed deformation coefficients to obtain the personalized 3D face model;

[0024] S32: deforming a mouth region of the personalized 3D face model based on a head posture of the original target video;

[0025] S33: calculating UV texture coordinates of the personalized 3D face model; and mapping the deformed face of the target person to a 3D face mesh of the personalized 3D face model based on the UV texture coordinates to obtain a low-resolution 3D rendered face image.

[0026] Optionally, step S4 comprises:

[0027] S41: constructing a neural texture, specifically comprising: generating a high-dimensional neural texture by the 3D face mesh of the 3D rendered face image and the facial expression coefficients of the target person;

[0028] The neural texture is used for storing personalized facial details of the target person; the personalized facial details include skin texture, wrinkles, and dynamic change characteristics of the mouth;

[0029] S42: mapping the neural texture to a 2D image space of the 3D rendered face image by using the UV texture coordinates of the personalized 3D face model, combining a neural rendering network, and a multi-resolution dynamic texture mapping method to generate a feature map.

[0030] Optionally, step S5 comprises:

[0031] rendering the mouth region of the 3D rendered face image by the feature map and a lip-shaped neural rendering network based on a U-Net structure to obtain a high-resolution mouth rendered image;

[0032] The lip-shaped neural rendering network comprises a dynamic lip shape prediction module, a neural texture enhancement module, and an adaptive light rendering module.

[0033] Optionally, step S6 comprises:

[0034] S61: removing the mouth region of the original target video to obtain a background frame;

[0035] S62: repairing edges of the high-resolution mouth rendered image and the mouth region of the background frame by a background synthesis network to obtain a speech-driven lip shape generation video.

[0036] An electronic device comprises a processor, a memory, a user interface, and a network interface, the memory is used for storing instructions, the user interface and the network interface are used for communicating with other devices, and the processor is used for executing the instructions stored in the memory to enable the electronic device to perform a speech-driven lip shape generation method.

[0037] A computer-readable storage medium stores instructions, when the instructions are executed, a speech-driven lip shape generation method is performed.

[0038] The technical scheme provided by the present application has the beneficial effects that:

[0039] The application innovatively introduces a two-stage lip shape reconstruction model and a corresponding training method. In the first stage, a general mapping relationship between an audio sequence and a speaker facial expression coefficient is constructed; in the second stage, under the guidance of the mapping relationship, a single-person rendering reconstruction between the audio sequence and the lip shape sequence is established. The method can significantly reduce the training cost, generate a lip shape sequence highly matched with the target identity when driving the video, maximize the preservation of the pixel details of the original video, and effectively solve the lip shape jitter during silent sound. BRIEF DESCRIPTION OF DRAWINGS

[0040] The application will be further described below in conjunction with the accompanying drawings and embodiments. In the drawings:

[0041] Figure 1 is a flowchart of the embodiment of the application;

[0042] Figure 2 is a schematic diagram of the structure of an electronic device in the embodiment of the application. DETAILED DESCRIPTION

[0043] In order to have a clearer understanding of the technical features, objectives and effects of the application, the specific implementation modes of the application will be described in detail with reference to the accompanying drawings.

[0044] The embodiment of the application provides a speech-driven lip shape generation method.

[0045] Reference is made to Figure 1 , Figure 1 is a flowchart of a speech-driven lip shape generation method in the embodiment of the application, comprising:

[0046] S1: obtaining an audio signal of an original target video;

[0047] S2: constructing an audio-expression coefficient mapping network; inputting the audio signal into the audio-expression coefficient mapping network for training to obtain a mixed deformation coefficient of a target person;

[0048] The audio-expression coefficient mapping network comprises a feature extraction module, a self-attention enhancement module and a personalized expression conversion module;

[0049] S3: constructing a 3D rendered face image of the target person based on the mixed deformation coefficient;

[0050] S4: performing neural texture mapping on the 3D rendered face image to generate a feature map;

[0051] S5: rendering the mouth region of the 3D rendered face image through the feature map to obtain a high-resolution mouth rendered image;

[0052] S6: Obtain the speech-driven lip shape generation video by high-resolution mouth rendering image and background synthesis network.

[0053] The audio-expression coefficient mapping network comprises 4 layers of 2D convolution layers and 3 layers of fully connected networks; each 2D convolution layer uses a 3x1 convolution kernel and has a time dimension step of 2.

[0054] As an embodiment, a deep convolutional-fully connected hybrid neural network (i.e., an audio-expression coefficient mapping network) is designed. The main process of the network is as follows: first, 4 layers of 2D convolution layers are used to reduce the sampling in the time dimension and extract time-related features; a 3x1 convolution kernel is used with a time dimension step of 2 to reduce the time window size. Hierarchical feature extraction is performed to reduce the dimension from 16x29 to a compact representation of 1x64. Then, 3 layers of fully connected networks are used to map the 64-dimensional features to a 32-dimensional audio-expression vector. Finally, a general facial expression space is output, which is shared by multiple individuals.

[0055] Step S2 comprises:

[0056] S21: Extract audio features of the audio signal through a feature extraction module and a self-attention enhancement module; the audio features comprise time sequence information and pronunciation content; the feature extraction module is a speech recognition framework based on a recurrent neural network, and a multi-scale convolution kernel is used;

[0057] Compared with a traditional convolutional neural network using a fixed window size, the feature extraction module uses a multi-scale convolution kernel, so that the model can capture short-time phoneme information (consonants) and long-time syllable information (vowels) at the same time, thereby improving the accuracy of lip shape prediction.

[0058] As an embodiment, the pronunciation content refers to the actual textual meaning of the speech, and the time sequence information refers to the sequential relationship information in the time dimension of various factors before and after a segment of audio.

[0059] As an embodiment, the audio-expression coefficient mapping network further comprises a self-attention enhancement module. Since the speech component in the audio has strong time dependence, the relationship between different phonemes needs to be globally modeled, so the self-attention mechanism is introduced to enhance the modeling ability of the audio features. The query, key and value are generated respectively through linear transformation, and the attention score is calculated. Eight attention heads are used, each head learns different levels of audio features, and after splicing, the features are fused to improve the expression ability of the features. Finally, normalization and residual connection are performed, the original audio features and the self-attention output are added, and layer normalization is performed to improve the stability of the model and prevent gradient disappearance. Innovation: Compared with the traditional long short-term memory network which can only capture short-term time sequence information, the self-attention mechanism of this module can calculate the importance of features within the entire audio sequence, so that the model can pay attention to the long-distance dependence relationship between different phonemes, improve the matching degree of audio and lip shape, and especially in fast-paced speech, it can more accurately adjust the mouth movement.

[0060] As an embodiment, the input audio signal needs to be converted into a feature format suitable for the audio-expression coefficient mapping network for subsequent processing. First, data preprocessing is performed: the input audio signal is sampled at 16kHz, and a short-time Fourier transform is performed to obtain the frequency spectrum of the audio signal. A Hamming window is used to reduce spectral leakage, and a Mel filter is used to map the spectral information to the Mel frequency domain to extract 80-dimensional Mel spectral features. Then, feature normalization is performed: a logarithmic compression method is used to enhance the dynamic range of the audio features, and mean and variance normalization is performed to reduce the influence of environmental noise on the feature distribution. Then, deep time convolution is used, which uses a one-dimensional convolutional neural network with multiple layers of different size convolution kernels (3, 5, 7) to extract local time features of the audio, and combines with the maximum pooling layer to reduce the computational complexity and enhance the time modeling ability of the model.

[0061] As an embodiment, the audio feature extraction part is the starting step of the entire algorithm process, and the main task is to extract meaningful feature information from the input audio. First, the input audio is divided into multiple 20ms time windows, and each time window represents a small segment of audio signal. The time windows are processed using a deep speech recognition model, which analyzes the audio signal through its built-in recurrent neural network to extract corresponding character logical features, which are usually represented as a 29-dimensional feature vector containing pronunciation information for each time period in the audio. Then, the extracted character logical features are rearranged into a 16x29 feature matrix, with each column representing the features of the audio in a time window and each row representing the changes of the features in consecutive time windows. In this way, the timing information and pronunciation content in the audio signal are accurately converted into a matrix format as input to the subsequent audio-to-expression mapping neural network, providing rich speech feature information for further facial expression prediction.

[0062] S22: Map the audio features to the facial expression space of the target person to obtain facial expression coefficients;

[0063] The application adopts the above technical solutions, and the audio-expression coefficient mapping network includes an audio feature extraction module, a self-attention enhancement module, and a personalized expression conversion module.

[0064] The personalized expression conversion module can efficiently map local features in the audio signal to the facial expression space to generate accurate facial expression coefficients and drive the lip shape and expression of the target person. By extracting key timing features in the audio through a multi-layer convolutional network, the changes in phonemes and intonation in the audio signal can be effectively captured, thereby improving the accuracy and naturalness of the facial animation. In addition, the role of the fully connected layer ensures smooth conversion from audio to expression, so that different speech inputs can generate personalized expressions according to the characteristics of the target person. The design of the model can ensure high synchronization and stability during the process of driving facial animation with audio, significantly improve the accuracy of lip shape generation, and enhance the naturalness of the overall visual effect.

[0065] As an embodiment, the generation of audio-expression coefficients is a crucial step in the entire audio-driven facial animation process, aiming to convert the characteristics of audio signals into expression parameters of facial animation. First, the audio feature extraction part converts the audio signal into a feature matrix with timing information, and then the audio-expression coefficient mapping network is responsible for mapping these audio features to the facial expression space. The audio-expression coefficient mapping network is composed of multiple convolutional layers and fully connected layers, where the convolutional layers are used to extract local timing features of the audio signal, capturing the instantaneous changes and feature patterns in the audio, and the fully connected layers are responsible for further mapping these features to the expression coefficient space. During this process, the output of the audio-expression coefficient mapping network is a 32-dimensional vector, each dimension corresponding to a facial expression coefficient, which represents lip shape, eye movement, facial muscle activity, and other facial features. These expression coefficients will drive the deformation of the 3D face model, ensuring that the target person's facial expression matches the input audio content. In this way, the language information and intonation changes in the audio signal (intonation changes refer to the changes between consecutive phonemes, including the transition of emotional information contained in a sentence) can be accurately converted into facial expressions, making the generated lip shape highly synchronized and natural.

[0066] As an embodiment, the audio-expression coefficient mapping network adopts a combined architecture of deep convolutional neural networks and fully connected layers, aiming to map audio features to facial expression space. First, the network extracts local timing features in the audio signal through multiple convolutional layers, capturing the instantaneous changes of speech and dynamic features of phonemes. Then, the features output by the convolutional layer are further processed by the fully connected layer, converting these local features into 32-dimensional audio-expression coefficients representing the dynamic changes of the target person's lip shape, facial muscle movement, etc. To improve the stability and accuracy of the conversion, the convolutional layer in the network uses batch normalization technology to ensure that the model has high robustness and stability during the mapping of different audio features. This structure can effectively process the time variation information in the audio, ensuring the synchronization of facial expressions with audio.

[0067] S23: mapping the facial expression coefficients to the personalized expression space of the target person in the facial expression space through the personalized expression conversion module, to obtain the mixed deformation coefficients of the target person;

[0068] By adopting the above technical solutions, the traditional method usually needs a large amount of video data of the target person for training, while the present method only needs 2-3 minutes of video of the target person to learn the personalized transformation matrix, thereby efficiently adapting to new characters while maintaining high lip synchronization and style matching.

[0069] As an embodiment, since different people have different speaking styles, it is necessary to map the generic expression to the personalized expression of the target person. This conversion is achieved through a linear mapping matrix: in the training phase, a 76x32 mapping matrix is automatically learned from the video of the target person, which is used to convert the facial expression coefficients to the mixed morphing coefficients of the target person. The mapping matrix enables different target persons to share the generic audio facial expression space without losing personalized features (such as lip size, micro-expression, etc.), which greatly improves the identity information retention of the generated lip shape. The generated mixed morphing coefficients are used to drive the 3D face model of the target person in the second stage, generating lip shape synchronization animation.

[0070] S24: Smooth the facial expression space of the current frame through a time filter.

[0071] As an embodiment, a 1D convolutional neural network is used as a time filter, and the expression parameters of the last preset number of frames are input to calculate the optimal weighted time window to smooth the facial expression space of the current frame.

[0072] As an embodiment, since the expression generated frame by frame may produce jitter or incoherence, a neural network-based time filter is added, and the specific process is as follows: a 1D convolutional neural network is used as an adaptive filter, and the expression parameters of the last T frames (usually T = 8) are input. Calculate the optimal weighted time window to smooth the current frame expression. By normalizing the time weighting factor, the influence weight of different frames is more reasonable, ensuring that the lip animation is coherent and natural.

[0073] Step S23 includes:

[0074] S231: Learning a linear mapping matrix from the original target video of the target person;

[0075] S232: Through the personalized expression conversion module, combine the linear mapping matrix to convert the facial expression coefficients to the mixed morphing coefficients of the target person.

[0076] Step S3 includes:

[0077] S31: Constructing a personalized 3D face model of the target person, the specific steps including: constructing a statistical 3D face model, adjusting the lip shape and expression of the statistical 3D face model using the mixed morphing coefficients to obtain a personalized 3D face model;

[0078] S32: Deforming the mouth region of the personalized 3D face model based on the head pose of the original target video;

[0079] S33: calculating UV texture coordinates of the personalized 3D face model; mapping the deformed face of the target person to the 3D face grid of the personalized 3D face model based on the UV texture coordinates to obtain a low-resolution 3D rendered face image.

[0080] As an embodiment, after the audio-driven expression parameter calculation is completed, a 3D face model needs to be generated first, and the specific process is as follows: first, a statistical 3D face model is used, a mixed deformation coefficient is used to adjust the lip shape and expression to make it consistent with the audio input. Then, the mouth area is deformed through the personalized 3D face model of the target person, and the head pose (from the original target video) is applied. Then, the UV texture coordinates are calculated, and the face of the target person is mapped to the 3D grid for use in subsequent neural rendering. Finally, a low-resolution 3D rendered face image (containing mouth movement) is output.

[0081] Step S4 includes:

[0082] S41: constructing a neural texture, and the specific steps include: generating a high-dimensional neural texture through the 3D face grid of the 3D rendered face image and the facial expression coefficient of the target person;

[0083] The neural texture is used to store the personalized facial details of the target person; the personalized facial details include skin texture, wrinkles, and dynamic features of the mouth;

[0084] S42: using the UV texture coordinates of the personalized 3D face model, combining the neural rendering network and the multi-resolution dynamic texture mapping method, mapping the neural texture to the 2D image space of the 3D rendered face image to generate a feature map.

[0085] By adopting the technical solution, the traditional method usually uses a static texture image. The neural rendering network adopts multi-resolution dynamic texture mapping. The neural texture mapping converter (neural rendering network) of the present application effectively improves the visual realism of the generated facial animation, especially in the presentation of facial details, which can retain the unique facial features of the target person, such as skin texture, subtle facial movements, and wrinkles. Through the learning of high-dimensional neural texture, the model can accurately reproduce facial dynamics on the 3D face grid, greatly improving the delicacy and naturalness of the lip shape synchronous animation. The introduction of dilated convolution avoids common visual artifacts, reduces texture distortion, enhances the detail performance of the rendered image, and seamlessly connects each part of the facial animation. The converter can ensure that the visual effect of the facial animation is accurately matched with the audio, ensuring that the generated lip animation is not only synchronous but also realistic, significantly enhancing the realism and immersion of the virtual image.

[0086] The application improves the rendering effect by using the above technical solution through more refined mapping technology. UV coordinate mapping: generate UV texture coordinates for the target person's 3D face mesh, combine the target person's facial features with neural textures. Multi-resolution mapping: through multi-level mapping technology, ensure that different areas of the face (such as lips, eyes, nose) can dynamically adjust the texture according to the facial movement, expression change and muscle movement. This can avoid the texture stretching or deformation phenomenon in traditional mapping methods.

[0087] As an embodiment, the neural rendering module: completes the generation and mapping of neural textures, and the final task is to render these textures into target video frames. Traditional rendering methods rely on hardware-accelerated graphics rendering technology, while this module uses neural rendering technology to generate more natural and detailed images: neural rendering network: uses convolutional neural networks and dilated convolution to generate the final rendered image of the face, ensuring that details such as lip movement, skin lighting and shadow changes are realistically presented. Feature fusion: fuse the feature map from the neural texture with the background image of the target video, ensuring that facial details seamlessly integrate with the background, avoiding jarring boundaries or imperfections in generated images. Innovation: by introducing neural rendering, this module can fully consider skin lighting changes, facial detail dynamics, and lip shape changes during facial rendering, generating high-quality facial images. Compared to traditional rendering methods, neural rendering can provide higher visual realism and detail restoration.

[0088] As an embodiment, in order to enhance the realism of 3D rendered faces, neural texture technology is used, and the process of this step is as follows: first, construct a neural texture: learn a 256x256x16-dimensional neural texture on the 3D face mesh, which is used to store the personalized facial details of the target person (such as skin texture, wrinkles, mouth features). Then through texture mapping, use the UV coordinates of the 3D face model to map the neural texture to the 2D image space and generate feature maps. These feature maps are then input into the neural rendering network to generate the final realistic image.

[0089] As an embodiment, neural texture mapping is a crucial step in achieving high-fidelity 3D face rendering. Its main role is to store and reconstruct the personalized facial details of the target person, making the generated lip animation visually more realistic and natural. First, after the deformation of the mouth region of the personalized 3D face model is completed, neural textures need to be generated for the target person's face. This is a high-dimensional feature representation, usually stored as a 256x256x16 tensor, where each pixel point contains 16-dimensional feature information, not just traditional RGB color values. Then, through UV texture mapping, these neural textures are projected onto the 3D face mesh, so that the skin texture, wrinkles, and details of the mouth region of the target person can be accurately restored.

[0090] As an embodiment, neural textures are input into a neural rendering network, which uses convolutional layers to gradually decode the high-dimensional features in the neural textures and generate the final high-resolution mouth rendering image. This process allows the neural network to automatically learn and supplement the complex details of the mouth, such as tooth and tongue lighting effects, and changes in facial shadows, ensuring that the lip animation is not only synchronized with the audio, but also highly realistic in vision. Finally, neural texture mapping ensures that the generated facial animation can retain the unique facial features of the target person, making the entire 3D rendering result more realistic and natural.

[0091] Step S5 includes:

[0092] The mouth region of the 3D rendered face image is rendered by the feature map and the lip neural rendering network based on the U-Net structure to obtain a high-resolution mouth rendering image.

[0093] The lip neural rendering network includes a dynamic lip prediction module, a neural texture enhancement module, and an adaptive lighting rendering module.

[0094] The lip neural renderer significantly improves the accuracy and realism of the generated facial animation, especially in the rendering effect of the mouth region, which can accurately reproduce the movement of the lips, the activity of the tongue, and the subtle changes of the lips, thereby ensuring the naturalness and synchronization of facial expressions. Through the combination of dilated convolution and multi-layer convolution, the renderer effectively reduces artifacts and distortions in the image, maintaining high-quality visual effects, especially in complex lip movements. This makes the generated facial animation not only visually realistic, but also maintains good synchronization with the audio signal, providing a smoother and more natural visual experience. At the same time, the renderer can seamlessly integrate with the background of the target person, ensuring the visual coordination of the face and the background, thereby enhancing the overall video immersion.

[0095] As an embodiment, after the neural texture mapping is completed, the mouth region is first rendered to ensure accurate and natural mouth movement. The feature map (16-dimensional channel) generated by the neural texture is used with the neural rendering network, and dilated convolution is used instead of pooling layers to maintain spatial resolution and reduce visual artifacts such as blurring or distortion. Finally, a high-resolution mouth rendering image is generated, where the lip shape is completely synchronized with the audio.

[0096] As an embodiment, traditional lip shape rendering methods usually rely on static texture mapping based on 3D models or 2D generative adversarial network synthesized mouth animation. However, these methods have the following problems:

[0097] (1) Static texture cannot adapt to dynamic lip shape changes: Traditional methods often use predefined lip shape textures, resulting in stiff and unnatural lip shapes during rapid speech changes.

[0098] (2) 2D generative adversarial network method lacks structural consistency: Although 2D-based methods can synthesize mouth animation, they cannot guarantee the consistency of lip shapes under different angles and lighting conditions, and are prone to unnatural mouth distortion or blurring.

[0099] (3) Insufficient synchronization between lip shape and audio: Traditional methods may ignore the micro features of the audio during lip shape generation, resulting in subtle synchronization problems between the lips and the pronunciation.

[0100] To solve these problems, the lip shape neural renderer uses a dynamic neural rendering method based on 3D structure, which ensures structural consistency of the lip shape under different angles and lighting conditions, and ensures high synchronization between the lip shape and the audio. By combining temporal consistency modeling, adaptive lighting correction, and high-resolution feature generation, the model can generate more natural and high-fidelity mouth animation. The core algorithm of the lip shape neural renderer consists of the following three key modules: dynamic lip shape prediction module, neural texture enhancement module, and adaptive lighting rendering module.

[0101] (1) Dynamic lip shape prediction module

[0102] The main task of the module is to predict realistic mouth movements and reconstruct them in 3D space according to the input audio and expression parameters. A dual-stream temporal convolutional network is used to model the timing of audio features and expression parameters, ensuring that the lip shape prediction results are highly synchronized with the audio. The audio stream is responsible for learning the temporal variation pattern of speech, capturing the transition features between phonemes, and ensuring that the lip shape does not jump abruptly. The expression stream is responsible for learning the individualized style of the target person's lip shape, ensuring that the generated lip shape conforms to the target person's speaking manner. By fusing the deformation parameters, the individualized mouth mesh of the target person is generated, and the mouth movement simulation in 3D space is performed. The dynamic deformation of each frame of the mouth mesh is calculated to ensure that the lip animation is smooth and consistent with the speech content.

[0103] Innovations:

[0104] Compared with traditional generative adversarial networks or long short-term memory network methods, this module uses a dual-stream temporal convolutional network + fusion deformation structure modeling to accurately capture the relationship between audio and lip shape and ensure the consistency of lip shape at different angles, avoiding the common lip shape distortion problem of generative adversarial network methods.

[0105] (2) Neural texture enhancement module

[0106] To improve the detail quality of the mouth animation, this module uses neural texture to generate individualized mouth, teeth, and tongue movement features: through a 256x256x16-dimensional neural texture feature map, the mouth, teeth, and tongue regions are individually modeled to ensure that the details of the lip shape area are consistent with the original video of the target person. The texture not only contains color information, but also contains normal information, lighting features, and mouth movement patterns to ensure the naturalness of the lip shape under different lighting conditions. Finally, a temporal long short-term memory network is used to process the neural texture parameters to ensure that the mouth, teeth, and tongue change smoothly between different frames without sudden texture jumps or visual artifacts.

[0107] Innovations:

[0108] Compared with traditional static texture mapping, this module uses dynamic neural texture + temporal long short-term memory network texture adjustment to ensure smooth and natural dynamic changes in the mouth area and avoid the problem of incoherent mouth movement.

[0109] (3) Adaptive lighting rendering module

[0110] After the lip shape prediction and neural texture enhancement, the last task is to ensure that the mouth region can be naturally integrated into the target person's video. A conditional generation network is used to predict the global lighting information of the target video, and the mouth region is compensated for lighting through ambient light mapping to conform to the overall lighting style of the target video. Through the transparency mask and background synthesis network, the generated lip region can be seamlessly integrated into the target video, avoiding the problem of abrupt boundaries or color mismatch between the lip region and the surrounding face.

[0111] Innovation: Compared with traditional video synthesis methods, this module uses a conditional generative adversarial network for lighting matching, which can dynamically adjust the lighting of the lip region, making the lip animation perfectly adapt to different target video scenes, greatly improving the realism of the rendering.

[0112] Step S6 includes:

[0113] S61: Remove the mouth region of the original target video to obtain a background frame;

[0114] S62: Repair the edge of the high-resolution mouth rendering image and the mouth region of the background frame through the background synthesis network to obtain a speech-driven lip shape generation video.

[0115] As an embodiment, since the mouth region rendered in the first stage may have edge mismatch problems with the original target video, background synthesis is needed to naturally integrate the mouth into the target video. The mouth region image generated by the first stage neural rendering and the background frame of the target video (with the mouth region removed) are used to provide complete facial information. A background synthesis network is combined: a similar structure as the first stage is used, but the input is 6-dimensional (mouth rendering + target video background). This network is used to repair the edge transition of the mouth region, ensuring the natural integration of the mouth and face. The final output is the final complete frame, with the mouth region seamlessly integrated into the original video.

[0116] As an embodiment, a large number of speaker videos are crawled from major video platforms to train a one-stage model, and a general audio-expression coefficient mapping network with strong generalization ability is trained. Then, for a specific high-resolution speaker image-driven video, the general model is fine-tuned and personalized image conversion is performed. Finally, the fine-tuned one-stage general model is used to guide the training of the two-stage model, which can greatly reduce the training cost. Since the two-stage model is completely trained on the driving video, it has strong lip shape synchronization and detail restoration, and can maximize the preservation of personal identity information.

[0117] As an embodiment, when the driving audio is acquired, the audio is preprocessed in the same way as the training model, and then the two-stage end-to-end inference is directly performed to obtain the complete audio-synchronized lip shape sequence picture sequence. The sequence is connected into a video by using opencv or ffmpeg, and the complete generated video is obtained by converting the frame rate according to the requirement.

[0118] The application also discloses an electronic device. Referring to Figure 2 , Figure 2 is a structural schematic diagram of an electronic device disclosed by an embodiment of the application. The electronic device 500 can include at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0119] The communication bus 502 is configured to realize the connection and communication between the components.

[0120] The user interface 503 can include a display screen, and optionally, the user interface 503 can further include a standard wired interface and a wireless interface.

[0121] The network interface 504 can optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0122] The application also discloses a computer readable storage medium, which stores a plurality of instructions adapted to be loaded by a processor to execute the above-described lip shape generation method based on voice driving.

[0123] The above are only exemplary embodiments of the disclosure, and cannot limit the scope of the disclosure. That is, any equivalent changes and modifications made according to the teachings of the disclosure are still within the scope of the disclosure.

[0124] The application is intended to cover any variations, uses, or adaptive changes of the disclosure that follow the general principles of the disclosure and include common knowledge or conventional technical means in the technical field of the disclosure not recorded in the disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the disclosure are defined by the claims.

Claims

1. A speech-driven lip-shape generation method, characterized in that, The method includes the following steps: S1: Acquire the audio signal of the original target video; S2: Construct an audio-expression coefficient mapping network; The audio signal is input into the audio-expression coefficient mapping network for training to obtain the hybrid deformation coefficients of the target character; The audio-expression coefficient mapping network includes: a feature extraction module, a self-attention enhancement module, and a personalized expression conversion module; Step S2 includes: S21: Audio features of the audio signal are extracted through the feature extraction module and the self-attention enhancement module; the audio features include: temporal information and pronunciation content; the feature extraction module is a speech recognition framework based on recurrent neural networks, using multi-scale convolutional kernels; S22: Map the audio features to the facial expression space of the target person to obtain facial expression coefficients; S23: Through the personalized expression conversion module, the facial expression coefficients are mapped to the personalized expression space of the target person in the facial expression space to obtain the mixed deformation coefficients of the target person. S24: Smooth the facial expression space of the current frame using a temporal filter. S3: Construct a 3D rendered facial image of the target person based on the hybrid deformation coefficient; Step S3 includes: S31: Construct a personalized 3D facial model of the target person. The specific steps include: constructing a statistical 3D face model, adjusting the lip shape and expression of the statistical 3D face model using a hybrid deformation coefficient, and obtaining a personalized 3D face model. S32: Based on the head pose of the original target video, deform the mouth area of ​​the personalized 3D facial model; S33: Calculate the UV texture coordinates of the personalized 3D facial model; based on the UV texture coordinates, map the deformed face of the target person to the 3D facial mesh of the personalized 3D facial model to obtain a low-resolution 3D rendered facial image. S4: Perform neural texture mapping on the 3D rendered facial image to generate a feature map; Step S4 includes: S41: Constructing neural textures, the specific steps of which include: generating high-dimensional neural textures by using a 3D facial mesh of a 3D rendered facial image and the facial expression coefficients of the target person; Neural textures are used to store personalized facial details of a target person; personalized facial details include: skin texture, wrinkles, and dynamic changes in the mouth. S42: Using the UV texture coordinates of a personalized 3D facial model, combined with a neural rendering network and a multi-resolution dynamic texture mapping method, the neural texture is mapped to the 2D image space of the 3D rendered facial image to generate a feature map. S5: Render the mouth region of the 3D rendered facial image using feature maps to obtain a high-resolution mouth rendering image; S6: By combining a high-resolution mouth rendering image with a background synthesis network, a speech-driven lip-shape generation video is obtained.

2. The speech-driven lip-shape generation method as described in claim 1, characterized in that, The audio-expression coefficient mapping network includes: 4 layers of 2D convolutional layers and 3 layers of fully connected networks; each 2D convolutional layer uses a 3×1 convolutional kernel and has a temporal step size of 2.

3. The speech-driven lip-shape generation method as described in claim 1, characterized in that, Step S23 includes: S231: Learn a linear mapping matrix from the target person's original target video; S232: Through the personalized expression conversion module, combined with the linear mapping matrix, facial expression coefficients are converted into mixed deformation coefficients of the target person.

4. The speech-driven lip-shape generation method as described in claim 1, characterized in that, Step S5 includes: The mouth region of a 3D rendered facial image is rendered using feature maps and a lip-shaped neural rendering network based on the U-Net structure, resulting in a high-resolution mouth rendering image. The lip shape neural rendering network includes: a dynamic lip shape prediction module, a neural texture enhancement module, and an adaptive lighting rendering module.

5. The speech-driven lip-shape generation method as described in claim 1, characterized in that, Step S6 includes: S61: Remove the mouth area from the original target video to obtain the background frame; S62: By using a background synthesis network, the edges of the mouth region in the high-resolution mouth rendering image and the background frame are repaired to obtain a speech-driven lip-shape generation video.

6. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform the method as described in any one of claims 1-5.

Images