Lip type generation method and device based on voice driving and medium

Abstract

The invention provides a lip shape generation method and device based on voice driving and a medium, and relates to the field of computer vision, and the method comprises the steps: obtaining an audio signal of an original target video; constructing an audio-expression coefficient mapping network; inputting the audio signal into an audio-expression coefficient mapping network for training to obtain a mixed deformation coefficient of the target person; the audio-expression coefficient mapping network comprises a feature extraction module, a self-attention enhancement module and a personalized expression conversion module; constructing a 3D rendering face image of the target person; performing neural texture mapping on the 3D rendered face image to generate a feature map; rendering a mouth region of the 3D rendering face image through the feature map to obtain a high-resolution mouth rendering image; and through the high-resolution mouth rendering image and in combination with the background synthesis network, a lip type generation video based on voice driving is obtained. According to the technical scheme, the training cost can be remarkably reduced, and lip-shaped shaking during silence is effectively avoided.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a method, device, and medium for generating lip shapes based on speech-driven operation. Background Art

[0002] Speech-driven lip shape generation technology: This refers to technology that generates corresponding lip movements by analyzing and processing speech signals. This technology leverages the correlation between speech audio features (such as pitch, volume, and audio spectrum) and lip shape changes. Through modeling and algorithmic analysis, it achieves a precise mapping from speech signals to lip movements. This technology is commonly used in animation production, virtual character interaction, spoken language recognition, and other fields. It can generate realistic lip shape changes that match the speech content, enhancing the expressiveness and interactivity of virtual characters.

[0003] Traditional lip synchronization methods typically focus on generating universal lip shape sequences, relying on training on large amounts of speech video data, hoping to generate universal lip shapes applicable to all speech videos. However, this method can still achieve certain results when generating videos with 1080P or even 2K resolution, but with the rapid development of media technology, the generation of 4K video involves more complex pixel details and higher data storage costs. Continuing to use traditional technical solutions may lead to problems such as loss of lip shape details, reduced synchronization, and lip jitter when generating closed-mouth sounds. In addition, the high computing power required to process 4K data also poses severe challenges to the application of existing technologies. Summary of the Invention

[0004] The purpose of the present invention is to provide a voice-driven lip shape generation method, device and medium in order to solve the problems of blurred lip shape details, loss of personal image, high training cost and lip shape jitter that occur in the existing technology during the generation of 4K high-definition video.

[0005] The above-mentioned purpose of this application is achieved through the following technical solutions: S1: Get the audio signal of the original target video; S2: Construct an audio-expression coefficient mapping network; input the audio signal into the audio-expression coefficient mapping network for training to obtain the mixed deformation coefficients of the target person; The audio-expression coefficient mapping network includes: feature extraction module, self-attention enhancement module and personalized expression conversion module; S3: Construct a 3D rendered facial image of the target person based on the blend deformation coefficients; S4: Perform neural texture mapping on the 3D rendered facial image to generate feature maps; S5: Render the mouth area of ​​the 3D rendered facial image using the feature map to obtain a high-resolution mouth rendered image; S6: By combining high-resolution mouth rendering images with a background synthesis network, we obtain a speech-driven lip shape generation video.

[0006] Optionally, the audio-expression coefficient mapping network includes: 4 layers of 2D convolutional layers and 3 layers of fully connected networks; each of the 2D convolutional layers uses a 3×1 convolution kernel and a time dimension step size of 2.

[0007] Optionally, step S2 includes: S21: Extract audio features of the audio signal through the feature extraction module and the self-attention enhancement module. Audio features include: timing information and pronunciation content. The feature extraction module is a speech recognition framework based on a recurrent neural network and uses multi-scale convolution kernels. S22: Map the audio features to the facial expression space of the target person to obtain the facial expression coefficient; S23: Mapping the facial expression coefficients to the personalized expression space of the target person in the facial expression space through the personalized expression conversion module to obtain the mixed deformation coefficients of the target person; S24: Smoothing the facial expression space of the current frame through a temporal filter.

[0008] Optionally, step S23 includes: S231: learning a linear mapping matrix from the original target video of the target person; S232: The facial expression coefficients are converted into mixed deformation coefficients of the target person through the personalized expression conversion module in combination with the linear mapping matrix.

[0009] Optionally, step S3 includes: S31: constructing a personalized 3D facial model of the target person, specifically comprising: constructing a statistical 3D face model, and adjusting the lip shape and expression of the statistical 3D face model using a blending deformation coefficient to obtain a personalized 3D face model; S32: deforming the mouth area of ​​the personalized 3D facial model based on the head pose of the original target video; S33: Calculate UV texture coordinates of the personalized 3D facial model; based on the UV texture coordinates, map the deformed face of the target person to the 3D facial mesh of the personalized 3D facial model to obtain a low-resolution 3D rendered facial image.

[0010] Optionally, step S4 includes: S41: Constructing a neural texture, specifically comprising: generating a high-dimensional neural texture by using a 3D facial mesh of a 3D rendered facial image and the facial expression coefficients of the target person; Neural textures are used to store the target person’s personalized facial details, including skin texture, wrinkles, and dynamic changes in the mouth. S42: Using the UV texture coordinates of the personalized 3D facial model, combined with a neural rendering network and a multi-resolution dynamic texture mapping method, the neural texture is mapped to the 2D image space of the 3D rendered facial image to generate a feature map.

[0011] Optionally, step S5 includes: The mouth area of ​​the 3D rendered facial image is rendered using feature maps and a U-Net-based lip neural rendering network to obtain a high-resolution mouth rendering image. The lip shape neural rendering network includes: dynamic lip shape prediction module, neural texture enhancement module and adaptive lighting rendering module.

[0012] Optionally, step S6 includes: S61: remove the mouth area of ​​the original target video to obtain a background frame; S62: Through the background synthesis network, the edges of the mouth area of ​​the high-resolution mouth rendering image and the background frame are repaired to obtain a speech-driven lip shape generation video.

[0013] An electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs a lip shape generation method based on voice drive.

[0014] A computer-readable storage medium stores instructions. When the instructions are executed, a lip shape generation method based on speech drive is performed.

[0015] The beneficial effects of the technical solution provided by this application are: This paper innovatively introduces a two-stage lip shape reconstruction model and corresponding training method. In the first stage, a universal mapping relationship is established between the audio sequence and the speaker's facial expression coefficients. In the second stage, guided by this mapping relationship, a single-person rendering reconstruction between the audio sequence and the lip shape sequence is established. This method significantly reduces training costs while generating a lip shape sequence that closely matches the target identity when driving the video, maximally preserving the pixel details of the original video and effectively addressing lip jitter during silence. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present application will be further described below with reference to the accompanying drawings and embodiments, in which: Figure 1 is a flow chart in an embodiment of the present application; Figure 2 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to have a clearer understanding of the technical features, purposes and effects of this application, the specific implementation methods of this application are now described in detail with reference to the accompanying drawings.

[0018] The embodiments of the present application provide a lip shape generation method based on speech drive.

[0019] Please refer to Figure 1 , Figure 1 This is a flow chart of a method for generating lip shapes based on speech drive in an embodiment of the present application, including: S1: Get the audio signal of the original target video; S2: Construct an audio-expression coefficient mapping network; input the audio signal into the audio-expression coefficient mapping network for training to obtain the mixed deformation coefficients of the target person; The audio-expression coefficient mapping network includes: feature extraction module, self-attention enhancement module and personalized expression conversion module; S3: Construct a 3D rendered facial image of the target person based on the blend deformation coefficients; S4: Perform neural texture mapping on the 3D rendered facial image to generate feature maps; S5: Render the mouth area of ​​the 3D rendered facial image using the feature map to obtain a high-resolution mouth rendered image; S6: By combining high-resolution mouth rendering images with a background synthesis network, we obtain a speech-driven lip shape generation video.

[0020] The audio-expression coefficient mapping network includes: 4 layers of 2D convolutional layers and 3 layers of fully connected networks; each 2D convolutional layer uses a 3×1 convolution kernel and a time dimension step size of 2.

[0021] As an embodiment, a deep convolutional-fully connected hybrid neural network (i.e., audio-expression coefficient mapping network) is designed. The main process of the network is as follows: first, through 4 layers of 2D convolution layers, the time dimension is downsampled to extract time-related features: a 3×1 convolution kernel is used with a step size of 2 in the time dimension to reduce the time window size. Hierarchical feature extraction is carried out, reducing the dimension from 16×29 to a compact representation of 1×64 dimensions. After a 3-layer fully connected network, the 64-dimensional features are mapped to a 32-dimensional audio-expression vector. The final output is a universal facial expression space that is shared by multiple individuals.

[0022] Step S2 includes: S21: Extract audio features of the audio signal through the feature extraction module and the self-attention enhancement module. Audio features include: timing information and pronunciation content. The feature extraction module is a speech recognition framework based on a recurrent neural network and uses multi-scale convolution kernels. By adopting the above-mentioned technical solution, compared with the traditional convolutional neural network with a fixed window size, the feature extraction module of this application adopts a multi-scale convolution kernel, so that the model can simultaneously capture short-time phoneme information (consonants) and long-time syllable information (vowels), thereby improving the accuracy of lip shape prediction.

[0023] As an embodiment, the pronunciation content refers to the actual textual meaning of the speech, and the timing information refers to the sequential relationship information of various factors before and after a section of audio in the time dimension.

[0024] As an embodiment, the audio-expression coefficient mapping network also includes a self-attention enhancement module. Since the speech components in the audio have strong time dependence, the relationship between different phonemes needs to be modeled globally. Therefore, the self-attention mechanism is introduced to enhance the modeling ability of audio features. The query, key and value are generated respectively through linear transformation, and the attention score is calculated. Eight attention heads are used, each head learns audio features at a different level, and feature fusion is performed after splicing to improve the expressiveness of the features. Finally, normalization and residual connection are performed, the original audio features are added to the self-attention output, and layer normalization is performed to improve the stability of the model and prevent gradient disappearance. Innovation: Compared with the traditional long short-term memory network that can only capture short-term temporal information, the self-attention mechanism of this module can calculate the importance of features across the entire audio sequence, allowing the model to focus on the long-range dependency between different phonemes, improve the matching degree between audio and lip shape, especially in fast-paced speech, and more accurately adjust mouth movements.

[0025] As an embodiment, the input audio signal needs to be converted into a feature format suitable for the audio-expression coefficient mapping network for subsequent processing. Data preprocessing is first performed: the input audio signal adopts a 16kHz sampling rate and performs a short-time Fourier transform to obtain a spectrum of the audio signal. A Hamming window is used to reduce spectrum leakage, and a Mel filter is performed to map the spectrum information to the Mel frequency domain to extract 80-dimensional Mel spectrum features. Feature normalization is then performed: a logarithmic compression method is used to enhance the dynamic range of the audio features, and mean and variance normalization is performed to reduce the impact of environmental noise on the feature distribution. Then, deep temporal convolution is used, using a one-dimensional convolutional neural network and multiple layers of convolution kernels of different sizes (3, 5, 7) to extract the local temporal features of the audio, and combined with a maximum pooling layer to reduce computational complexity while enhancing the temporal modeling capability of the model.

[0026] As an example, the audio feature extraction part is the starting step of the entire algorithm process, and its main task is to extract meaningful feature information from the input audio. First, the input audio is divided into multiple 20ms time windows, each time window representing a small segment of audio signal. These time windows are processed using the deep speech recognition model. The deep speech recognition model analyzes the audio signal through its built-in recurrent neural network and extracts the corresponding character logical features, which are usually represented as a 29-dimensional feature vector that contains the pronunciation information of each time period in the audio. Next, these extracted character logical features are rearranged into a 16×29 feature matrix, where each column represents the feature of the audio in a time window, and each row represents the change of the feature in the continuous time window. In this way, the timing information and pronunciation content in the audio signal are accurately converted into a matrix format, which serves as the input of the subsequent audio-to-expression mapping neural network, providing rich speech feature information for further facial expression prediction.

[0027] S22: Map the audio features to the facial expression space of the target person to obtain the facial expression coefficient; By adopting the above technical solution, the audio-expression coefficient mapping network of this application includes: an audio feature extraction module, a self-attention enhancement module and a personalized expression conversion module.

[0028] By adopting the above technical solution, the personalized expression conversion module of this application can efficiently map the local features in the audio signal to the facial expression space, generate accurate facial expression coefficients, and drive the lip shape and expression of the target character. By extracting key timing features in the audio through a multi-layer convolutional network, it is possible to effectively capture the phonemes and intonation changes in the audio signal, thereby improving the accuracy and naturalness of facial animation. In addition, the role of the fully connected layer ensures a smooth conversion from audio to expression, so that different voice inputs can generate personalized expressions based on the characteristics of the target character. The design of this model can ensure that in the process of audio-driven facial animation, it shows high synchronization and stability, significantly improves the accuracy of lip shape generation, and enhances the naturalness of the overall visual effect.

[0029] As an example, the generation of audio-to-expression coefficients is a crucial step in the entire audio-driven facial animation process, aiming to convert audio signal features into expression parameters for facial animation. First, the audio feature extraction component converts the audio signal into a feature matrix containing temporal information. Next, the audio-to-expression coefficient mapping network is responsible for mapping these audio features into the facial expression space. The audio-to-expression coefficient mapping network consists of multiple convolutional layers and fully connected layers. The convolutional layers extract local temporal features of the audio signal, capturing transient changes and characteristic patterns in the audio, while the fully connected layers further map these features into the expression coefficient space. The output of the audio-to-expression coefficient mapping network is a 32-dimensional vector, with each dimension corresponding to a facial expression coefficient. The facial expression coefficient represents facial features such as lip shape, eye movement, and facial muscle activity. These expression coefficients drive the deformation of the 3D face model, ensuring that the target person's facial expression matches the input audio content. In this way, the language information and intonation changes in the audio signal (intonation changes refer to the changes in the connection between the previous and subsequent phonemes, including the shift in the emotional information contained in a sentence) can be accurately converted into facial expressions, making the generated lip shape highly synchronized and natural.

[0030] As an embodiment, the audio-expression coefficient mapping network adopts a combined architecture of deep convolutional neural networks and fully connected layers, aiming to map audio features to facial expression space. First, the network extracts local temporal features in the audio signal through multiple convolutional layers to capture the instantaneous changes in speech and the dynamic characteristics of phonemes. Then, the features output by the convolutional layer are further processed by the fully connected layer to convert these local features into 32-dimensional audio-expression coefficients, which represent the dynamic changes of the target person's lip shape, facial muscle movements, etc. In order to improve the stability and accuracy of the conversion, the convolutional layer in the network uses batch normalization technology to ensure that the model has high robustness and stability in the mapping process of different audio features. This structure can effectively process the time-varying information in the audio and ensure the synchronization of facial expressions and audio.

[0031] S23: Mapping the facial expression coefficients to the personalized expression space of the target person in the facial expression space through the personalized expression conversion module to obtain the mixed deformation coefficients of the target person; By adopting the above-mentioned technical solution, the present application uses a method that only requires 2-3 minutes of target person video to learn a personalized transformation matrix, thereby efficiently adapting to new people while maintaining high lip synchronization and style matching.

[0032] As an embodiment, since different people have different speaking styles, it is necessary to map universal expressions to personalized expressions of the target person. This conversion is achieved through a linear mapping matrix: during the training phase, a 76×32 mapping matrix is ​​automatically learned from the target person's video to convert facial expression coefficients into mixed deformation coefficients of the target person. This mapping matrix enables different target persons to share a universal audio facial expression space without losing personalized features (such as lip size, micro-expressions, etc.), which can greatly improve the identity information retention of the generated lip shape. The generated mixed deformation coefficients are used to drive the target person's 3D face model in a two-stage process to generate lip synchronization animation.

[0033] S24: Smoothing the facial expression space of the current frame through a temporal filter.

[0034] As an embodiment, a 1D convolutional neural network is used as a temporal filter, the expression parameters of the most recent preset number of frames are input, the optimal weighted time window is calculated, and the facial expression space of the current frame is smoothed.

[0035] As an embodiment, since the expressions generated frame by frame may produce jitter or incoherence, a neural network-based time filter is added here. The specific process is as follows: a 1D convolutional neural network is used as an adaptive filter, and the expression parameters of the most recent T frames (usually T=8) are input. The optimal weighted time window is calculated to smooth the expression of the current frame. The time weighting factor is calculated by normalization to make the influence weights of different frames more reasonable, ensuring that the lip animation is coherent and natural.

[0036] Step S23 includes: S231: learning a linear mapping matrix from the original target video of the target person; S232: The facial expression coefficients are converted into mixed deformation coefficients of the target person through the personalized expression conversion module in combination with the linear mapping matrix.

[0037] Step S3 includes: S31: constructing a personalized 3D facial model of the target person, specifically comprising: constructing a statistical 3D face model, and adjusting the lip shape and expression of the statistical 3D face model using a blending deformation coefficient to obtain a personalized 3D face model; S32: deforming the mouth area of ​​the personalized 3D facial model based on the head pose of the original target video; S33: Calculate UV texture coordinates of the personalized 3D facial model; based on the UV texture coordinates, map the deformed face of the target person to the 3D facial mesh of the personalized 3D facial model to obtain a low-resolution 3D rendered facial image.

[0038] As an embodiment, after the audio-driven expression parameters are calculated, a 3D facial model needs to be generated first. The specific process is as follows: First, a statistical 3D face model is used, and the lip shape and expression are adjusted using the mixed deformation coefficients to make them consistent with the audio input. Then, the mouth area is deformed using the personalized 3D facial model of the target person, and the head posture (from the original target video) is applied. The UV texture coordinates are then calculated and the target person's face is mapped to a 3D mesh for use in subsequent neural rendering. Finally, a low-resolution 3D rendered facial image (including mouth movement) is output.

[0039] Step S4 includes: S41: Constructing a neural texture, specifically comprising: generating a high-dimensional neural texture by using a 3D facial mesh of a 3D rendered facial image and the facial expression coefficients of the target person; Neural textures are used to store the target person’s personalized facial details, including skin texture, wrinkles, and dynamic changes in the mouth. S42: Using the UV texture coordinates of the personalized 3D facial model, combined with a neural rendering network and a multi-resolution dynamic texture mapping method, the neural texture is mapped to the 2D image space of the 3D rendered facial image to generate a feature map.

[0040] This application adopts the above-mentioned technical solution. Traditional methods usually use static texture images. The neural rendering network uses multi-resolution dynamic texture mapping. The neural texture mapping converter (neural rendering network) of this application effectively improves the visual realism of generated facial animation, especially in the presentation of facial details, and can preserve the unique facial features of the target person, such as skin texture, subtle facial movements and wrinkles. By learning high-dimensional neural textures, the model can accurately reproduce facial dynamics on a 3D facial mesh, greatly improving the delicacy and naturalness of lip-sync animation. The introduction of dilated convolution avoids common visual artifacts, reduces texture distortion, enhances the detail of the rendered image, and makes the various parts of the facial animation seamlessly connected. The converter can ensure that the visual effects of facial animation are precisely synchronized with the audio, ensuring that the generated lip animation is not only synchronized but also realistic, significantly enhancing the realism and immersion of the virtual image.

[0041] This application utilizes the aforementioned technical solutions to improve rendering quality through more sophisticated mapping techniques. UV coordinate mapping: UV texture coordinates are generated from the target person's 3D facial mesh, combining the target person's facial features with the neural texture. Multi-resolution mapping: This uses multi-level mapping techniques to ensure that different facial regions (such as the lips, eyes, and nose) dynamically adjust their textures based on facial movement, expression changes, and muscle movements. This avoids the stretching or deformation of textures that can occur with traditional mapping methods.

[0042] As an embodiment, the neural rendering module: completes the generation and mapping of neural textures, and the final task is to render these textures into the target video frame. Traditional rendering methods rely on hardware-accelerated graphics rendering technology, while this module uses neural rendering technology to generate more natural and detailed images: Neural rendering network: uses convolutional neural networks and dilated convolutions to generate the final rendered image of the face, ensuring that details such as lip movement, skin lighting and shadow changes can be realistically presented. Feature fusion: The feature map from the neural texture is fused with the background image of the target video to ensure that the facial details are seamlessly connected to the background, avoiding abrupt boundaries or flaws when generating the image. Innovation: By introducing neural rendering, this module can fully consider the skin lighting changes, facial details and dynamics, and lip shape changes during the facial rendering process to generate high-quality facial images. Compared with traditional rendering methods, neural rendering can provide higher visual realism and detail restoration.

[0043] As an example, neural texturing technology is used to enhance the realism of 3D facial rendering. The process for this step is as follows: First, a neural texture is constructed: a 256×256×16-dimensional neural texture is learned on the 3D facial mesh to store the target person's personalized facial details (such as skin texture, wrinkles, and mouth features). Then, through texture mapping, the neural texture is mapped to 2D image space using the UV coordinates of the 3D facial model, and feature maps are generated. These feature maps are then input into the neural rendering network to generate the final realistic image.

[0044] As an example, neural texture mapping is a key step in achieving high-fidelity 3D face rendering. Its primary function is to store and reconstruct the target person's personalized facial details, making the generated lip animation more visually realistic and natural. First, after the personalized 3D facial model's mouth area is deformed, a neural texture is generated for the target person's face. This is a high-dimensional feature representation, typically stored as a 256×256×16-dimensional tensor, where each pixel contains 16 dimensions of feature information, rather than just traditional RGB color values. These neural textures are then projected onto the 3D facial mesh through UV texture mapping, accurately reproducing the target person's skin texture, wrinkles, and mouth area details.

[0045] As an example, the neural texture is input into a neural rendering network, where convolutional layers gradually decode the high-dimensional features within the neural texture and generate a final high-resolution mouth rendering. This process allows the neural network to automatically learn and complement the complex details of the mouth, such as the lighting effects of the teeth and tongue, and the variations in facial shadows, ensuring that lip animation is not only synchronized with the audio but also visually realistic. Ultimately, neural texture mapping ensures that the generated facial animation retains the target person's unique facial features, making the overall 3D rendering more realistic and natural.

[0046] Step S5 includes: The mouth area of ​​the 3D rendered facial image is rendered using feature maps and a U-Net-based lip neural rendering network to obtain a high-resolution mouth rendering image. The lip shape neural rendering network includes: dynamic lip shape prediction module, neural texture enhancement module and adaptive lighting rendering module.

[0047] By adopting the above-mentioned technical solution, the lip neural renderer in this application significantly improves the accuracy and realism of generated facial animations, especially in the rendering effect of the mouth area, which can accurately reproduce the movement of the lips, the activity of the tongue, and the subtle changes of the lips, thereby ensuring the naturalness and synchronization of facial expressions. Through the combination of dilated convolution and multi-layer convolution, the renderer effectively reduces artifacts and distortion in the image and maintains high-quality visual effects, especially in complex lip movements. This makes the generated facial animation not only visually highly realistic, but also able to maintain good synchronization with the audio signal, providing a smoother and more natural visual experience. At the same time, the renderer can seamlessly integrate with the background of the target person to ensure the visual coordination between the face and the background, thereby enhancing the immersion of the overall video.

[0048] As an example, after neural texture mapping is complete, the mouth area is rendered first to ensure accurate and natural mouth movement. The feature map (16-dimensional channels) generated by the neural texture is combined with the neural rendering network, and dilated convolutions are used instead of pooling layers to maintain spatial resolution and reduce visual artifacts such as blurring or distortion. The result is a high-resolution rendered mouth image with lip movements fully synchronized with the audio.

[0049] As an example, traditional lip rendering methods typically rely on static texture mapping based on 3D models or mouth animation synthesized by 2D generative adversarial networks. However, these methods have the following problems: (1) Static textures cannot adapt to dynamic lip shape changes: Traditional methods often use predefined lip shape textures, which makes the lip shape appear stiff and unnatural when the speech changes rapidly.

[0050] (2) The 2D generative adversarial network method lacks structural consistency: Although the 2D generation-based method can synthesize mouth animation, it cannot guarantee the consistency of lip shape at different angles and lighting, and is prone to unnatural mouth distortion or blur.

[0051] (3) Insufficient synchronization between lip shape and audio: Traditional methods may ignore the microscopic features of audio during the lip shape generation process, resulting in subtle asynchrony between lips and pronunciation.

[0052] To address these issues, the Lip Shape Neural Renderer employs a dynamic neural rendering method based on 3D structures. This ensures that lip shapes maintain structural consistency across different angles and lighting conditions, and ensures high synchronization between lip shapes and audio. By combining temporal consistency modeling, adaptive lighting correction, and high-resolution feature generation, the model can produce more natural and high-fidelity mouth animations. The core algorithm of the Lip Shape Neural Renderer is divided into three key modules: dynamic lip shape prediction, neural texture enhancement, and adaptive lighting rendering.

[0053] (1) Dynamic lip shape prediction module The main task of this module is to predict realistic mouth movements based on the input audio and expression parameters and reconstruct them in 3D space. A two-stream temporal convolutional network is used to perform temporal modeling of audio features and expression parameters respectively, ensuring that the lip shape prediction results are highly synchronized with the audio. The audio stream is responsible for learning the temporal variation pattern of speech, capturing the transition characteristics between phonemes, and ensuring that the lip shape does not jump abruptly. The expression stream is responsible for learning the personalized lip shape style of the target person and ensuring that the generated lip shape conforms to the target person's speaking style. The personalized mouth mesh of the target person is generated by fusing deformation parameters, and the mouth movement is simulated in 3D space. The dynamic deformation of the mouth mesh is calculated for each frame to ensure that the lip shape animation is smooth and consistent with the speech content.

[0054] Innovation: Compared with traditional generative adversarial networks or long short-term memory network methods, this module uses a dual-stream temporal convolutional network + fusion deformation structure modeling, which can accurately capture the relationship between audio and lip shape, and ensure the consistency of lip shape at different angles, avoiding the lip distortion problem common in generative adversarial network methods.

[0055] (2) Neural texture enhancement module To improve the detail quality of mouth animation, this module uses neural textures to generate personalized lip, tooth, and tongue motion features. Using a 256×256×16-dimensional neural texture feature map, the lip, tooth, and tongue regions are individually modeled, ensuring that the lip shape details are consistent with the original video of the target person. The texture contains not only color information but also normal information, lighting characteristics, and mouth motion patterns, ensuring natural lip shape under different lighting conditions. Finally, a temporal long-short temporal memory network is used to process the neural texture parameters, ensuring that the lips, teeth, and tongue change smoothly between frames without abrupt texture jumps or visual artifacts.

[0056] Innovation: Compared with traditional static texture mapping, this module uses dynamic neural texture + temporal long and short temporal memory network texture adjustment, which can ensure that the dynamic changes of the mouth area are smooth and natural, and avoid the problem of incoherent lip movement.

[0057] (3) Adaptive Lighting Rendering Module After lip shape prediction and neural texture enhancement, the final task is to ensure that the mouth region blends naturally into the target person's video. A conditional generative network is used to predict the target video's global illumination information, and ambient light mapping is used to compensate the mouth region for illumination, ensuring that it matches the overall lighting style of the target video. A transparency mask and background synthesis network are used to ensure that the generated lip region seamlessly blends into the target video, avoiding abrupt boundaries or color mismatches between the lip region and the surrounding face.

[0058] Innovation: Compared with traditional video synthesis methods, this module uses a conditional generative adversarial network for lighting matching, which can dynamically adjust the lighting of the lip area, allowing the lip animation to perfectly adapt to different target video scenes, greatly improving the realism of the rendering.

[0059] Step S6 includes: S61: remove the mouth area of ​​the original target video to obtain a background frame; S62: Through the background synthesis network, the edges of the mouth area of ​​the high-resolution mouth rendering image and the background frame are repaired to obtain a speech-driven lip shape generation video.

[0060] As an example, since the mouth area rendered in the first stage may have edge mismatches with the original target video, background synthesis is required to naturally blend the mouth into the target video. The mouth area image generated by the first stage neural rendering and the background frame of the target video (excluding the mouth area) are used to provide complete facial information. Combined with the background synthesis network: a similar structure to the first stage is used, but the input is 6-dimensional (mouth rendering + target video background). This network is used to repair the edge transition of the mouth area to ensure the natural fusion of the mouth and the face. Finally, the final complete frame is output, and the mouth area is seamlessly combined with the original video.

[0061] As an example, a first-stage model is trained by crawling a large number of speaker videos from major video platforms, resulting in a universal audio-to-expression coefficient mapping network with strong generalization capabilities. Then, for a specific high-resolution speaker image driving video, the universal model is fine-tuned and converted to the individual image. Finally, the fine-tuned first-stage universal model is used to guide the training of the second-stage model, significantly reducing training costs. Because the second-stage model is trained entirely on the driving video, it achieves strong lip sync and detail reproduction, while preserving individual identity information to the greatest extent possible.

[0062] As an embodiment, when the driving audio is obtained, the audio is first preprocessed in the same way as the training model, and then a one- or two-stage end-to-end inference is directly performed to obtain a complete audio-synchronized lip sequence image sequence, which is then connected into a video using tools such as opencv or ffmpeg, and the complete generated video is obtained by converting the frame rate according to the requirements.

[0063] This application also discloses an electronic device. Figure 2 , Figure 2 Schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0064] The communication bus 502 is used to implement the connection and communication between these components.

[0065] The user interface 503 may include a display screen, and the optional user interface 503 may also include a standard wired interface or a wireless interface.

[0066] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0067] The present application also discloses a computer-readable storage medium storing a plurality of instructions suitable for loading by a processor to execute the above-mentioned speech-driven lip shape generation method.

[0068] The above are merely exemplary embodiments of the present disclosure and are not intended to limit the scope of the present disclosure. In other words, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure.

[0069] This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not described herein. The description and examples are to be considered as exemplary only, and the scope and spirit of the present disclosure are to be defined by the claims.

Claims

1. A lip shape generation method based on speech drive, characterized in that: The method comprises the following steps: S1: Get the audio signal of the original target video; S2: Construct audio-expression coefficient mapping network; The audio signal is input into the audio-expression coefficient mapping network for training to obtain the mixed deformation coefficients of the target person; The audio-expression coefficient mapping network includes: feature extraction module, self-attention enhancement module and personalized expression conversion module; S3: Construct a 3D rendered facial image of the target person based on the blend deformation coefficients; S4: Perform neural texture mapping on the 3D rendered facial image to generate feature maps; S5: Render the mouth area of ​​the 3D rendered facial image using the feature map to obtain a high-resolution mouth rendered image; S6: By combining high-resolution mouth rendering images with a background synthesis network, we obtain a speech-driven lip shape generation video.

2. The method for generating lip shapes based on speech drive according to claim 1, wherein: The audio-expression coefficient mapping network includes: 4 layers of 2D convolutional layers and 3 layers of fully connected networks; each 2D convolutional layer uses a 3×1 convolution kernel and a time dimension step size of 2.

3. The method for generating lip shapes based on speech drive according to claim 1, wherein: Step S2 includes: S21: Extract audio features of the audio signal through the feature extraction module and the self-attention enhancement module. Audio features include: timing information and pronunciation content. The feature extraction module is a speech recognition framework based on a recurrent neural network and uses multi-scale convolution kernels. S22: Map the audio features to the facial expression space of the target person to obtain the facial expression coefficient; S23: Mapping the facial expression coefficients to the personalized expression space of the target person in the facial expression space through the personalized expression conversion module to obtain the mixed deformation coefficients of the target person; S24: Smoothing the facial expression space of the current frame through a temporal filter.

4. The method for generating lip shapes based on speech drive according to claim 3, wherein: Step S23 includes: S231: learning a linear mapping matrix from the original target video of the target person; S232: The facial expression coefficients are converted into mixed deformation coefficients of the target person through the personalized expression conversion module in combination with the linear mapping matrix.

5. The method for generating lip shapes based on speech drive according to claim 1, wherein: Step S3 includes: S31: constructing a personalized 3D facial model of the target person, specifically comprising: constructing a statistical 3D face model, and adjusting the lip shape and expression of the statistical 3D face model using a blending deformation coefficient to obtain a personalized 3D face model; S32: deforming the mouth area of ​​the personalized 3D facial model based on the head pose of the original target video; S33: Calculate UV texture coordinates of the personalized 3D facial model; based on the UV texture coordinates, map the deformed face of the target person to the 3D facial mesh of the personalized 3D facial model to obtain a low-resolution 3D rendered facial image.

6. The method for generating lip shapes based on speech drive according to claim 3, wherein: Step S4 includes: S41: Constructing a neural texture, specifically comprising: generating a high-dimensional neural texture by using a 3D facial mesh of a 3D rendered facial image and the facial expression coefficients of the target person; Neural textures are used to store the target person’s personalized facial details, including skin texture, wrinkles, and dynamic changes in the mouth. S42: Using the UV texture coordinates of the personalized 3D facial model, combined with a neural rendering network and a multi-resolution dynamic texture mapping method, the neural texture is mapped to the 2D image space of the 3D rendered facial image to generate a feature map.

7. The method for generating lip shapes based on speech drive according to claim 1, wherein: Step S5 includes: The mouth area of ​​the 3D rendered facial image is rendered using feature maps and a U-Net-based lip neural rendering network to obtain a high-resolution mouth rendering image. The lip shape neural rendering network includes: dynamic lip shape prediction module, neural texture enhancement module and adaptive lighting rendering module.

8. The method for generating lip shapes based on speech drive according to claim 1, wherein: Step S6 includes: S61: remove the mouth area of ​​the original target video to obtain a background frame; S62: Through the background synthesis network, the edges of the mouth area of ​​the high-resolution mouth rendering image and the background frame are repaired to obtain a speech-driven lip shape generation video.

9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed by a computer, the method according to any one of claims 1 to 8 is executed.

Images