Method and device for driving high reduction of facial features of speaker

By extracting oral keyframes and using the lip alignment module and oral supplement module that drives the speaker model, combined with the wenet voice model, the problem of inaccurate oral features and unnatural expressions when mutes the facial feature restoration is solved, and high-quality facial Feature Restore.

CN118941682AInactive Publication Date: 2024-11-12XIAMEN LIMAYAO NETWORK TECH CO LTD

Patent Information

Application Number
CN202411436768.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-15
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art is difficult to achieve real-person facial features restoration, especially when the facial expression is unnatural when silent, and the model has not learned the relative positional relationship between the cheek boundary point and the mouth boundary point when shutting down, resulting in inaccurate oral characteristics.

Method used

The preset face segmentation model is used to extract oral keyframes, combine the pre-trained driver speaker model and wenet voice model, and perform audio and picture synchronization processing through the lip alignment module and the oral supplement module to obtain the oral sound and picture synchronization mouth shape. , and introduce key point loss function to improve training effect.

Benefits of technology

It realizes a high degree of restoration of the oral shape of the original video in each frame, and naturally shuts up during the silent period to maintain coordination of facial and mouth movements, solving the problems of inaccurate oral characteristics and unnatural expressions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941682A_ABST
    Figure CN118941682A_ABST
Patent Text Reader

Abstract

The invention provides a method and a device for driving high restoration of facial features of a speaker, and relates to the technical field of facial feature restoration, the method comprises three steps: acquiring 15 oral cavity key frames of the speaker, driving a speaker model, and extracting audio features by using wenet; and a key point loss function is introduced in the training stage. In actual use of the method, a speaker can be driven to maintain high restoration of each frame of image to the oral cavity form of an original video, and the mouth can be naturally closed and coordination of the face and mouth actions can be maintained in a mute time period. The problems that in the prior art, the oral cavity of a speaker and the oral cavity of a person are inconsistent and distorted are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of facial feature restoration, and in particular to a method and device for driving a speaker's facial features to be highly restored. Background Art

[0002] Currently in the field of artificial intelligence and computer science, the speech driving scheme of digital humans, that is, the driving speaker scheme, can already maintain the synchronization of sound and picture, but there are still challenges in achieving the same facial features as real people; among them, the two most important points are the similarity between the teeth of the driving speaker and a real person and the naturalness of the facial expressions of the driving speaker when silent.

[0003] Specifically, when the original reference frame of the driven speaker is in a closed mouth state, the model lacks feature references of the mouth, so the speaker's mouth is a general mouth learned by the model, which is different from the speaker's own mouth; when the original reference frame of the driven speaker exposes the mouth, the driven speaker can restore his own mouth. Another distortion problem is that when the speaker is driven in a silent segment, although the speaker can close his mouth, the facial and mouth muscle movements still move in the same way as the original video, which visually produces an unnatural closing of the mouth. This is because the model has not learned the relative position relationship between the cheek boundary points and the mouth boundary points when the mouth is closed.

[0004] In view of this, this application is filed. Summary of the invention

[0005] The present invention provides a method and device for driving a speaker's facial features to be highly restored, which can at least partially improve the above-mentioned problem.

[0006] To achieve the above object, the present invention adopts the following technical solutions: A method for driving a speaker's facial features to be highly restored, comprising: Obtain the original video, extract the oral region using a preset face segmentation model, extract oral frames of all shapes in the original video based on the oral region, and select multiple oral key frames with the greatest shape differences based on preset indicator information; Calling a pre-trained driving speaker model to drive and process the audio of the original video, the image of the original video and the oral key frame to obtain a sound-and-picture synchronized lip shape with an oral cavity, wherein the driving speaker model includes a lip alignment module and an oral supplement module; The wenet speech model is used for extraction to obtain audio features, and the lip shape and audio features of the oral cavity are combined to obtain a high-reduction video of the driving speaker.

[0007] The present invention also provides a device for driving a speaker's facial features to be highly restored, comprising: An oral key frame extraction unit is used to obtain the original video, extract the oral region using a preset face segmentation model, extract oral frames of all shapes in the original video based on the oral region, and select multiple oral key frames with the greatest shape differences according to preset indicator information; A driving speaker model unit is used to call a pre-trained driving speaker model to drive and process the audio of the original video, the image of the original video and the oral key frame to obtain a sound-image synchronized lip shape with an oral cavity, wherein the driving speaker model includes a lip alignment module and an oral supplement module; The model generation unit is used to extract using the wenet speech model to obtain audio features, and combine the lip shape and audio features of the oral cavity to obtain a high-reduction video of the driving speaker.

[0008] In summary, the method of driving the speaker's facial features to be highly restored mainly involves three steps. The first step is to obtain 15 oral key frames of the speaker, the second step is to drive the speaker model, and the third step is to use wenet to extract audio features, and a key point loss function is also introduced in the training stage. This method drives the speaker to maintain a high degree of restoration of the original video's oral morphology in each frame, and to close the mouth naturally during silent periods, maintaining coordination between the face and mouth movements. It aims to solve the problem of existing restoration methods that the speaker's oral cavity is different from the speaker's own oral cavity, as well as distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flowchart of a method for driving a speaker's facial features to be highly restored provided by the first embodiment of the present invention; Figure 2 is a schematic diagram of a model for driving a speaker model provided by the first embodiment of the present invention; Figure 3 is a schematic diagram of a polygon discriminator provided by a first embodiment of the present invention; Figure 4 Schematic diagram of the distribution of 203 key points provided in the first embodiment of the present invention.

[0010] Figure 5 It is a module schematic diagram of a device for driving a speaker's facial features with high restoration provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0012] As shown in the reference figure, the first embodiment of the present invention discloses a method for driving a speaker's facial features to be highly restored, which can be executed by a device for driving a speaker's facial features to be highly restored (hereinafter referred to as a high-reduction device), and in particular, is executed by one or more processors in the high-reduction device to implement the following method: S1, obtaining the original video, extracting the oral region using a preset face segmentation model, extracting oral frames of all shapes in the original video based on the oral region, and screening out multiple oral key frames with the greatest shape differences according to preset indicator information; Specifically, step S1 includes: obtaining an original video, and performing detection processing on the original video using a preset facial key point detection model to detect facial key points in the video; Extracting the oral cavity area from the facial key points using a preset face segmentation model to obtain the mouth key points, and obtaining the mouth area, oral cavity area, and mouth shape changes based on the mouth key points; The first 15 oral key frames with the largest morphological differences in the original video are extracted based on the indicator information, mouth area, oral cavity area and mouth shape changes, wherein the indicator information includes mouth shape changes, oral cavity size and brightness changes.

[0013] In this embodiment, 15 oral key frames of the speaker are obtained. The oral region is extracted using a face segmentation model. This is used to extract the oral cavity in all forms (lowering the head, raising the head, left face, right face, strong lighting, mouth opening size, etc.) in the original video. Because the oral cavity (teeth, tongue, inside of the oral cavity) of the speaker will present different forms at different time points due to the head angle and lighting, etc. The 15 frames of the oral cavity with the largest form difference are retained.

[0014] Specifically, the facial key points are first detected using the facial key point detection model; secondly, the mouth area, oral cavity area, and mouth shape change are obtained based on the mouth key points. The indicators include: mouth shape change, oral cavity size, and brightness change. Finally, based on these three indicators, the top 15 frames with the largest differences are extracted.

[0015] S2, calling a pre-trained driving speaker model to drive and process the audio of the original video, the image of the original video and the oral key frame to obtain a sound-and-picture synchronized lip shape with an oral cavity, wherein the driving speaker model includes a lip alignment module and an oral cavity supplement module; Specifically, step S2 includes: acquiring an image and audio from the original video, wherein the image includes a masked oral cavity; Use the lip alignment module that drives the speaker model to align the image and audio to obtain the lip shape that is synchronized with the sound and picture without the mouth. The oral cavity supplement module driving the speaker model is used to supplement the lip shape and oral cavity key frames of the audio and video synchronization without the oral cavity to obtain the lip shape of the audio and video synchronization with the oral cavity.

[0016] Preferably, the lip alignment module driving the speaker model is used to align the image and the audio to obtain the lip shape with synchronized sound and picture without the oral cavity, specifically: The image is input into an image encoding module for encoding processing, and the audio is input into a wenet speech model for extraction processing; The encoded image and the extracted audio are decoded using an image decoding module to obtain lip shape with synchronized sound and picture without oral cavity.

[0017] Preferably, the oral cavity supplement module driving the speaker model is used to supplement the lip shape and oral cavity key frame of the audio-visual synchronization without the oral cavity to obtain the lip shape of the audio-visual synchronization with the oral cavity, specifically: Use the concat function to connect the lip shape and oral key frames of the audio and video synchronization without the oral cavity; The connected lip shape and oral key frame of the audio and video synchronization without the oral cavity are input into the image encoding module and the image decoding module in sequence, and are respectively encoded and decoded to obtain the lip shape of the audio and video synchronization with the oral cavity.

[0018] In this embodiment, there are two problems with the existing restoration scheme. The first is that the oral cavity accounts for a relatively small proportion of the entire speaker's face, and its features vary widely, making it easy for the generation model to ignore details. Second, audio features are highly correlated with changes in the lips, and lip changes are highly correlated with oral cavity details. Therefore, the model is designed as two modules, namely a lip synchronization module and an oral cavity supplementation module. The first module is used to generate a pair of lips synchronized with the audio, and the second module is used to generate a speaker's mouth (teeth, tongue, and inside of the mouth) related to the lips. The training will be divided into two stages. In the first stage, the first module (lip alignment module) is trained, and in the second stage, the second module (oral cavity supplementation module) is trained. The weight of the first module is frozen, and the output of the first module is used as part of the input of the second module. The other part of the input of the second module is the oral cavity sequence. Model such as Figure 2 shown.

[0019] S3, using wenet speech model to extract, obtain audio features, and combine the lip shape and audio features of the oral cavity to obtain a high-reduction video of the driving speaker.

[0020] In this embodiment, the wenet speech model is used to extract richer audio features. Using the audio features extracted by wenet to replace the mel features can provide the generative model with richer audio features, so that the generative model can learn the relationship between pitch, volume, emotion, lip shape and facial muscle movement.

[0021] Preferably, before calling the pre-trained driving speaker model to drive the audio of the original video, the image of the original video and the oral key frame, the method further includes: Obtain a facial polygon sequence of 10 consecutive frames, input the facial polygon sequence of 10 consecutive frames into a preset polygon discriminator, and output a vector with a value of 0 to 1, where each element in the vector represents the probability that the corresponding frame is a real frame. The polygon discriminator consists of an encoder, a fully connected layer, and a softmax function; The cross entropy loss function is used to calculate the loss value. The formula is: ,in, is the true value, is the model prediction value; Repeat the above steps until the loss value reaches the preset threshold, completing the first stage of training for the lip alignment module that drives the speaker model.

[0022] In this embodiment, a polygon discriminator is introduced. This discriminator will participate in the first stage of the training of the first module, that is, the first stage of the training of the lip alignment module. Based on the fact that the changes of face polygons in real videos have continuity and spatial geometric restrictions, and are not affected by factors such as lighting, face color, and size, a discriminator based on face polygons is used. This discriminator discriminates whether it is a fake speaking face polygon sequence generated by the generator or a real speaking face polygon sequence based on the 2D spatial changes of face polygons in 10 consecutive frames. The model structure of the discriminator is as follows: Figure 3 shown.

[0023] The face polygon image is a pentagon with key points 119, 48, 66, 132, and 126 as vertices. The discriminator consists of an encoder, a fully connected layer, and a softmax function. The input is a sequence of 10 consecutive frames of face polygons, and the output is a vector of 0 to 1. Each element in the vector represents the probability that the corresponding frame is a real frame. The cross entropy loss function is used to calculate the loss. The formula is: ,in, is the true value, is the model prediction value. The face polygon discriminator is not affected by the pixel changes of the image and focuses on learning the deformation characteristics of the face geometry. Figure 4 The distribution of 203 key points is shown.

[0024] In summary, the method for driving the speaker's facial features to be highly restored can drive the speaker to maintain a high degree of restoration of the oral morphology of the original video in each frame, and can naturally close the mouth during silent periods to maintain coordination of facial and mouth movements.

[0025] See also Figure 5 The second embodiment of the present invention provides a device for driving a speaker's facial features to be highly restored, comprising: The oral key frame extraction unit 201 is used to obtain the original video, extract the oral region using a preset face segmentation model, extract oral frames of all shapes in the original video based on the oral region, and select multiple oral key frames with the greatest shape difference according to preset indicator information; The driving speaker model unit 202 is used to call the pre-trained driving speaker model to drive the audio of the original video, the image of the original video and the oral key frame to obtain the audio-video synchronization lip shape with the oral cavity, wherein the driving speaker model includes a lip alignment module and an oral supplement module; The model generation unit 203 is used to extract using the wenet speech model to obtain audio features, and combine the lip shape and audio features of the oral cavity to obtain a high-reduction video of the driving speaker.

[0026] Preferably, the original video is obtained, the oral region is extracted using a preset face segmentation model, oral frames of all shapes in the original video are extracted based on the oral region, and multiple oral key frames with the greatest shape differences are screened out according to preset indicator information, specifically: Obtaining an original video, and performing detection processing on the original video using a preset facial key point detection model to detect facial key points in the video; Extracting the oral cavity area from the facial key points using a preset face segmentation model to obtain the mouth key points, and obtaining the mouth area, oral cavity area, and mouth shape changes based on the mouth key points; The first 15 oral key frames with the largest morphological differences in the original video are extracted based on the indicator information, mouth area, oral cavity area and mouth shape changes, wherein the indicator information includes mouth shape changes, oral cavity size and brightness changes.

[0027] Preferably, a pre-trained driving speaker model is called to drive the audio of the original video, the image of the original video and the oral key frame to obtain the audio-visual synchronized lip shape with the oral cavity, specifically: Acquire an image and an audio from the original video, wherein the image includes a masked mouth; Use the lip alignment module that drives the speaker model to align the image and audio to obtain the lip shape that is synchronized with the sound and picture without the mouth. The oral cavity supplement module driving the speaker model is used to supplement the lip shape and oral cavity key frames of the audio and video synchronization without the oral cavity to obtain the lip shape of the audio and video synchronization with the oral cavity.

[0028] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for driving a speaker's facial features to be highly restored, characterized in that: include: Obtain the original video, extract the oral region using a preset face segmentation model, extract oral frames of all shapes in the original video based on the oral region, and select multiple oral key frames with the greatest shape differences based on preset indicator information; Calling a pre-trained driving speaker model to drive and process the audio of the original video, the image of the original video and the oral key frame to obtain a sound-and-picture synchronized lip shape with an oral cavity, wherein the driving speaker model includes a lip alignment module and an oral supplement module; The wenet speech model is used for extraction to obtain audio features, and the lip shape and audio features of the oral cavity are combined to obtain a high-reduction video of the driving speaker.

2. The method for driving a speaker's facial features to be highly restored according to claim 1, characterized in that: Get the original video, use the preset face segmentation model to extract the oral region, extract all the oral frames in the original video based on the oral region, and select multiple oral key frames with the largest morphological differences according to the preset indicator information, specifically: Obtaining an original video, and performing detection processing on the original video using a preset facial key point detection model to detect facial key points in the video; Extracting the oral cavity area from the facial key points using a preset face segmentation model to obtain the mouth key points, and obtaining the mouth area, oral cavity area, and mouth shape changes based on the mouth key points; The first 15 oral key frames with the largest morphological differences in the original video are extracted based on the indicator information, mouth area, oral cavity area and mouth shape changes, wherein the indicator information includes mouth shape changes, oral cavity size and brightness changes.

3. The method for driving a speaker's facial features to be highly restored according to claim 1, characterized in that: The pre-trained driving speaker model is called to drive the audio of the original video, the image of the original video and the oral key frame to obtain the audio-visual synchronized lip shape with the oral cavity, specifically: Acquire an image and an audio from the original video, wherein the image includes a masked mouth; Use the lip alignment module that drives the speaker model to align the image and audio to obtain the lip shape that is synchronized with the sound and picture without the mouth. The oral cavity supplement module driving the speaker model is used to supplement the lip shape and oral cavity key frames of the audio and video synchronization without the oral cavity to obtain the lip shape of the audio and video synchronization with the oral cavity.

4. The method for driving a speaker's facial features to be highly restored according to claim 3, characterized in that: The lip alignment module that drives the speaker model is used to align the image and audio to obtain the lip shape of the synchronized sound and picture without the mouth, specifically: The image is input into an image encoding module for encoding processing, and the audio is input into a wenet speech model for extraction processing; The encoded image and the extracted audio are decoded using an image decoding module to obtain lip shape with synchronized sound and picture without oral cavity.

5. The method for driving a speaker's facial features to be highly restored according to claim 3, characterized in that: The oral supplement module driving the speaker model is used to supplement the lip shape and oral key frames of the sound and picture synchronization without the oral cavity to obtain the lip shape of the sound and picture synchronization with the oral cavity, specifically: Use the concat function to connect the lip shape and oral key frames of the audio and video synchronization without the oral cavity; The connected lip shape and oral key frame of the audio and video synchronization without the oral cavity are input into the image encoding module and the image decoding module in sequence, and are respectively encoded and decoded to obtain the lip shape of the audio and video synchronization with the oral cavity.

6. The method for driving a speaker's facial features to be highly restored according to claim 1, characterized in that: Before calling the pre-trained driving speaker model to drive the audio of the original video, the image of the original video and the oral key frame, the method further includes: Obtain a facial polygon sequence of 10 consecutive frames, input the facial polygon sequence of 10 consecutive frames into a preset polygon discriminator, and output a vector with a value of 0 to 1, where each element in the vector represents the probability that the corresponding frame is a real frame. The polygon discriminator consists of an encoder, a fully connected layer, and a softmax function; The cross entropy loss function is used to calculate the loss value. The formula is: ,in, is the true value, is the model prediction value; Repeat the above steps until the loss value reaches the preset threshold, completing the first stage of training for the lip alignment module that drives the speaker model.

7. A device for driving a speaker's facial features to be highly restored, characterized in that: include: An oral key frame extraction unit is used to obtain the original video, extract the oral region using a preset face segmentation model, extract oral frames of all shapes in the original video based on the oral region, and select multiple oral key frames with the greatest shape differences according to preset indicator information; A driving speaker model unit is used to call a pre-trained driving speaker model to drive and process the audio of the original video, the image of the original video and the oral key frame to obtain a sound-image synchronized lip shape with an oral cavity, wherein the driving speaker model includes a lip alignment module and an oral supplement module; The model generation unit is used to extract using the wenet speech model to obtain audio features, and combine the lip shape and audio features of the oral cavity to obtain a high-reduction video of the driving speaker.

8. The device for driving a speaker's facial features to be highly restored according to claim 6, characterized in that: Get the original video, use the preset face segmentation model to extract the oral region, extract all the oral frames in the original video based on the oral region, and select multiple oral key frames with the largest morphological differences according to the preset indicator information, specifically: Obtaining an original video, and performing detection processing on the original video using a preset facial key point detection model to detect facial key points in the video; Extracting the oral cavity area from the facial key points using a preset face segmentation model to obtain the mouth key points, and obtaining the mouth area, oral cavity area, and mouth shape changes based on the mouth key points; The first 15 oral key frames with the largest morphological differences in the original video are extracted based on the indicator information, mouth area, oral cavity area and mouth shape changes, wherein the indicator information includes mouth shape changes, oral cavity size and brightness changes.

9. The device for driving a speaker's facial features to be highly restored according to claim 6, characterized in that: The pre-trained driving speaker model is called to drive the audio of the original video, the image of the original video and the oral key frame to obtain the audio-visual synchronized lip shape with the oral cavity, specifically: Acquire an image and an audio from the original video, wherein the image includes a masked mouth; Use the lip alignment module that drives the speaker model to align the image and audio to obtain the lip shape that is synchronized with the sound and picture without the mouth. The oral cavity supplement module driving the speaker model is used to supplement the lip shape and oral cavity key frames of the audio and video synchronization without the oral cavity to obtain the lip shape of the audio and video synchronization with the oral cavity.

Citation Information

Patent Citations

  • Speech state recognition method and device, storage medium and terminal

    CN111666820A

  • Virtual face generation method

    CN113781610A

  • Method and system for driving digital human to speak through audio, and storage medium

    CN117152319A

  • Method and device for training stable mouth shape of digital human

    CN118471251A

  • Automatic alignment method and system for digital population types

    CN118612490A

Cited By

  • Method and device for matching and repairing human mouth shape and voice content in video, electronic equipment and storage medium

    CN121353126A