Digital human lip shape driving method based on multi-modal fusion and consistency constraint

Abstract

The invention discloses a digital human lip shape driving method based on multi-modal fusion and consistency constraint, and belongs to the field of digital human generation, and the method comprises the following steps: S1, constructing a multi-modal feature extraction and representation framework by taking a reference face image as input, and generating personalized face feature representation based on the constructed multi-modal feature extraction and representation framework; s2, using QKV consistency constraint to drive personalized face feature representation and voice features to perform dynamic coupling, and generating a lip action under voice driving; and S3, coding the reference video by using discrete coding to obtain discrete features, limiting the variation range of the coding space based on a Lipschitz continuity constraint mechanism, and optimizing the lip action. By adopting the digital human lip shape driving method based on multi-modal fusion and consistency constraint, the voice driving consistency, local detail reduction and overall generation quality are comprehensively improved, and the microscopic dynamic performance and detail fidelity of lip shape generation actions are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human generation, and in particular to a digital human lip shape driving method based on multimodal fusion and consistency constraints. Background Art

[0002] As digital human technology flourishes, high-fidelity speech-driven lip shape generation, as a key component, has attracted considerable attention and faces numerous challenges. Current technology has the following shortcomings in accurately modeling digital human personality traits, ensuring semantic consistency between lip shape and speech, and improving detailed representation:

[0003] 1. Difficulty in modeling personalized features: Existing facial feature extraction methods are unable to fully and accurately capture the unique facial features of an individual. On the one hand, relying solely on single-modal data, such as using only image information, cannot fully explore the semantic connotations of facial features, resulting in an inaccurate and incomplete description of the global structure and local details of the face. On the other hand, traditional methods lack effective processing of the differences in motion characteristics of different facial regions. In the lip shape driven generation task, it is impossible to perform targeted modeling for the different degrees of motion intensity in areas such as the mouth, eyes, and nose, making it difficult for the generated lip shape to fit the individual's unique facial structure and muscle movement pattern, reducing the personalized performance of the digital human's lip shape.

[0004] 2. Poor semantic consistency between speech and lip shape: There is a complex multimodal relationship between speech and lip shape, but existing technologies fail to fully explore and utilize this information. The combined influence of phoneme characteristics, speech rate, and personalized facial features on lip shape movements is not properly modeled, resulting in inaccurate matching of lip shape movements with speech rhythm and pronunciation. In practical applications, problems such as lip shape being out of sync with speech and mouth shape not matching pronunciation frequently occur when digital humans speak, seriously affecting the realism and interactive experience of digital humans.

[0005] 3. Inadequate detail generation: When generating lip movements, the processing of local details is weak. Existing technologies struggle to accurately render details such as tooth morphology and lip motion texture, resulting in a stiff and unrealistic micro-dynamic appearance of generated lip movements. Furthermore, existing methods fail to fully consider the temporal continuity of the video when optimizing the generated results, resulting in unnatural transitions between adjacent frames. This further reduces the overall generation quality and fails to meet the requirements for high-fidelity digital human lip motion driving.

[0006] Due to the above problems, the current application of high-fidelity voice-driven lip shape generation technology in film and television production, virtual live broadcasting, intelligent customer service and other fields is limited (in film and television production, it is impossible to provide realistic lip shape animation for virtual characters, which affects the visual effects of the film; in virtual live broadcasting, the host's digital human image's lip shape does not match the voice, reducing the audience's viewing experience; in intelligent customer service scenarios, the unnatural performance of digital human customer service makes it difficult to provide users with a good interactive experience and cannot effectively improve service quality). Summary of the Invention

[0007] The purpose of the present invention is to provide a digital human lip shape driving method based on multimodal fusion and consistency constraints to solve the above technical problems.

[0008] To achieve the above objectives, the present invention provides a digital human lip shape driving method based on multimodal fusion and consistency constraint, comprising the following steps:

[0009] S1. Using a reference facial image as input, a multimodal feature extraction and representation framework is constructed, and personalized facial feature representation is generated based on the constructed multimodal feature extraction and representation framework.

[0010] S2, using QKV consistency constraints to drive the dynamic coupling of the personalized facial feature representation generated in step S1 and the voice features to generate voice-driven lip movements;

[0011] S3. Use discrete coding to encode the reference video to obtain discrete features, and limit the range of variation of the coding space based on the Lipschitz continuity constraint mechanism to optimize the lip movements generated in step S2.

[0012] Preferably, step S1 specifically includes the following steps:

[0013] S11. Input the reference face image I into the visual language model to generate a language description of the face appearance and structured statistical parameterized features S(I), providing a semantic-level quantitative expression for personalized feature modeling.

[0014] S12, inputting the reference face image I into the segmentation model for decomposition to obtain multiple segmentation regions, thereby generating a segmentation mask M(I) to capture local characteristics;

[0015] S13, using the high-low frequency decoupling encoder to perform feature fusion encoding on the language description, structured statistical parameterized features and segmentation mask, to generate a personalized facial feature representation F with both global and local characteristics face :

[0016] F face =ε(I,S(I),M(I)) (1);

[0017] Where ε represents the high- and low-frequency decoupling encoder.

[0018] Preferably, the language description and structured statistical parameterized features in step S11 include geometric shape, texture distribution and regional feature description.

[0019] Preferably, in step S13, taking into account the differences in motion characteristics of different facial regions, high-frequency features and low-frequency features are processed separately during modeling;

[0020] The low-frequency features extract global structural features from the language description and structured statistical parameterized features S(I) and segmentation masks:

[0021]

[0022] Where, ε low (S(I),M(I)) represents low-frequency features; φ low Represents the function of calculating low-frequency features; N represents the total number of segmented regions; M k (I) represents the segmentation mask of the kth segmentation region; T k (I) represents the language description and structured statistical parameterized features of the k-th segmented region; α k represents weight;

[0023] High-frequency features extract local texture and dynamic detail features from the reference face image I:

[0024] ε high (I)=H high (I) (3);

[0025] Where, ε high (I) represents high-frequency features, which are the feature representations obtained by extracting local texture and dynamic details from the reference face image I; H high (I) indicates the decomposition of the reference face image I using a high-frequency encoder.

[0026] Preferably, in step S2, based on the QKV consistency constraint mechanism, the voice features and personalized face features are represented by F face Jointly projected into a shared semantic space to obtain the dynamic consistency and semantic relevance features of speech-driven lip movements F shared :

[0027] F shared =W shared ·(F audio +F face ) (4);

[0028] Where W shared The projection matrix representing the shared semantic space; F audioRepresents phonetic features;

[0029] During the generation process, the QKV consistency constraint mechanism guides the generation network to explicitly model speech features and personalized face features F face The synergistic relationship between the dynamic lip shape movement and the personalized facial features F face Joint optimization with dynamic lip motion and generation of high-fidelity lip motion based on diffusion model.

[0030] Preferably, the diffusion model expression is as follows:

[0031]

[0032] Q=W Q *F i ,K=W K ·F shared ,V=W V ·F shared (6);

[0033]

[0034] Where, Indicates the total loss; represents the generation loss; λ represents the weight of consistency regularization; represents the consistency regularization loss; Q, K, and V represent query features, key features, and value features, respectively; W Q 、W K and W V Represents the weight matrices of query features, key features, and value features respectively; F i and F i+1 Represent the feature representations obtained by the i-th and i+1-th iterations respectively; d k represents the dimension of the key feature K;

[0035] in,

[0036]

[0037] Preferably, in step S3, the lip shape movement is optimized from the perspectives of speech-driven consistency and local detail restoration;

[0038] Objective function for optimizing lip shape movement by restoring local details The expression is as follows:

[0039]

[0040] Where λ1, λ2 and λ3 are weight coefficients; z represents the discrete code obtained by the diffusion model, which is the optimization object; z refrepresents the discrete features obtained by discrete coding of the reference video; z orig represents the original discrete code; R(z) represents the Lipschitz continuity constraint;

[0041] in,

[0042]

[0043] Where z1 and z2 represent two elements in the discrete coding set Z; f(z1) and f(z2) represent the results of the function for calculating the Lipschitz continuity constraint on z1 and z2 respectively;

[0044] Objective function for optimizing lip movements using speech-driven consistency The expression is as follows:

[0045]

[0046] Where, represents reconstruction losses; represents the time continuity regularization loss function;

[0047]

[0048] Where z t and z t+1 They represent the discrete codes of the t-th frame and the t+1-th frame respectively; T-1 represents the number of video frames.

[0049] Therefore, the present invention adopts the above-mentioned digital human lip shape driving method based on multimodal fusion and consistency constraint, which has the following beneficial effects:

[0050] 1. Multimodal fusion facial feature representation: This innovatively integrates visual language models with general image segmentation technology to generate personalized facial feature representations through high- and low-frequency decoupling encoders. This method can comprehensively capture the global structure and local detail features of the face, taking into account the differences in motion characteristics of different facial regions to achieve more accurate personalized feature modeling. For example, the visual language model is used to obtain the linguistic description and structured statistical parameterized features of the facial appearance, while image segmentation technology is used to generate accurate segmentation masks. The combination of the two is encoded through high- and low-frequency decoupling to generate a feature representation that is both globally consistent and locally refined.

[0051] 2. QKV Consistency-Driven Speech-Lip Coupling: Based on the multimodal shared information between speech and facial features, a speech-lip dynamic coupling method based on QKV consistency constraints is proposed. By projecting speech and facial features into a shared semantic space, the generative network is guided to explicitly model their synergistic relationship with dynamic lip movements. Consistency regularization is introduced to enhance feature alignment, resulting in a high degree of consistency in the generated lip movements in terms of speech semantic consistency, natural dynamic expression, and personalized detail expression, effectively resolving the issue of inaccurate matching between speech and lip movements.

[0052] 3. Discrete Coding Optimization and Detail Enhancement: A generative framework based on discrete coding optimization is introduced. The discrete coding is predicted using a diffusion model. Using the discrete coding of the original reference video as a constraint, the Lipschitz continuity constraint is combined to optimize the coding distribution, enhancing the model's ability to fit the real-world distribution. Temporal continuity regularization is also introduced to improve the dynamic consistency of lip movements.

[0053] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flowchart of the digital human lip shape driving method based on multimodal fusion and consistency constraint according to the present invention;

[0055] Figure 2 This is a simulation example diagram of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purposes, technical solutions and advantages disclosed in the embodiments of the present invention clearer, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present invention and are not intended to limit the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, where the same or similar numbers throughout represent the same or similar elements or elements with the same or similar functions.

[0057] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or devices.

[0058] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0059] like Figure 1 and Figure 2 As shown in FIG, the digital human lip shape driving method based on multimodal fusion and consistency constraint includes the following steps:

[0060] S1. Construct a multimodal feature extraction and representation framework using a reference face image as input, and generate a personalized face feature representation based on the constructed multimodal feature extraction and representation framework; it should be noted that the reference face image is taken from the first frame (i.e., t=0) of the video.

[0061] Step S1 specifically includes the following steps:

[0062] S11. Input the reference face image I into the visual language model to generate a language description of the face appearance and structured statistical parameterized features S(I), providing a semantic-level quantitative expression for personalized feature modeling.

[0063] The language description and structured statistical parameterized features described in step S11 include geometric shape, texture distribution and regional feature description.

[0064] S12, inputting the reference face image I into the segmentation model for decomposition to obtain multiple segmentation regions, thereby generating a segmentation mask M(I) to capture local characteristics;

[0065] S13, using the high-low frequency decoupling encoder to perform feature fusion encoding on the language description, structured statistical parameterized features and segmentation mask, to generate a personalized facial feature representation F with both global and local characteristics face :

[0066] F face =ε(I,S(I),M(I)) (1);

[0067] Where ε represents the high- and low-frequency decoupling encoder.

[0068] In step S13, considering the differences in motion characteristics of different facial regions, high-frequency features and low-frequency features are processed separately during modeling;

[0069] The low-frequency features extract global structural features from the language description and structured statistical parameterized features S(I) and segmentation masks:

[0070]

[0071] Where, ε low (S(I),M(I)) represents low-frequency features; φ low Represents the function of calculating low-frequency features; N represents the total number of segmented regions; M k (I) represents the segmentation mask of the kth segmentation region; Tk (I) represents the language description and structured statistical parameterized features of the k-th segmented region; α k represents weight;

[0072] High-frequency features extract local texture and dynamic detail features from the reference face image I:

[0073] ε high (I)=H high (I) (3);

[0074] Where, ε high (I) represents high-frequency features, which are the feature representations obtained by extracting local texture and dynamic details from the reference face image I; H high (I) represents the decomposition of the reference face image I using the high-frequency encoder;

[0075] S2, using QKV consistency constraints to drive the dynamic coupling of the personalized facial feature representation generated in step S1 and the voice features to generate voice-driven lip movements;

[0076] In step S2, based on the QKV consistency constraint mechanism, the voice features and personalized face features are represented by F face Jointly projected into a shared semantic space to obtain the dynamic consistency and semantic relevance features of speech-driven lip movements F shared :

[0077] F shared =W shared ·(F audio +F face ) (4);

[0078] Where W shared The projection matrix representing the shared semantic space; F audio Represents phonetic features;

[0079] During the generation process, the QKV consistency constraint mechanism guides the generation network to explicitly model speech features and personalized face features F face The synergistic relationship between the dynamic lip shape movement and the personalized facial features F face Joint optimization with dynamic lip motion and generation of high-fidelity lip motion based on diffusion model.

[0080] The diffusion model expression is as follows:

[0081]

[0082] Q=W Q ·F i ,K=W K ·F shared ,V=WV ·F shared (6);

[0083]

[0084] Where, Indicates the total loss; represents the generation loss; λ represents the weight of consistency regularization; represents the consistency regularization loss, which is used to measure the degree of alignment of speech features and facial features in the shared space. The smaller the value, the more aligned they are in the shared space. Q, K, and V represent query features, key features, and value features, respectively. W Q 、W K and W V Represents the weight matrices of query features, key features, and value features respectively; F i and F i+1 Represent the feature representations obtained by the i-th and i+1-th iterations respectively; d k represents the dimension of the key feature K;

[0085] in,

[0086]

[0087] S3. Use discrete coding to encode the reference video to obtain discrete features, and limit the range of variation of the coding space based on the Lipschitz continuity constraint mechanism to optimize the lip movements generated in step S2.

[0088] In step S3, the lip shape movement is optimized based on the speech-driven consistency and local detail restoration respectively;

[0089] Objective function for optimizing lip shape movement by restoring local details The expression is as follows:

[0090]

[0091] Where λ1, λ2 and λ3 are weight coefficients; z represents the discrete code obtained by the diffusion model, which is the optimization object; z ref represents the discrete features obtained by encoding the reference video using discrete coding, which is used as a reference standard to constrain the optimization process and make the generated coding closer to the real reference; orig represents the original discrete code; R(z) represents the Lipschitz continuity constraint, which is used to limit the range of variation of the coding space, ensure smooth coding changes, and enhance the fidelity of generated details;

[0092] in,

[0093]

[0094] Where z1 and z2 represent two elements in the discrete coding set Z; f(z1) and f(z2) represent the results of the function for calculating the Lipschitz continuity constraint on z1 and z2 respectively;

[0095] Objective function for optimizing lip movements using speech-driven consistency The expression is as follows:

[0096]

[0097] Where, represents reconstruction losses; represents the temporal continuity regularization loss function, which is used to measure the difference between the discrete encodings of adjacent frames to improve the dynamic consistency of lip movements in the temporal dimension;

[0098]

[0099] Where z t and z t+1 They represent the discrete codes of the t-th frame and the t+1-th frame respectively; T-1 represents the number of video frames.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A digital human lip shape driving method based on multimodal fusion and consistency constraints, characterized by: The following steps are involved: S1. Using a reference facial image as input, a multimodal feature extraction and representation framework is constructed, and personalized facial feature representation is generated based on the constructed multimodal feature extraction and representation framework. S2, using QKV consistency constraints to drive the dynamic coupling of the personalized facial feature representation generated in step S1 and the voice features to generate voice-driven lip movements; S3. Use discrete coding to encode the reference video to obtain discrete features, and limit the range of variation of the coding space based on the Lipschitz continuity constraint mechanism to optimize the lip movements generated in step S2.

2. The digital human lip shape driving method based on multimodal fusion and consistency constraint according to claim 1 is characterized by: Step S1 specifically includes the following steps: S11. Input the reference face image I into the visual language model to generate a language description of the face appearance and structured statistical parameterized features S(I), providing a semantic-level quantitative expression for personalized feature modeling. S12, inputting the reference face image I into the segmentation model for decomposition to obtain multiple segmentation regions, thereby generating a segmentation mask M(I) to capture local characteristics; S13, using the high-low frequency decoupling encoder to perform feature fusion encoding on the language description, structured statistical parameterized features and segmentation mask, to generate a personalized facial feature representation F with both global and local characteristics face : F face =ε(I,S(I),M(I)) (1); Where ε represents the high- and low-frequency decoupling encoder.

3. The digital human lip shape driving method based on multimodal fusion and consistency constraint according to claim 2 is characterized by: The language description and structured statistical parameterized features described in step S11 include geometric shape, texture distribution and regional feature description.

4. The digital human lip shape driving method based on multimodal fusion and consistency constraint according to claim 2 is characterized by: In step S13, considering the differences in motion characteristics of different facial regions, high-frequency features and low-frequency features are processed separately during modeling; in Low-frequency features extract global structural features from language descriptions and structured statistical parameterized features S(I) and segmentation masks: Where, ε low (S(I),M(I)) represents low-frequency features; φ low Represents the function of calculating low-frequency features; N represents the total number of segmented regions; M k (I) represents the segmentation mask of the kth segmentation region; T k (I) represents the language description and structured statistical parameterized features of the k-th segmented region; α k represents weight; High-frequency features extract local texture and dynamic detail features from the reference face image I: ε high (I)=H high (I) (3); Where, ε high (I) represents high-frequency features, which are the feature representations obtained by extracting local texture and dynamic details from the reference face image I; H high (I) indicates the decomposition of the reference face image I using a high-frequency encoder.

5. The digital human lip shape driving method based on multimodal fusion and consistency constraint according to claim 4 is characterized by: In step S2, based on the QKV consistency constraint mechanism, the voice features and personalized face features are represented by F face Jointly projected into a shared semantic space to obtain the dynamic consistency and semantic relevance features of speech-driven lip movements F shared : F shared =w shared ·(F audio +F face ) (4); Where W shared The projection matrix representing the shared semantic space; F audio Represents phonetic features; During the generation process, the QKV consistency constraint mechanism guides the generation network to explicitly model speech features and personalized face features F face The synergistic relationship between the dynamic lip shape movement and the personalized facial features F face Joint optimization with dynamic lip motion and generation of high-fidelity lip motion based on diffusion model.

6. The digital human lip shape driving method based on multimodal fusion and consistency constraint according to claim 5 is characterized by: The diffusion model expression is as follows: Q=W Q ·F i ,K=W K ·F shared ,V=W V ·F shared (6); Where, Indicates the total loss; represents the generation loss; λ represents the weight of consistency regularization; represents the consistency regularization loss; Q, K, and V represent query features, key features, and value features, respectively; W Q 、W K and W V Represents the weight matrices of query features, key features, and value features respectively; F i and F i+1 Represent the feature representations obtained by the i-th and i+1-th iterations respectively; d k represents the dimension of the key feature K; in, 7. The digital human lip shape driving method based on multimodal fusion and consistency constraint according to claim 6 is characterized by: In step S3, the lip shape movement is optimized based on the speech-driven consistency and local detail restoration respectively; Objective function for optimizing lip shape movement by restoring local details The expression is as follows: Where λ1, λ2 and λ3 are weight coefficients; z represents the discrete code obtained by the diffusion model, which is the optimization object; z ref represents the discrete features obtained by discrete coding of the reference video; z orig represents the original discrete code; R(z) represents the Lipschitz continuity constraint; in, Where z1 and z2 represent two elements in the discrete coding set Z; f(z1) and f(z2) represent the results of the function for calculating the Lipschitz continuity constraint on z1 and z2 respectively; Objective function for optimizing lip movements using speech-driven consistency The expression is as follows: Where, represents reconstruction losses; represents the time continuity regularization loss function; Where z t and z t+1 They represent the discrete codes of the t-th frame and the t+1-th frame respectively; T-1 represents the number of video frames.

Images