Emotion-driven speaker face generation method, device and equipment based on 3DMM

Abstract

This application discloses a method, apparatus, and device for generating emotion-driven speaking faces based on 3DMM. The method includes: extracting emotion model expression coefficients from a preset face image and lip movement model coefficients from the corresponding audio using a 3DMM; performing eye coefficient change analysis based on the distance correlation method and SIPCA algorithm, obtaining relevant emotion model coefficients; using a preset LSTM network to fit and generate target face expression coefficients based on the relevant emotion model coefficients and lip movement model coefficients; and applying the target face expression coefficients to a target 3DMM to render a target emotion speaking face video. This application solves the technical problem that existing technologies cannot clearly define the influence of 3DMM coefficients on facial region control and do not pay attention to other facial expression features, resulting in a lack of realism in the generated face videos.

Description

Technical Field

[0001] This application relates to the field of intelligent face modeling technology, and in particular to a method, apparatus and device for generating emotion-driven speaking faces based on 3DMM. Background Technology

[0002] Previous methods based on facial landmarks and 2D stream-based audio-to-expression networks for generating speaking faces can produce distorted faces. SadTalker addresses this by using implicit 3D coefficient modulation to drive stylized audio. Besides lip movement features, emotion is another crucial factor in generating speaking faces. Currently, there are two main methods for achieving emotionally expressive speaking faces: obtaining emotional information through emotion tags and identifying emotional information from the audio. Speech emotion-aware speaking faces refer to videos that can match the spoken audio and exhibit conditional emotion.

[0003] Existing technologies cannot clearly define the control effect of coefficients in 3DMM on different areas of the face. Moreover, existing solutions only focus on lip movements and do not pay attention to other facial expression features. Therefore, the generated face videos have fixed emotions, which reduces the realism of the generated content. Summary of the Invention

[0004] This application provides a method, apparatus, and device for generating emotion-driven speaking faces based on 3DMM, which addresses the technical problem that existing technologies cannot clearly define the influence of 3DMM coefficients on facial region control and do not pay attention to other facial expression features, resulting in a lack of realism in the generated face videos.

[0005] In view of this, the first aspect of this application provides a method for generating emotion-driven speaking faces based on 3DMM, including:

[0006] The emotion model expression coefficients of the pre-set face image and the lip movement model coefficients of the corresponding audio were extracted using 3DMM.

[0007] Based on the distance correlation method and the SIPCA algorithm, the eye coefficient change analysis is performed according to the preset face image and the emotion model expression coefficient to obtain the correlation emotion model coefficient;

[0008] A preset LSTM network is used to fit and generate target facial expression coefficients based on the correlation emotion model coefficients and the lip movement model coefficients;

[0009] The target facial expression coefficients are applied to the target 3DMM to render a video of the target emotional speaking face.

[0010] Preferably, the step of extracting the emotion model expression coefficients of the preset face image and the lip movement model coefficients in the corresponding audio using 3DMM includes:

[0011] The expression coefficients of the emotion model are generated by extracting the expression coefficients from the pre-set face images using 3DMM.

[0012] The lip movement model coefficients are generated from the audio corresponding to the preset face image using the 3DMM.

[0013] Preferably, the step of performing eye coefficient change analysis based on the distance correlation method and SIPCA algorithm, according to the preset face image and the emotion model expression coefficient, to obtain the relevant emotion model coefficient, includes:

[0014] Extract multiple key points around the eyes from the preset face image, and calculate the vertical height of the key points to obtain the change in eye height;

[0015] Extract coefficient vectors from multiple time points in the emotion model expression coefficients, and calculate the inter-frame variation of the coefficient vectors to obtain the coefficient variation.

[0016] The correlation coefficient between the change in eye height and the change in coefficient is calculated using the distance correlation method, resulting in a correlation coefficient sequence.

[0017] The target correlation sequence is formed by selecting a predetermined number of correlation coefficients from the descending sequence of correlation coefficients.

[0018] The SIPCA algorithm is used to perform correlation optimization analysis based on the target correlation sequence to obtain the correlation sentiment model coefficients.

[0019] Preferably, the step of using a preset LSTM network to fit and generate target facial expression coefficients based on the relevance emotion model coefficients and the lip movement model coefficients includes:

[0020] A preset LSTM network is used to generate 3DMM expression coefficients of emotion based on the correlation emotion model coefficients.

[0021] The target facial expression coefficients are generated by fitting the 3DMM emotion expression coefficients and the lip movement model coefficients together.

[0022] Preferably, the step of applying the target facial expression coefficients to the target 3DMM to render a target emotional speaking face video includes:

[0023] The target facial expression coefficients are applied to the target 3DMM, and the target emotional speaking face video is rendered by combining identity feature coefficients, texture coefficients, face shape parameters and texture parameters.

[0024] The second aspect of this application provides a 3DMM-based emotion-driven speaking face generation device, including:

[0025] The coefficient extraction unit is used to extract the emotion model expression coefficients of the preset face image and the lip movement model coefficients in the corresponding audio through 3DMM.

[0026] The correlation analysis unit is used to perform eye coefficient change analysis based on the distance correlation method and the SIPCA algorithm, according to the preset face image and the emotion model expression coefficient, to obtain the correlation emotion model coefficient.

[0027] The coefficient fitting unit is used to generate target facial expression coefficients by fitting the coefficients of the relevant emotion model and the coefficients of the lip movement model using a preset LSTM network.

[0028] The face generation unit is used to apply the target facial expression coefficients to the target 3DMM to render a target emotional speaking face video.

[0029] Preferably, the coefficient extraction unit is specifically used for:

[0030] The expression coefficients of the emotion model are generated by extracting the expression coefficients from the pre-set face images using 3DMM.

[0031] The lip movement model coefficients are generated from the audio corresponding to the preset face image using the 3DMM.

[0032] Preferably, the correlation analysis unit is specifically used for:

[0033] Extract multiple key points around the eyes from the preset face image, and calculate the vertical height of the key points to obtain the change in eye height;

[0034] Extract coefficient vectors from multiple time points in the emotion model expression coefficients, and calculate the inter-frame variation of the coefficient vectors to obtain the coefficient variation.

[0035] The correlation coefficient between the change in eye height and the change in coefficient is calculated using the distance correlation method, resulting in a correlation coefficient sequence.

[0036] The target correlation sequence is formed by selecting a predetermined number of correlation coefficients from the descending sequence of correlation coefficients.

[0037] The SIPCA algorithm is used to perform correlation optimization analysis based on the target correlation sequence to obtain the correlation sentiment model coefficients.

[0038] Preferably, the coefficient fitting unit is specifically used for:

[0039] A preset LSTM network is used to generate 3DMM expression coefficients of emotion based on the correlation emotion model coefficients.

[0040] The target facial expression coefficients are generated by fitting the 3DMM emotion expression coefficients and the lip movement model coefficients together.

[0041] A third aspect of this application provides an emotion-driven speaking face generation device based on 3DMM, the device including a processor and a memory;

[0042] The memory is used to store program code and transmit the program code to the processor;

[0043] The processor is used to execute the emotion-driven speaking face generation method based on 3DMM as described in the first aspect, according to the instructions in the program code.

[0044] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0045] This application provides a 3DMM-based emotion-driven speaking face generation method, including: extracting the emotion model expression coefficients of a preset face image and the lip movement model coefficients in the corresponding audio using a 3DMM; performing eye coefficient change analysis based on the distance correlation method and the SIPCA algorithm, obtaining the relevant emotion model coefficients; using a preset LSTM network to fit and generate the target face expression coefficients based on the relevant emotion model coefficients and the lip movement model coefficients; and applying the target face expression coefficients to the target 3DMM to render the target emotion speaking face video.

[0046] This application provides a 3DMM-based emotion-driven speaking face generation method. It employs distance correlation and the SIPCA algorithm to quantify the correlation of coefficients generated by the 3DMM, thereby determining the system's influence on eye and lip control. Based on this, a pre-defined LSTM network is used to combine emotional facial features with lip coefficients in the audio for analysis, fitting target facial expression coefficients for emotion-driven facial expressions. Then, based on these coefficients, an emotional speaking face video is generated. This process simultaneously considers the emotional features of the eyes and the lip movement features in the audio, enabling accurate and natural generation of speaking faces with facial expressions, making the generated video more vivid and realistic, meeting the needs of real-world scenarios. Therefore, this application solves the technical problem of existing technologies failing to clearly define the influence of 3DMM coefficients on facial region control and neglecting other facial expression features, resulting in a lack of realism in the generated face videos. Attached Figure Description

[0047] Figure 1 A flowchart illustrating the emotion-driven speaking face generation method based on 3DMM provided in this application embodiment;

[0048] Figure 2A schematic diagram of the structure of the emotion-driven speaking face generation device based on 3DMM provided in the embodiments of this application;

[0049] Figure 3 This is a schematic diagram illustrating the process of extracting face information using Deep3DFaceReconstruction, as provided in an embodiment of this application.

[0050] Figure 4 Example images of facial expressions provided for embodiments of this application;

[0051] Figure 5 A schematic diagram illustrating the matrix calculation process of the emotion expression coefficient and lip movement coefficient provided in the embodiments of this application. Detailed Implementation

[0052] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0053] For easier understanding, please refer to Figure 1 The embodiments of the emotion-driven speaking face generation method based on 3DMM provided in this application include:

[0054] Step 101: Extract the emotion model expression coefficients of the preset face image and the lip movement model coefficients of the corresponding audio using 3DMM.

[0055] Further, step 101 includes:

[0056] The expression coefficients of the emotion model are generated by extracting the expression coefficients from the pre-set face images using 3DMM.

[0057] Lip motion model coefficients are generated from the audio corresponding to a pre-set face image using 3DMM.

[0058] It should be noted that this embodiment primarily uses 3DMM coefficients as intermediate motion representations, fitting facial images with expressions such as happiness, anger, and surprise to generate their corresponding 3DMM coefficients; then, 3DMM coefficients for lip movements are generated from the audio. The lip movement coefficients are extracted from facial images showing lip movements; these images are generated from audio using SadTalker. 3DMM includes coefficients for identity, expression, and texture. This embodiment mainly analyzes and fits the expression coefficients, then preserves the identity and texture features of the face during video generation.

[0059] Please see Figure 3 Based on Deep3DFaceReconstruction, the coefficients of a 3DMM, including identity, expression, and texture coefficients, can be determined from the face image and face coordinates. In a 3DMM, the face shape S and texture T can be decoupled as follows:

[0060]

[0061] in, These are average face shape and average texture, respectively; B id B exp B t Let represent the identity basis, representation basis, and texture basis, respectively, all scaled by standard deviation; α, β, and δ are the corresponding coefficient vectors for generating the 3D face. Where α∈R 80 and β∩R 64 They describe the basic identity of a person, B. id Coefficients and expression base B exp Coefficient. Therefore, the coefficient β can be expressed as:

[0062] β={β0,β1,β2,...β 62 ,β 63}

[0063] In the specific extraction process of emotion model expression coefficients and lip movement model coefficients, the expression base B in the 3DMM must first be extracted from pre-set face images with different emotions. exp The expression coefficient β, used to represent emotions such as happiness, anger, and surprise, is shown in [reference]. Figure 4 Therefore, the expression coefficient β of the emotion model with emotional faces can be obtained. emo :

[0064] β emo ={β emo0 ,β emo1 ,β emo2 ,...β emo62 ,β emo63}

[0065] Since the lip movement coefficients are extracted from the lip movement face image, and the lip movement face image is generated by SadTalker based on the input audio, it is necessary to preserve identity features and texture features when generating lip movement model coefficients from the audio. This yields the 3DMM identity feature coefficients α, expression coefficients β, and texture coefficients δ for the emotional face, where the lip expression coefficients β are the most significant. mouth for:

[0066] β mouth ={β mouth0 ,β mouth1 ,βmouth2 ,...β mouth62 ,β mouth63}

[0067] The above method can be used to extract coefficients for each frame of an image. For videos, coefficients can be extracted from consecutive video frames. The principle is the same, so I will not repeat it here.

[0068] Step 102: Based on the distance correlation method and the SIPCA algorithm, analyze the changes in eye coefficients according to the preset face images and emotion model expression coefficients to obtain the correlation emotion model coefficients.

[0069] Further, step 102 includes:

[0070] Extract multiple key points around the eyes from a pre-set face image and calculate the vertical height of the key points to obtain the change in eye height;

[0071] Extract coefficient vectors from multiple time points in the emotion model expression coefficients and calculate the inter-frame variation of the coefficient vectors to obtain the coefficient variation.

[0072] The correlation coefficient between the change in eye height and the change in coefficient was calculated using the distance correlation method, resulting in a correlation coefficient sequence.

[0073] Select a predetermined number of correlation coefficients from the descending sequence of correlation coefficients to form the target correlation sequence;

[0074] The SIPCA algorithm is used to perform correlation optimization analysis based on the target related sequence to obtain the correlation sentiment model coefficient.

[0075] It should be noted that key feature points around the eyes can be extracted from the preset face image, and then the vertical distance between these key points can be calculated to obtain the change in eye height. Assume that in each frame of the preset face image, the key point of the left eye is (x... 36 ,y 36 ), (x 37 ,y 37 ), ... (x) 41 ,y 41 The key point coordinates of the right eye are (x... 42 ,y 42 ), (x 43 ,y 43 ), ... (x) 47 ,y 47 The change in eye height is represented by calculating the average eye height; the change in left eye height H can be represented by the vertical distance between the upper and lower boundaries of the eye. left :

[0076]

[0077] Similarly, the height change H of the right eye right for:

[0078]

[0079] When there are inter-frame changes, suppose the change in eye height between frame j and frame j+1 can be expressed as:

[0080] ΔH (j) =H (j+1) -H (j)

[0081] ΔH (j) This refers to the change in eye height.

[0082] If we extract 64 emotion model expression coefficients β={β0,β1,β2,...β} to represent facial expressions... 62 ,β 63 Each frame of the image corresponds to a coefficient vector, which changes with time / frame count. Assuming that at frame j, the coefficient vector can be expressed as:

[0083]

[0084] in, This represents the i-th expression coefficient at frame j. This set of coefficients represents the expression state of each frame and changes with time and frame number. Specifically, during inter-frame changes, assuming the change in coefficients between frame j and frame (j+1), it is expressed as:

[0085] Δβ (j) =β (j+1) -β (j)

[0086] Where, Δβ (j) This represents the change in the 3DMM representation coefficients between frame j and frame j+1.

[0087] Assuming there are N frames of images, their corresponding eye height change sequence and coefficient change sequence can be expressed as:

[0088] ΔH=[ΔH (0) ,ΔH (1) ,...,ΔH (N-1) ]

[0089]

[0090] Since the expression coefficients of 3DMM may vary non-linearly or with mixed dependencies, especially when they are related to complex facial features, this embodiment uses a distance correlation method to capture non-linear dependencies. Specifically, for each ΔH and Δβ...i The distance correlation value between the two can be calculated using the following formula:

[0091] dCorr(ΔH,Δβ i )=Distance Correlation(ΔH,Δβ i )

[0092] Define dCov(X,Y) as the distance covariance:

[0093]

[0094] Where (X,Y) represent two vectors for similarity comparison, E represents the expected value, represents the mean of the random variables, and ||·|| represents the Euclidean distance. The specific calculation formula is as follows:

[0095]

[0096] Where dVar(ΔH) represents the distance variance of the eye height variation, and dVar(Δβ) represents the distance variance of the eye height variation. i ) represents the distance variance of the change in the i-th coefficient, dCorr(ΔH,Δβ) i () represents the distance correlation value between the change in the i-th coefficient and the change in eye height; the larger the correlation value, the greater the influence of the coefficient on the change in eye height. Therefore, comparing all distance correlation values ​​allows us to select the coefficient with the greatest influence on the change in eye height. The calculated distance correlation values ​​are sorted in descending order, with a quantity of i = 1, 2, 3, ..., 63, so we can select the coefficients from the previously preset quantity k:

[0097]

[0098] Where, {i0,i1,...,i k-1} represents the index of the group of coefficients with the highest correlation. The sequence consisting of the k most correlated coefficients is called the correlation coefficient sequence, which is expressed as: {β} i0 ,β i1 ,...,β i(k-1)}

[0099] To avoid overfitting or interference from redundant coefficients, this embodiment also employs the SIPCA algorithm to perform correlation optimization and filtering on the generated correlation coefficient sequence, obtaining some coefficients that have a significant impact on changes in eye height. Specifically, the k coefficients of each frame can form a data matrix K of size N×k, where N represents the number of frames, and each row represents the coefficients of one frame:

[0100]

[0101] Then, the data matrix K is standardized so that each column of data has zero mean and unit variance:

[0102]

[0103] Where μ is the column mean and σ is the column standard deviation.

[0104] The standardized matrix K' is used as the starting data for the SIPCA algorithm analysis. The goal is to find a set of sparse independent components Z and use them as an explanatory factor for changes in eye height. Z is specifically expressed as:

[0105] Z = SIPCA(K')

[0106] For a standardized matrix K', a weight matrix W of coefficients needs to be found first based on the SIPCA algorithm. This weight matrix W is then used to project matrix K' onto the principal component space, generating a sparse and independent component matrix Z. The optimization problem for this process is expressed as:

[0107]

[0108] The goal of this optimization process is to maximize the projected variance of matrix K' onto the new principal components W, i.e., to ensure that the principal components can explain as much variance as possible. The optimization process must satisfy the following constraints:

[0109] W T W = I

[0110] ||w i ||0≤s

[0111] W T W = I indicates that the principal components are orthogonal, which ensures the independence of the principal components; ||w i ||0≤s is a sparsity constraint, where each principal component vector w i The sparsity is restricted to s non-zero elements, controlling the sparsity of each principal component so that each principal component consists of only a few coefficients. Detailed optimization methods can be implemented based on existing technologies and will not be elaborated here; moreover, the sparse independent component Z obtained is the correlation emotion model coefficient obtained in this embodiment, which is a small number of emotion model coefficients obtained after further correlation optimization analysis.

[0112] Step 103: Use a preset LSTM network to fit and generate the target facial expression coefficients based on the correlation emotion model coefficients and lip movement model coefficients.

[0113] Further, step 103 includes:

[0114] A pre-defined LSTM network is used to generate 3DMM expression coefficients of emotion based on the coefficients of the relevance emotion model.

[0115] The target facial expression coefficients are generated by fitting the 3DMM emotion expression coefficients and lip movement model coefficients together.

[0116] It should be noted that the correlation emotion model coefficients obtained based on the above correlation analysis can determine which types of coefficients in a face image are related to the eyes, i.e., these types of coefficients can control eye movements. Then, a preset LSTM network can be used to generate these correlation coefficients that can control eye movements, i.e., emotion 3DMM expression coefficients, and participate in subsequent coefficient fitting operations. It is evident that the coefficient selection in the above correlation analysis is a selection of specific coefficient types, while the preset LSTM network here is used to generate coefficients of the corresponding types.

[0117] Specifically, the preset LSTM network in this embodiment is based on the lip movement 3DMM coefficient β. mouth Generate coefficients with emotional expression. Specifically, to train the defined initial LSTM, specific facial parameters related to the eyes or eyebrows can be extracted from the input face image, expressed as β. emo Then, extract other 3DMM coefficients from the audio besides the specific facial parameters mentioned above. These generally refer to lip movement coefficients, expressed as β. mouth,t Then β mouth,t As input X, specific facial parameters can be used as repeated image features as output, i.e., output labels Y:

[0118] X = [β] mouth,1 ,β mouth,2 ,...,β mouth,T ]

[0119] Y = [β] emo ,β emo ,...,β emo ]

[0120] The loss function of an LSTM network is expressed as:

[0121]

[0122] The generated emotion 3DMM expression coefficients are expressed as follows:

[0123]

[0124] The emotion 3DMM expression coefficient generated corresponding to the face image in frame t is: Based on this, if the lip movement model coefficient corresponding to the audio graph of frame t is β mouth,t Therefore, the target facial expression coefficient fitting operation can be expressed as:

[0125]

[0126] Where w1 and w2 are the weights for controlling eye and lip changes, respectively, on the fitted facial expression; β object,t It is a 64×m matrix, where m is the number of video frames used to generate the human face speaking; w1 and w2 are both 64×64 matrices. For the detailed calculation process, please refer to [link / reference]. Figure 5 .

[0127] w1 indicates from The specific rows and columns extracted, w2 represents from β mouth,t The remaining portion is extracted except for the position specified by w1; the desired portion can be extracted from the two matrices using two weights, β is selected. emo The position of the i-th row and j-th column in β: emo [i,j]; Choose β mouth The portion excluding the position in row i and column j:

[0128] Then the matrix β of the target facial expression coefficients object,t The formula for generating the formula is as follows:

[0129]

[0130] Based on this, the location of the coefficient most relevant to changes in eye height can be obtained. It can be seen that changes in eye height are affected by β. emo Influenced by β, changes in lip movement are affected. mouth Impact. Furthermore, it's possible to preserve facial features and texture while adjusting expression coefficients.

[0131] When people express emotions through their faces, the distance and scale between key facial features change. For example, the distance between eyebrows is smaller when angry than when surprised, the mouth is narrower when in a neutral expression than when happy, and the mouth is wider when sad than when in a neutral expression. Therefore, this embodiment assigns different weights to each expression so that the face can be interpreted as different expressions.

[0132] The parameters in 3DMM are expressed as follows:

[0133] Identity feature coefficient: α object =α mouth

[0134] Expression coefficient: β object =w1β emo +w2β mouth

[0135] Texture coefficient: δ object =δ mouth

[0136] Face shape:

[0137] Texture:

[0138] Step 104: Apply the target facial expression coefficients to the target 3DMM to render the target emotional speaking face video.

[0139] Further, step 104 includes:

[0140] The target facial expression coefficients are applied to the target 3DMM, and the target emotional speaking face video is rendered by combining identity feature coefficients, texture coefficients, face shape parameters and texture parameters.

[0141] It should be noted that by applying the fitted target facial expression coefficients to the target 3DMM, a face with emotion can be generated. This process utilizes FaceRender in SadTalker to render a video based on the generated 3D perceived image animation, i.e., a video of the target emotional speaking face.

[0142] Specifically, FaceRender technology applies the calculated 3DMM coefficients to generate a frame-by-frame video of the original human face. FaceRender maps 3DMM coefficients onto the actual facial animation using a network model, i.e., the target 3DMM. This process relies on various key parameters in the 3DMM, including identity feature coefficients, target facial expression coefficients, texture coefficients, face shape parameters, and texture parameters. Through precise control of these coefficients, detailed dynamic rendering of the target face is achieved. All generated frames are combined into a complete video, ultimately outputting a speaking face video, i.e., a speaking face video expressing the target emotion.

[0143] The emotion-driven speaking face generation method based on 3DMM provided in this application uses distance correlation and SIPCA algorithm to quantitatively analyze the correlation of coefficients generated by 3DMM, thereby determining the system's influence on eye and lip control. Based on this, a pre-set LSTM network is used to combine and analyze emotional facial features with lip coefficients in the audio, fitting target facial expression coefficients for emotion-driven facial expressions. Then, an emotional speaking face video is generated based on these coefficients. This process simultaneously considers the emotional features of the eyes and the lip movement features in the audio, enabling accurate and natural generation of speaking faces with facial expressions, making the generated video more vivid and realistic, meeting the needs of real-world scenarios. Therefore, this application solves the technical problem of existing technologies failing to clearly define the influence of 3DMM coefficients on facial region control and neglecting other facial expression features, resulting in a lack of realism in the generated face videos.

[0144] For easier understanding, please refer to Figure 2 This application provides an embodiment of a 3DMM-based emotion-driven speaking face generation device, including:

[0145] The coefficient extraction unit 201 is used to extract the emotion model expression coefficients of the preset face image and the lip movement model coefficients in the corresponding audio through 3DMM.

[0146] The correlation analysis unit 202 is used to analyze the changes in eye coefficients based on the distance correlation method and the SIPCA algorithm, according to the preset face image and emotion model expression coefficients, to obtain the correlation emotion model coefficients.

[0147] The coefficient fitting unit 203 is used to generate target facial expression coefficients by fitting the coefficients of the relevant emotion model and the coefficients of the lip movement model using a preset LSTM network.

[0148] The face generation unit 204 is used to apply the target face expression coefficients to the target 3DMM to render the target emotional speaking face video.

[0149] Furthermore, the coefficient extraction unit 201 is specifically used for:

[0150] The expression coefficients of the emotion model are generated by extracting the expression coefficients from the pre-set face images using 3DMM.

[0151] Lip motion model coefficients are generated from the audio corresponding to a pre-set face image using 3DMM.

[0152] Furthermore, the correlation analysis unit 202 is specifically used for:

[0153] Extract multiple key points around the eyes from a pre-set face image and calculate the vertical height of the key points to obtain the change in eye height;

[0154] Extract coefficient vectors from multiple time points in the emotion model expression coefficients and calculate the inter-frame variation of the coefficient vectors to obtain the coefficient variation.

[0155] The correlation coefficient between the change in eye height and the change in coefficient was calculated using the distance correlation method, resulting in a correlation coefficient sequence.

[0156] Select a predetermined number of correlation coefficients from the descending sequence of correlation coefficients to form the target correlation sequence;

[0157] The SIPCA algorithm is used to perform correlation optimization analysis based on the target related sequence to obtain the correlation sentiment model coefficient.

[0158] Furthermore, the coefficient fitting unit 203 is specifically used for:

[0159] A pre-defined LSTM network is used to generate 3DMM expression coefficients of emotion based on the coefficients of the relevance emotion model.

[0160] The target facial expression coefficients are generated by fitting the 3DMM emotion expression coefficients and lip movement model coefficients together.

[0161] This application also provides a 3DMM-based emotion-driven speaking face generation device, which includes a processor and a memory;

[0162] The memory is used to store program code and transfer the program code to the processor;

[0163] The processor is used to execute the emotion-driven speaking face generation method based on 3DMM in the above method embodiment according to the instructions in the program code.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0167] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods described in the various embodiments of this application through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0168] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A 3DMM-based emotion-driven speaking face generation method, characterized in that, include: The emotion model expression coefficients of the pre-set face image and the lip movement model coefficients of the corresponding audio were extracted using 3DMM. Based on the distance correlation method and the SIPCA algorithm, eye coefficient changes are analyzed according to the preset face image and the emotion model expression coefficients to obtain the relevant emotion model coefficients. The specific process is as follows: Extract multiple key points around the eyes from the preset face image, and calculate the vertical height of the key points to obtain the change in eye height; Extract coefficient vectors from multiple time points in the emotion model expression coefficients, and calculate the inter-frame variation of the coefficient vectors to obtain the coefficient variation. The correlation coefficient between the change in eye height and the change in coefficient is calculated using the distance correlation method, resulting in a correlation coefficient sequence. The target correlation sequence is formed by selecting a predetermined number of correlation coefficients from the descending sequence of correlation coefficients. The SIPCA algorithm is used to perform correlation optimization analysis based on the target correlation sequence to obtain the correlation sentiment model coefficients. A preset LSTM network is used to fit and generate target facial expression coefficients based on the correlation emotion model coefficients and the lip movement model coefficients; The target facial expression coefficients are applied to the target 3DMM to render a video of the target emotional speaking face.

2. The emotion-driven speaking face generation method based on 3DMM according to claim 1, characterized in that, The step of extracting the emotion model expression coefficients of the pre-set facial image and the lip movement model coefficients in the corresponding audio using 3DMM includes: The expression coefficients of the emotion model are generated by extracting the expression coefficients from the pre-set face images using 3DMM. The lip movement model coefficients are generated from the audio corresponding to the preset face image using the 3DMM.

3. The emotion-driven speaking face generation method based on 3DMM according to claim 1, characterized in that, The step of using a preset LSTM network to fit and generate target facial expression coefficients based on the relevance emotion model coefficients and the lip movement model coefficients includes: A preset LSTM network is used to generate 3DMM expression coefficients of emotion based on the correlation emotion model coefficients. The target facial expression coefficients are generated by fitting the 3DMM emotion expression coefficients and the lip movement model coefficients together.

4. The emotion-driven speaking face generation method based on 3DMM according to claim 1, characterized in that, The step of applying the target facial expression coefficients to the target 3DMM to render a target emotional speaking face video includes: The target facial expression coefficients are applied to the target 3DMM, and the target emotional speaking face video is rendered by combining identity feature coefficients, texture coefficients, face shape parameters and texture parameters.

5. A 3DMM-based emotion-driven speaking face generation device, characterized in that, include: The coefficient extraction unit is used to extract the emotion model expression coefficients of the preset face image and the lip movement model coefficients in the corresponding audio through 3DMM. The correlation analysis unit is used to perform eye coefficient change analysis based on the distance correlation method and the SIPCA algorithm, according to the preset face image and the emotion model expression coefficient, to obtain the correlation emotion model coefficient. Specifically, the correlation analysis unit is used for: Extract multiple key points around the eyes from the preset face image, and calculate the vertical height of the key points to obtain the change in eye height; Extract coefficient vectors from multiple time points in the emotion model expression coefficients, and calculate the inter-frame variation of the coefficient vectors to obtain the coefficient variation. The correlation coefficient between the change in eye height and the change in coefficient is calculated using the distance correlation method, resulting in a correlation coefficient sequence. The target correlation sequence is formed by selecting a predetermined number of correlation coefficients from the descending sequence of correlation coefficients. The SIPCA algorithm is used to perform correlation optimization analysis based on the target correlation sequence to obtain the correlation sentiment model coefficients. The coefficient fitting unit is used to generate target facial expression coefficients by fitting the coefficients of the relevant emotion model and the coefficients of the lip movement model using a preset LSTM network. The face generation unit is used to apply the target facial expression coefficients to the target 3DMM to render a target emotional speaking face video.

6. The emotion-driven speaking face generation device based on 3DMM according to claim 5, characterized in that, The coefficient extraction unit is specifically used for: The expression coefficients of the emotion model are generated by extracting the expression coefficients from the pre-set face images using 3DMM. The lip movement model coefficients are generated from the audio corresponding to the preset face image using the 3DMM.

7. The emotion-driven speaking face generation device based on 3DMM according to claim 5, characterized in that, The coefficient fitting unit is specifically used for: A preset LSTM network is used to generate 3DMM expression coefficients of emotion based on the correlation emotion model coefficients. The target facial expression coefficients are generated by fitting the 3DMM emotion expression coefficients and the lip movement model coefficients together.

8. A 3DMM-based emotion-driven speech face generation device, characterized in that, The device includes a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the emotion-driven speaking face generation method based on any one of claims 1-4 according to the instructions in the program code.

Images