Emotion-driven speaking face generation method, device and equipment based on 3DMM

Abstract

The invention discloses an emotion-driven speaking face generation method, device and equipment based on 3DMM. The method comprises the following steps: respectively extracting an emotion model expression coefficient of a preset face image and a lip motion model coefficient in a corresponding audio through the 3DMM; based on a distance correlation method and an SIPCA algorithm, performing eye coefficient change analysis according to a preset face image and an emotion model expression coefficient to obtain a correlation emotion model coefficient; adopting a preset LSTM network to perform fitting according to the correlation emotion model coefficient and the lip motion model coefficient to generate a target face expression coefficient; and applying the target facial expression coefficient to the target 3DMM, and rendering a target emotion speaking face video. According to the method and the device, the technical problem that the generated face video lacks a sense of reality due to the fact that the influence condition of the 3DMM coefficient on the face region control cannot be determined and other facial expression features are not concerned in the prior art can be solved.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent face modeling, and in particular to a method, device and equipment for generating emotion-driven speaking faces based on 3DMM. Background Art

[0002] Previous methods based on facial landmarks and audio-to-expression networks based on 2D streams may produce distorted faces. SadTalker modulates the stylized audio drive through implicit 3D coefficients. In addition to lip movement features, emotion is another important factor in generating talking faces. Currently, emotional talking faces are mainly achieved by obtaining emotional information from emotion tags and identifying emotional information in audio. Speech emotion-aware talking faces means that the generated talking face video can match the speech audio and show conditional emotions.

[0003] The existing technology cannot clearly define the control effect of the coefficients in 3DMM on different facial areas, and the existing solutions only focus on lip movements and ignore other facial expression features. Therefore, the generated face videos have fixed emotions, which reduces the realism of the generated content. Summary of the invention

[0004] The present application provides a method, device and equipment for generating emotion-driven speaking faces based on 3DMM, which is used to solve the technical problem that the prior art cannot clearly define the influence of 3DMM coefficients on facial area control and does not pay attention to other facial expression features, resulting in the lack of realism in the generated face video.

[0005] In view of this, the first aspect of the present application provides a method for generating emotion-driven speaking faces based on 3DMM, comprising:

[0006] The emotion model expression coefficients of the preset face image and the lip movement model coefficients in the corresponding audio are extracted through 3DMM;

[0007] Based on the distance correlation method and the SIPCA algorithm, the eye coefficient change analysis is performed according to the preset face image and the emotion model expression coefficient to obtain the correlation emotion model coefficient;

[0008] Using a preset LSTM network to fit the relevant emotion model coefficients and the lip movement model coefficients to generate the target facial expression coefficients;

[0009] The target facial expression coefficient is applied to the target 3DMM to render the target emotional speaking face video.

[0010] Preferably, extracting the emotion model expression coefficients of the preset face image and the lip movement model coefficients in the corresponding audio by 3DMM respectively includes:

[0011] Extract the expression coefficients in the preset face image through 3DMM to generate the expression coefficients of the emotion model;

[0012] The lip motion model coefficients are generated from the audio corresponding to the preset face image through the 3DMM.

[0013] Preferably, the distance correlation method and the SIPCA algorithm are used to perform eye coefficient change analysis according to the preset face image and the emotion model expression coefficient to obtain the correlation emotion model coefficient, including:

[0014] Extracting a plurality of key points around the eyes in the preset face image, and calculating the vertical heights of the key points to obtain the eye height change;

[0015] Extracting multiple coefficient vectors at different times from the emotion model expression coefficients, and calculating the inter-frame variation of the coefficient vectors to obtain the coefficient variation;

[0016] Calculating the correlation coefficient between the eye height change and the coefficient change by using a distance correlation method to obtain a correlation coefficient sequence;

[0017] Selecting a preset number of correlation coefficients from the correlation coefficient sequence arranged in descending order to form a target correlation sequence;

[0018] The SIPCA algorithm is used to perform correlation optimization analysis according to the target correlation sequence to obtain the correlation emotion model coefficient.

[0019] Preferably, the method of using a preset LSTM network to fit and generate a target facial expression coefficient according to the correlation emotion model coefficient and the lip movement model coefficient includes:

[0020] Using a preset LSTM network to generate emotion 3DMM expression coefficients according to the correlation emotion model coefficients;

[0021] The emotion 3DMM expression coefficient and the lip movement model coefficient are combined to generate the target facial expression coefficient.

[0022] Preferably, applying the target facial expression coefficients to the target 3DMM to render the target emotional speaking face video includes:

[0023] The target facial expression coefficient is applied to the target 3DMM, and the target emotional speaking face video is rendered by combining the identity feature coefficient, texture coefficient, face shape parameter and texture parameter.

[0024] The second aspect of the present application provides an emotion-driven speaking face generation device based on 3DMM, comprising:

[0025] A coefficient extraction unit, used for respectively extracting the emotion model expression coefficients of the preset face image and the lip movement model coefficients in the corresponding audio through 3DMM;

[0026] A correlation analysis unit, configured to perform eye coefficient change analysis based on the preset face image and the emotion model expression coefficient based on a distance correlation method and a SIPCA algorithm, to obtain a correlation emotion model coefficient;

[0027] A coefficient fitting unit, used for fitting and generating a target facial expression coefficient according to the correlation emotion model coefficient and the lip movement model coefficient by using a preset LSTM network;

[0028] The face generation unit is used to apply the target facial expression coefficients to the target 3DMM to render the target emotional speaking face video.

[0029] Preferably, the coefficient extraction unit is specifically used for:

[0030] Extract the expression coefficients in the preset face image through 3DMM to generate the expression coefficients of the emotion model;

[0031] The lip motion model coefficients are generated from the audio corresponding to the preset face image through the 3DMM.

[0032] Preferably, the correlation analysis unit is specifically used for:

[0033] Extracting a plurality of key points around the eyes in the preset face image, and calculating the vertical heights of the key points to obtain the eye height change;

[0034] Extracting multiple coefficient vectors at different times from the emotion model expression coefficients, and calculating the inter-frame variation of the coefficient vectors to obtain the coefficient variation;

[0035] Calculating the correlation coefficient between the eye height change and the coefficient change by using a distance correlation method to obtain a correlation coefficient sequence;

[0036] Selecting a preset number of correlation coefficients from the correlation coefficient sequence arranged in descending order to form a target correlation sequence;

[0037] The SIPCA algorithm is used to perform correlation optimization analysis according to the target correlation sequence to obtain the correlation emotion model coefficient.

[0038] Preferably, the coefficient fitting unit is specifically used for:

[0039] Using a preset LSTM network to generate emotion 3DMM expression coefficients according to the correlation emotion model coefficients;

[0040] The emotion 3DMM expression coefficient and the lip movement model coefficient are combined to generate the target facial expression coefficient.

[0041] The third aspect of the present application provides an emotion-driven speaking face generation device based on 3DMM, the device comprising a processor and a memory;

[0042] The memory is used to store program code and transmit the program code to the processor;

[0043] The processor is used to execute the emotion-driven speaking face generation method based on 3DMM described in the first aspect according to the instructions in the program code.

[0044] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0045] In the present application, a method for generating an emotion-driven speaking face based on 3DMM is provided, including: extracting the emotion model expression coefficients of a preset face image and the lip movement model coefficients in the corresponding audio through 3DMM; based on the distance correlation method and the SIPCA algorithm, analyzing the change of eye coefficients according to the preset face image and the emotion model expression coefficients to obtain the correlation emotion model coefficients; using a preset LSTM network to fit the correlation emotion model coefficients and the lip movement model coefficients to generate the target face expression coefficients; applying the target face expression coefficients to the target 3DMM to render the target emotion speaking face video.

[0046] The present application provides a method for generating emotion-driven speaking faces based on 3DMM, which uses the distance correlation method and the SIPCA algorithm to perform quantitative correlation analysis on the coefficients generated by 3DMM, so as to determine the influence of the system on the control of eyes and lips; on this basis, a preset LSTM network is used to combine and analyze the emotional facial features with the lip coefficients in the audio, and the target facial expression coefficients of emotion-driven facial expressions are fitted; then, based on this coefficient, an emotional speaking face video is generated; this process simultaneously considers the emotional characteristics of the facial eyes and the lip movement characteristics in the audio, and can accurately and naturally generate speaking faces with facial expressions, making the generated videos more vivid and realistic, and meeting the needs of actual scenarios. Therefore, the present application can solve the technical problem that the prior art cannot clearly define the influence of the 3DMM coefficients on the control of the facial area, and does not pay attention to other facial expression features, resulting in the lack of realism in the generated face video. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a method for generating emotion-driven speaking faces based on 3DMM provided in an embodiment of the present application;

[0048] Figure 2A schematic diagram of the structure of a 3DMM-based emotion-driven speaking face generation device provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the Deep3DFaceReconstruction process for extracting facial information provided in an embodiment of the present application;

[0050] Figure 4 An example diagram of facial expressions provided in the embodiments of the present application;

[0051] Figure 5 A schematic diagram of the matrix calculation process of the emotion expression coefficient and the lip movement coefficient provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] For easier understanding, see Figure 1 , the embodiment of the method for generating emotion-driven speaking faces based on 3DMM provided in the present application includes:

[0054] Step 101: extract the emotion model expression coefficients of the preset face image and the lip movement model coefficients in the corresponding audio through 3DMM.

[0055] Furthermore, step 101 includes:

[0056] Extract the expression coefficients in the preset face image through 3DMM to generate the expression coefficients of the emotion model;

[0057] The lip motion model coefficients are generated from the audio corresponding to the preset face image through 3DMM.

[0058] It should be noted that this embodiment mainly uses the coefficients of 3DMM as the intermediate motion representation, and fits facial images with expressions such as happiness, anger and surprise to generate the corresponding 3DMM coefficients; then the 3DMM coefficients of lip movement are generated from the audio. The lip movement coefficients are extracted from the lip movement facial images; and the lip movement facial images are generated from the audio through SadTalker. 3DMM includes coefficients such as identity, expression and texture. This embodiment mainly analyzes and fits the expression coefficients, and then retains the identity features and texture features of the face when generating the video.

[0059] See also Figure 3 , based on Deep3DFaceReconstruction, the coefficients of 3DMM can be determined according to the face image and face coordinates, including coefficients such as identity, expression and texture. In 3DMM, face shape S and texture T can be decoupled into:

[0060]

[0061]

[0062] in, , They are average face shape and average texture respectively; , , denote the identity basis, expression basis, and texture basis, respectively, all scaled by standard deviation; , , is the corresponding coefficient vector for generating a 3D face. and Describes the identity of the person Coefficients and expression bases So we can put the coefficient It is expressed as:

[0063]

[0064] In the specific process of extracting the emotion model expression coefficients and lip movement model coefficients, we first extract the expression coefficients of 3DMM from the preset face images with different emotions. The expression coefficient , used to express emotions such as happiness, anger, surprise, etc., see Figure 4 . From this, we can get the expression coefficient of the emotion model with emotional faces :

[0065]

[0066] Since the lip motion coefficients are extracted from the lip motion face image, and the lip motion face image is generated by SadTalker based on the input audio, it is necessary to retain the identity features and texture features when generating the lip motion model coefficients in the audio to obtain the 3DMM identity feature coefficients with emotional faces. , expression coefficient and texture coefficient , where the lip expression coefficient of emotional face is for:

[0067]

[0068] The above method can be used to extract coefficients for each frame of image, and the coefficients of continuous video frames can be extracted for video. The principle is the same and will not be repeated here.

[0069] Step 102: Based on the distance correlation method and the SIPCA algorithm, the eye coefficient change analysis is performed according to the preset face image and the emotion model expression coefficient to obtain the correlation emotion model coefficient.

[0070] Furthermore, step 102 includes:

[0071] Extract multiple key points around the eyes in the preset face image, and calculate the vertical height of the key points to obtain the change in eye height;

[0072] Extracting multiple coefficient vectors at different times from the emotion model expression coefficients, and calculating the inter-frame variation of the coefficient vectors to obtain the coefficient variation;

[0073] The distance correlation method is used to calculate the correlation coefficient between the change in eye height and the change in coefficient, and a correlation coefficient sequence is obtained;

[0074] Selecting a preset number of correlation coefficients from the correlation coefficient sequence arranged in descending order to form a target correlation sequence;

[0075] The SIPCA algorithm is used to perform correlation optimization analysis based on the target correlation sequence to obtain the correlation sentiment model coefficient.

[0076] It should be noted that the key feature points around the eyes can be extracted from the preset face image, and then the vertical distance of these key points can be calculated to obtain the eye height change. Assume that in each frame of the preset face image, the key point of the left eye is , ,…… , the key point coordinates of the right eye are , ,…… The change in eye height is expressed by calculating the average height of the eyes; the vertical distance between the upper and lower boundaries of the eyes can be used to express the change in height of the left eye. :

[0077]

[0078] Similarly, the height change of the right eye for:

[0079]

[0080] When changing between frames, it is assumed that the change in eye height between the jth frame and the j+1th frame can be expressed as:

[0081]

[0082] This is the change in eye height.

[0083] If 64 emotion model expression coefficients are extracted to represent facial expressions , then each frame of the image will correspond to a coefficient vector, and this coefficient vector will change with time / frame number; assuming that at frame j, the coefficient vector can be expressed as:

[0084]

[0085] in, represents the i-th expression coefficient at the j-th frame. This coefficient set is the expression state of each frame, which will change with time and frame number. Specifically, when changing between frames, assuming that between the j-th frame and the j+1-th frame, the coefficient change is expressed as:

[0086]

[0087] in, Indicates the coefficient change of the 3DMM expression coefficient between the j-th frame and the j+1-th frame.

[0088] Assuming there are N frames of images, the corresponding eye height change sequence and coefficient change sequence can be expressed as:

[0089]

[0090]

[0091] Since the 3DMM expression coefficient may vary nonlinearly or with mixed dependencies, especially when it is related to the complex features of the expression, this embodiment uses the distance correlation method to capture the nonlinear dependency. and , the distance correlation value between the two can be calculated using the following formula:

[0092]

[0093] definition is the distance covariance:

[0094]

[0095] in, Represents two vectors for similarity comparison, represents the expected value, represents the mean value of the random variable, Represents the Euclidean distance. Specifically, the following calculation formula can be obtained:

[0096]

[0097] in, The distance variance representing the eye height variation, represents the distance variance of the change in the i-th coefficient, Represents the distance correlation value between the change in the i-th coefficient and the change in the eye height; the larger the correlation value, the greater the influence of the coefficient on the change in eye height. Therefore, by comparing all the distance correlation values, the coefficient with the greatest influence on the change in eye height can be selected. The calculated distance correlation values ​​are arranged in descending order, the number is i=1,2,3,...,63, so the coefficients of the preset number k can be selected:

[0098]

[0099] in, Represents the index of a group of coefficients with the greatest correlation. The sequence of the selected k coefficients with the greatest correlation, that is, the correlation coefficient sequence, is expressed as: .

[0100] In order to avoid overfitting or redundant coefficient interference, this embodiment also uses the SIPCA algorithm to perform correlation optimization and screening operations on the generated correlation coefficient sequence to obtain some coefficients that have a greater impact on the change in eye height. Specifically, it can be seen that the k coefficients of each frame image can form a data matrix , whose size is ,in, Represents the number of frames, then each row represents the coefficient of one frame:

[0101]

[0102] Then the data matrix Standardize the data so that each column has zero mean and unit variance:

[0103]

[0104] in, is the column mean, is the column standard deviation.

[0105] The normalized matrix As the actual data analyzed by the SIPCA algorithm, the goal is to find a set of sparse independent components , and used it as an explanatory factor for changes in eye height, The specific expression is:

[0106]

[0107] For the standardized matrix , we need to find a coefficient weight matrix based on the SIPCA algorithm , through the weight matrix The matrix Projecting into the principal component space to generate a sparse independent component matrix , the optimization problem of this process is expressed as:

[0108]

[0109] The goal of this optimization process is to maximize the matrix In the new principal component The projected variance on , that is, to ensure that the principal component can explain as much variance as possible. The optimization process needs to meet the following constraints:

[0110]

[0111]

[0112] It means that the principal components are mutually orthogonal, which ensures the independence of the principal components; is a sparsity constraint, each principal component vector The sparsity of The sparsity of each principal component is controlled within the non-zero elements, so that each principal component is composed of only a few coefficients. The detailed optimization solution method can be implemented based on the existing technology and will not be described here. That is, the correlation emotion model coefficient required in this embodiment is a small amount of emotion model coefficients obtained by screening after re-correlation optimization analysis.

[0113] Step 103: Use a preset LSTM network to fit the relevant emotion model coefficients and the lip movement model coefficients to generate the target facial expression coefficients.

[0114] Further, step 103 includes:

[0115] The preset LSTM network is used to generate the emotion 3DMM expression coefficient according to the correlation emotion model coefficient;

[0116] The emotion 3DMM expression coefficient and the lip motion model coefficient are combined to generate the target facial expression coefficient.

[0117] It should be noted that the coefficients of the correlation emotion model obtained by the above correlation analysis can determine which types of coefficients in the face image are related to the eyes, that is, these types of coefficients can control the eye movements; then the preset LSTM network can be used to generate such correlation coefficients that can control the eye movements, that is, the emotion 3DMM expression coefficients, and participate in the subsequent coefficient fitting operation. It can be seen that the coefficient screening of the above correlation analysis is the screening of specific coefficient types, and the preset LSTM network here is used to generate coefficients of the corresponding type.

[0118] Specifically, the preset LSTM network of this embodiment is based on the lip movement 3DMM coefficients In order to train the defined initial LSTM, input data extraction can be performed, that is, specific facial parameters are extracted from the face image. These parameters are related to the eyes or eyebrows and are expressed as ; Then extract other 3DMM coefficients from the audio except for the above specific facial parameters, generally referring to the lip movement coefficients, expressed as Then As input X, specific facial parameters can be output as repeated image features, that is, output label Y:

[0119]

[0120]

[0121] The loss function of the LSTM network is expressed as:

[0122]

[0123] The generated emotion 3DMM expression coefficient is expressed as:

[0124]

[0125] The 3DMM expression coefficient of the emotion generated corresponding to the face image in the tth frame is: Based on , if the lip motion model coefficient corresponding to the t-th frame audio image is , then the target facial expression coefficient fitting operation can be expressed as:

[0126]

[0127] in, , are the weights of the contribution of controlling eye changes and controlling lip changes to the fitted expression respectively; is a A matrix of size, The number of video frames to generate the face speaking; , Both For the specific calculation process, please refer to Figure 5 .

[0128] Indicates from Extract specific rows and columns from Indicates from Re-extracted except Other parts outside the specified position; through two weights, the required parts can be extracted from the two matrices, selecting The position of row i and column j in is: ;choose Remove the rest of the position at row i and column j: .

[0129] Then the matrix of the target face expression coefficients is The generation formula is as follows:

[0130]

[0131] Based on this, we can get the coefficient position most related to the change of eye height. The lip movements are affected by In addition, the identity characteristics and texture of the face can be preserved while adjusting the expression coefficient.

[0132] When people express emotions through their faces, the distance and scale between key facial points will change. For example, the distance between eyebrows is smaller when angry than when surprised, the width of the mouth is smaller when the expression is neutral than when happy, and the width of the mouth is wider when the expression is sad than when the expression is neutral, and so on. Therefore, this embodiment sets different weights for each expression so that the face can be interpreted as different expressions.

[0133] The various parameters in 3DMM are expressed as follows:

[0134] Identity characteristic coefficient:

[0135] Expression coefficient:

[0136] Texture factor:

[0137] Face shape:

[0138] Texture:

[0139] Step 104: Apply the target facial expression coefficients to the target 3DMM to render the target emotional speaking face video.

[0140] Furthermore, step 104 includes:

[0141] The target facial expression coefficient is applied to the target 3DMM, and the target emotional speaking face video is rendered by combining the identity feature coefficient, texture coefficient, face shape parameter and texture parameter.

[0142] It should be noted that by applying the fitted target facial expression coefficients to the target 3DMM, a face with emotions can be generated; this process uses FaceRender in SadTalker to render the generated system into a video based on the 3D perceived image animation, that is, the target emotional speaking face video.

[0143] Specifically, through FaceRender technology, the calculated 3DMM coefficients are applied to the original person to generate a video frame by frame. FaceRender maps the 3DMM coefficients to the actual facial animation through the network model, namely the target 3DMM. This process relies on various key parameters in the 3DMM, including identity feature coefficients, target facial expression coefficients, texture coefficients, face shape parameters and texture parameters. Through the precise control of these coefficients, the detailed dynamic rendering of the target face is achieved. All generated frames are combined into a complete video, and finally the speaking face video, namely the target emotional speaking face video, is output.

[0144] The method for generating emotion-driven speaking faces based on 3DMM provided in the embodiment of the present application uses the distance correlation method and the SIPCA algorithm to perform quantitative correlation analysis on the coefficients generated by 3DMM, so as to determine the influence of the system on the control of eyes and lips; on this basis, a preset LSTM network is used to combine and analyze the emotional facial features with the lip coefficients in the audio, and the target facial expression coefficients of the emotion-driven facial expressions are fitted; then, based on this coefficient, an emotional speaking face video is generated; this process simultaneously considers the emotional characteristics of the facial eyes and the lip movement characteristics in the audio, and can accurately and naturally generate speaking faces with facial expressions, making the generated video more vivid and realistic, and meeting the needs of actual scenarios. Therefore, the embodiment of the present application can solve the technical problem that the prior art cannot clearly define the influence of the 3DMM coefficients on the control of the facial area, and does not pay attention to other facial expression features, resulting in the lack of realism in the generated face video.

[0145] For easier understanding, see Figure 2 The present application provides an embodiment of an emotion-driven speaking face generation device based on 3DMM, including:

[0146] A coefficient extraction unit 201 is used to extract the emotion model expression coefficients of the preset face image and the lip movement model coefficients in the corresponding audio respectively through 3DMM;

[0147] A correlation analysis unit 202 is used to perform eye coefficient change analysis based on a distance correlation method and a SIPCA algorithm according to preset face images and emotion model expression coefficients to obtain correlation emotion model coefficients;

[0148] A coefficient fitting unit 203 is used to generate a target facial expression coefficient by fitting the relevant emotion model coefficient and the lip movement model coefficient using a preset LSTM network;

[0149] The face generation unit 204 is used to apply the target face expression coefficients to the target 3DMM to render the target emotional speaking face video.

[0150] Further, the coefficient extraction unit 201 is specifically used for:

[0151] Extract the expression coefficients in the preset face image through 3DMM to generate the expression coefficients of the emotion model;

[0152] The lip motion model coefficients are generated from the audio corresponding to the preset face image through 3DMM.

[0153] Furthermore, the correlation analysis unit 202 is specifically configured to:

[0154] Extract multiple key points around the eyes in the preset face image, and calculate the vertical height of the key points to obtain the change in eye height;

[0155] Extracting multiple coefficient vectors at different times from the emotion model expression coefficients, and calculating the inter-frame variation of the coefficient vectors to obtain the coefficient variation;

[0156] The distance correlation method is used to calculate the correlation coefficient between the change in eye height and the change in coefficient, and a correlation coefficient sequence is obtained;

[0157] Selecting a preset number of correlation coefficients from the correlation coefficient sequence arranged in descending order to form a target correlation sequence;

[0158] The SIPCA algorithm is used to perform correlation optimization analysis based on the target correlation sequence to obtain the correlation sentiment model coefficient.

[0159] Furthermore, the coefficient fitting unit 203 is specifically used for:

[0160] The preset LSTM network is used to generate the emotion 3DMM expression coefficient according to the correlation emotion model coefficient;

[0161] The emotion 3DMM expression coefficient and the lip motion model coefficient are combined to generate the target facial expression coefficient.

[0162] The present application also provides an emotion-driven speaking face generation device based on 3DMM, the device comprising a processor and a memory;

[0163] The memory is used to store the program code and transmit the program code to the processor;

[0164] The processor is used to execute the emotion-driven speaking face generation method based on 3DMM in the above method embodiment according to the instructions in the program code.

[0165] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0166] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0169] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. The emotion-driven speaking face generation method based on 3DMM is characterized by: include: The emotion model expression coefficients of the preset face image and the lip movement model coefficients in the corresponding audio are extracted through 3DMM; Based on the distance correlation method and the SIPCA algorithm, the eye coefficient change analysis is performed according to the preset face image and the emotion model expression coefficient to obtain the correlation emotion model coefficient; Using a preset LSTM network to fit the relevant emotion model coefficients and the lip movement model coefficients to generate the target facial expression coefficients; The target facial expression coefficient is applied to the target 3DMM to render the target emotional speaking face video.

2. The method for generating emotion-driven speaking faces based on 3DMM according to claim 1, characterized in that: The step of extracting the emotion model expression coefficients of the preset face image and the lip movement model coefficients in the corresponding audio by 3DMM includes: Extract the expression coefficients in the preset face image through 3DMM to generate the expression coefficients of the emotion model; The lip motion model coefficients are generated from the audio corresponding to the preset face image through the 3DMM.

3. The method for generating emotion-driven speaking faces based on 3DMM according to claim 1, characterized in that: The distance correlation method and the SIPCA algorithm are used to analyze eye coefficient changes according to the preset face image and the emotion model expression coefficient to obtain the correlation emotion model coefficient, including: Extracting a plurality of key points around the eyes in the preset face image, and calculating the vertical heights of the key points to obtain the eye height change; Extracting multiple coefficient vectors at different times from the emotion model expression coefficients, and calculating the inter-frame variation of the coefficient vectors to obtain the coefficient variation; Calculating the correlation coefficient between the eye height change and the coefficient change by using a distance correlation method to obtain a correlation coefficient sequence; Selecting a preset number of correlation coefficients from the correlation coefficient sequence arranged in descending order to form a target correlation sequence; The SIPCA algorithm is used to perform correlation optimization analysis according to the target correlation sequence to obtain the correlation emotion model coefficient.

4. The method for generating emotion-driven speaking faces based on 3DMM according to claim 1, characterized in that: The method of using a preset LSTM network to fit and generate a target facial expression coefficient according to the correlation emotion model coefficient and the lip movement model coefficient includes: Using a preset LSTM network to generate emotion 3DMM expression coefficients according to the correlation emotion model coefficients; The emotion 3DMM expression coefficient and the lip movement model coefficient are combined to generate the target facial expression coefficient.

5. The method for generating emotion-driven speaking faces based on 3DMM according to claim 1, characterized in that: The step of applying the target facial expression coefficient to the target 3DMM to render a target emotional speaking face video includes: The target facial expression coefficient is applied to the target 3DMM, and the target emotional speaking face video is rendered by combining the identity feature coefficient, texture coefficient, face shape parameter and texture parameter.

6. Emotion-driven speaking face generation device based on 3DMM, characterized in that: include: A coefficient extraction unit, used for respectively extracting the emotion model expression coefficients of the preset face image and the lip movement model coefficients in the corresponding audio through 3DMM; A correlation analysis unit, configured to perform eye coefficient change analysis based on the preset face image and the emotion model expression coefficient based on a distance correlation method and a SIPCA algorithm, to obtain a correlation emotion model coefficient; A coefficient fitting unit, used for fitting and generating a target facial expression coefficient according to the correlation emotion model coefficient and the lip movement model coefficient by using a preset LSTM network; The face generation unit is used to apply the target facial expression coefficients to the target 3DMM to render the target emotional speaking face video.

7. The 3DMM-based emotion-driven speaking face generation device according to claim 6, characterized in that: The coefficient extraction unit is specifically used for: Extract the expression coefficients in the preset face image through 3DMM to generate the expression coefficients of the emotion model; The lip motion model coefficients are generated from the audio corresponding to the preset face image through the 3DMM.

8. The 3DMM-based emotion-driven speaking face generation device according to claim 6, characterized in that: The related analysis unit is specifically used for: Extracting a plurality of key points around the eyes in the preset face image, and calculating the vertical heights of the key points to obtain the eye height change; Extracting multiple coefficient vectors at different times from the emotion model expression coefficients, and calculating the inter-frame variation of the coefficient vectors to obtain the coefficient variation; Calculating the correlation coefficient between the eye height change and the coefficient change by using a distance correlation method to obtain a correlation coefficient sequence; Selecting a preset number of correlation coefficients from the correlation coefficient sequence arranged in descending order to form a target correlation sequence; The SIPCA algorithm is used to perform correlation optimization analysis according to the target correlation sequence to obtain the correlation emotion model coefficient.

9. The 3DMM-based emotion-driven speaking face generation device according to claim 6, characterized in that: The coefficient fitting unit is specifically used for: Using a preset LSTM network to generate emotion 3DMM expression coefficients according to the correlation emotion model coefficients; The emotion 3DMM expression coefficient and the lip movement model coefficient are combined to generate the target facial expression coefficient.

10. Emotion-driven speaking face generation device based on 3DMM, characterized in that: The device comprises a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the 3DMM-based emotion-driven speaking face generation method according to any one of claims 1 to 5 according to the instructions in the program code.

Images