A method for generating virtual human facial expression animation based on deep neural network

Abstract

The present invention relates to the field of expression reconstruction technology, specifically a method for generating virtual human facial expression animation based on a deep neural network, comprising the following steps: based on a 3D scanning environment, capturing the facial three-dimensional data of a user virtual character model, and adjusting the facial texture and structural information of the three-dimensional model through edge detection and texture mapping to generate facial feature coding. The present invention achieves highly natural and realistic reproduction of facial expressions through accurate capture of the three-dimensional facial data of the virtual character and detailed texture and structural adjustment. Combined with real-time audio analysis, by finely identifying the emotional and intonation features in the voice, it can generate facial expressions that are highly matched with the voice content, thereby improving the expressiveness of the animation and the audience's sense of immersion. Introducing real-time feedback from the audience to adjust the expression makes the generated animation more in line with the audience's expectations and emotional experience, improving the interactivity and personalization of the virtual expression animation, and optimizing the user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of expression reconstruction, and in particular to a method for generating virtual human facial expression animation based on a deep neural network. Background Art

[0002] The field of expression reconstruction technology focuses on capturing and reproducing subtle changes in facial expressions by using image processing methods such as deep learning and machine learning algorithms. This technology involves analyzing facial features from static images or video sequences, extracting key expression feature points, and using the data to reconstruct facial expressions. Reconstructed facial expressions can be used in a variety of applications, including animated film production, virtual reality, augmented reality, and human-computer interaction systems. The core of expression reconstruction technology lies in its ability to create highly realistic and dynamically expressive facial models, allowing virtual characters or robots to express human emotions more naturally.

[0003] Among them, the virtual human facial expression animation generation method based on deep neural network is a technology that uses deep learning technology to create and manipulate the facial expressions of virtual characters. This method trains a deep neural network model to recognize and imitate the dynamic characteristics of human facial expressions, and then generates new facial expression animations. The main uses of this technology include making high-quality animated movies and characters in video games, and providing a more realistic user experience in virtual reality and augmented reality environments.

[0004] Traditional methods are limited in capturing and expressing subtle changes in human emotions. They rely on preset facial expression libraries or simple facial tracking technology, which makes it difficult to achieve subtle synchronization with actual human expressions, resulting in the loss of realism and immersion in user experience in virtual reality and augmented reality applications. For example, in VR interactions, unnatural or delayed responses to facial expressions can make users feel disconnected and unreal, thereby reducing the overall quality of interaction and user satisfaction. The lack of an effective audience feedback integration mechanism makes it difficult for existing technologies to dynamically adjust to the specific needs of different audience groups, limiting their potential for widespread application. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and propose a method for generating virtual human facial expression animation based on a deep neural network.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme: a method for generating virtual human facial expression animation based on a deep neural network, comprising the following steps:

[0007] S1: Based on the 3D scanning environment, the facial 3D data of the user's virtual character model is captured, and the facial texture and structure information of the 3D model is adjusted through edge detection and texture mapping to generate facial feature coding;

[0008] S2: Based on the user's real-time audio input, extract the audio features, decompose the audio features through the Transformer network, identify and extract the voice and tone features in the audio, encode the extracted results, convert them into voice codes and tone codes, and generate matching lip shape and expression parameters to obtain lip shape and expression information;

[0009] S3: Based on the lip shape and expression information, combined with real-time feedback from external audiences, including audience satisfaction information, adjust the lip shape and expression parameters, optimize the naturalness and adaptability of the expression, and obtain expression adaptability parameter encoding;

[0010] S4: Based on the expression adaptive parameter encoding and facial feature encoding, the lip movements and expression synthesis are processed by stacking the causal Transformer network, the facial expressions of the 3D virtual characters are controlled, and the 3D virtual expression animation is generated.

[0011] The present invention is improved in that the method for capturing the three-dimensional facial data is:

[0012] S111: Based on the 3D scanning environment, the 3D data information of the user's face is captured and integrated into a set of 3D data points. The position data of the facial feature points in the 3D space are converted into numerical coordinate information. Each point is represented by a spatial coordinate. express;

[0013] S112: Based on the three-dimensional data point set, by formula:

[0014] ,

[0015] Filter the captured data to obtain three-dimensional facial data;

[0016] in, is the cleaned data set, represents a point in three-dimensional space, P is a set of three-dimensional data points, For point The average distance to the set P, is the standard deviation of the set P, is the preset scaling factor, is the adjustment factor, is the coefficient of variation of the point set P.

[0017] The present invention is improved in that the steps of acquiring the facial feature code are:

[0018] S121: Based on the facial three-dimensional data, adjust the facial three-dimensional data through edge detection and texture mapping, and extract key facial feature points, where the key facial feature points include eye corners, nose tip, and mouth corners;

[0019] S122: Based on the key facial feature points, the formula:

[0020] ,

[0021] Encode each feature point, convert the three-dimensional coordinates of each point into a corresponding numerical code, and integrate the numerical codes of the feature points to obtain the feature code of the face;

[0022] in, is the encoding of feature point i, , and is the three-dimensional coordinate of feature point i, , and Coordinate weight coefficient, is the scaling factor, and i is the index of the feature point.

[0023] The present invention is improved in that the method for extracting audio features is:

[0024] S211: based on the real-time audio input of the user, recording the audio data stream and intercepting the audio clip;

[0025] S212: performing denoising and volume normalization processing based on the audio clip to obtain a preprocessed audio signal;

[0026] S213: Based on the preprocessed audio signal, the audio signal is converted into continuous frames, the frequency domain features of each frame are calculated by fast Fourier transform, and the Mel spectrum is extracted by Mel filter bank, referring to the formula:

[0027] ,

[0028] Calculate the short-time frequency domain features of the audio clip to obtain audio feature information;

[0029] in, represents the extracted Mel spectrum features, is the amplitude of the kth frequency component after fast Fourier transform, N is the number of fast Fourier transform points, is the filter response at frequency k in Mel scale f.

[0030] The present invention is improved in that the method for converting into speech coding and tone coding is:

[0031] S221: Based on the audio feature information, decompose the audio features through Transformer to obtain the voice and tone features of the audio;

[0032] S222: Based on the voice and tone features of the audio, the formula:

[0033] ,

[0034] ,

[0035] Mapping speech features to speech coding vectors, and mapping tone features to tone coding vectors, to obtain speech coding and tone coding;

[0036] in, is the speech coding vector, is the tone encoding vector, and is the adjustment factor, and are the conversion coefficients of voice and tone features, is the value of the i-th speech feature, is the value of the i-th tone feature, p and q are powers, and n is the total number of features, representing the number of features extracted from the audio.

[0037] The present invention is improved in that the steps of obtaining the lip shape and expression information are as follows:

[0038] S231: Based on the speech coding vector and the tone coding vector, apply mapping logic:

[0039] ,

[0040] ,

[0041] Convert the speech coding vector into lip shape parameters, and convert the tone coding vector into expression parameters;

[0042] in, is the lip shape parameter, is the expression parameter, and is the overall adjustment factor, and is the mapping adjustment coefficient, L and E are the mapping functions, is the key parameter controlling the mapping process. and is an additional parameter, and is the speech and tone encoding vector, m is the number of mapping functions, and j is the index of the mapping function currently processed;

[0043] S232: Based on the lip shape parameters and expression parameters, the expression parameters corresponding to the lip shape parameters are integrated, the timestamps of the parameters are unified, and the lip shape and expression information is obtained.

[0044] The present invention is improved in that the step of acquiring the expression adaptability parameter encoding is:

[0045] S311: Collecting real-time feedback from external audiences and extracting audience satisfaction information on the current animated expressions;

[0046] S312: Based on the satisfaction information, statistical analysis is performed to identify the missing points of the naturalness and adaptability of the feedback expression, and the lip shape and expression information are combined through the formula:

[0047] ,

[0048] Adjusting lip shape and expression parameters to obtain adjusted lip shape and expression parameters;

[0049] in, is the adjusted parameter, is the original parameter, is the adjustment coefficient, is the satisfaction score, T is the desired satisfaction target, is the time decay factor, e is the base of the natural logarithm, t is the current time, is the evaluation start time;

[0050] S313: Based on the adjusted lip shape and expression parameters, the speech expression mapping code is updated to obtain the expression adaptability parameter code.

[0051] The present invention is improved in that the steps of obtaining the 3D virtual expression animation are:

[0052] S411: Based on the expression adaptability parameter encoding and facial feature encoding, the stacked causal Transformer network is used to process the encoding, through the formula:

[0053] ,

[0054] Adjust the vertex position of the 3D model and get the animation command of the 3D model:

[0055] in, is the new vertex position, is the original vertex position, is the global adjustment coefficient, is the weight factor, is the i-th feature value in the facial feature encoding, is the i-th adjustment value in the expression adaptability parameter encoding, is the dynamic adjustment factor, n is the total number of features;

[0056] S412: Apply the animation command to the 3D model, control the expression of the 3D virtual character, and generate a 3D virtual expression animation.

[0057] Compared with the prior art, the advantages and positive effects of the present invention are:

[0058] In the present invention, by accurately capturing the three-dimensional data of the virtual character's face and carefully adjusting the texture and structure, a highly natural and realistic reproduction of facial expressions is achieved. Combined with real-time audio analysis, by finely identifying the emotional and intonation features in the voice, facial expressions that highly match the voice content can be generated, thereby improving the expressiveness of the animation and the audience's sense of immersion. By introducing real-time feedback from the audience to adjust the expression, the adaptability and naturalness of the expression are increased, making the generated animation more in line with the audience's expectations and emotional experience, improving the interactivity and personalization of the virtual expression animation, and optimizing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of the method of the present invention;

[0060] Figure 2 A flowchart of facial three-dimensional data capture according to the present invention;

[0061] Figure 3 A flowchart for obtaining facial feature coding for the present invention;

[0062] Figure 4 A flowchart for extracting audio features for the present invention;

[0063] Figure 5 A flowchart for converting the present invention into speech coding and tone coding;

[0064] Figure 6 A flow chart of the present invention for obtaining lip shape and expression information;

[0065] Figure 7 A flow chart for obtaining expression adaptability parameter encoding for the present invention;

[0066] Figure 8 The present invention provides a flow chart for obtaining 3D virtual expression animation. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0068] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating positions or positional relationships, are based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0069] Example:

[0070] See also Figure 1 The present invention provides a technical solution: a method for generating virtual human facial expression animation based on a deep neural network, comprising the following steps:

[0071] S1: Based on the 3D scanning environment, the facial 3D data of the user's virtual character model is captured, and the facial texture and structure information of the 3D model is adjusted through edge detection and texture mapping to generate facial feature coding;

[0072] S2: Based on the user's real-time audio input, extract the audio features, decompose the audio features through the Transformer network, identify and extract the voice and tone features in the audio, encode the extracted results, convert them into voice codes and tone codes, and generate matching lip shape and expression parameters to obtain lip shape and expression information;

[0073] S3: Based on the lip shape and expression information, combined with the real-time feedback from external audiences, including audience satisfaction information, adjust the lip shape and expression parameters, optimize the naturalness and adaptability of the expression, and obtain the expression adaptability parameter encoding;

[0074] S4: Based on expression adaptive parameter encoding and facial feature encoding, the causal Transformer network is stacked to process the synthesis of lip movements and expressions, control the facial expressions of 3D virtual characters, and generate 3D virtual expression animations.

[0075] Facial feature encoding includes three-dimensional coordinate data, texture mapping parameters and facial edge detection results. Lip shape and expression information includes audio feature vectors, dynamic lip shape parameters and expression control variables. Expression adaptability parameter encoding includes expression sensitivity coefficient, expression dynamic range and time offset of expression synchronization. 3D virtual expression animation includes dynamic sequence of expressions and coordination of movements in facial areas.

[0076] See also Figure 2 , the method of capturing facial 3D data is:

[0077] S111: Based on the 3D scanning environment, the 3D data information of the user's face is captured and integrated into a set of 3D data points. The position data of the facial feature points in the 3D space are converted into numerical coordinate information. Each point is represented by a spatial coordinate. express;

[0078] S112: Based on a set of three-dimensional data points, by formula:

[0079] ,

[0080] Filter the captured data to obtain three-dimensional facial data;

[0081] in, is the cleaned data set, represents a point in three-dimensional space, P is a set of three-dimensional data points, For point The average distance to the set P, is the standard deviation of the set P, is a preset scaling factor used to adjust the sensitivity and strictness of data cleaning. To adjust the coefficient, it is used to balance the impact of the variability of the point set on the filtering threshold and improve the flexibility and accuracy of the filtering. is the coefficient of variation of the point set P, which measures the dispersion of the points in the set and is used to adjust the threshold to make it more adaptable to the actual distribution characteristics of data.

[0082] formula:

[0083] ,

[0084] The meaning and acquisition method of parameters:

[0085] 3D point coordinates : During the 3D scanning process, each captured point All have their coordinates in three-dimensional space , data is obtained directly from the output of the 3D scanner;

[0086] Point set P: P represents the set of all original data points obtained from 3D scanning, including points in facial and non-facial areas;

[0087] The average distance from a point to a set :point The average distance to all other points in the point set P is calculated by The sum of the Euclidean distances to each point in P is divided by the total number of points in the point set P;

[0088] Standard Deviation : Standard deviation Describes the degree of discreteness of the point set P. The calculation method is to first find the sum of the squares of the deviations of all points from the coordinate average, then divide it by the number of points, and finally take the square root;

[0089] Sensitivity scaling factor : It is a preset proportionality factor used to set the sensitivity of the filter threshold, usually adjusted according to actual application requirements;

[0090] Adjustment factor : It is an adjustment coefficient used to balance the impact of the variability of the point set on the filtering threshold, helping to adjust the filtering criteria to adapt to different data set characteristics, and is preset by experts;

[0091] Coefficient of variation : Coefficient of variation It is the ratio of the standard deviation of the point set P to the mean value, which is used to measure the dispersion of the points in the set.

[0092] Calculation example:

[0093] Assume that the point set P consists of three points: , , ;

[0094] It is necessary to calculate each point , and ;

[0095] Calculate the coordinate average of the point set P:

[0096] ,

[0097] Compute the Euclidean distance from each point to the mean point:

[0098] ,

[0099] ,

[0100] ,

[0101] Then, calculate the standard deviation of the point set P :

[0102] ,

[0103] Calculate the coefficient of variation :

[0104] ,

[0105] According to the above calculation, assuming and , you can determine whether to keep a point :

[0106] ,

[0107] For example, for the point , the calculation result is:

[0108] ,

[0109] like ,but ,Will Retained in the cleaned dataset.

[0110] See also Figure 3 , the steps to obtain facial feature encoding are:

[0111] S121: Based on the facial 3D data, the facial 3D data is adjusted through edge detection and texture mapping to extract key facial feature points, including the corners of the eyes, the tip of the nose, and the corners of the mouth. The key structures of the face are identified by calculating the distance between the geometric center of each feature point and the adjacent points;

[0112] S122: Based on key facial feature points, by formula:

[0113] ,

[0114] Encode each feature point, convert the three-dimensional coordinates of each point into a corresponding numerical code, and integrate the numerical codes of the feature points to obtain the feature code of the face;

[0115] in, is the encoding of feature point i, , and is the three-dimensional coordinate of feature point i, , and Coordinate weight coefficient, is a scaling factor used to adjust the generated encoding value to adapt to different application ranges. i is the index of the feature point, ranging from 1 to n, where n is the total number of feature points.

[0116] formula: ,

[0117] The meaning and acquisition method of parameters:

[0118] , and : Represents the coordinates of feature points in three-dimensional space. During 3D scanning, the scanner captures the face and outputs coordinate data, where the coordinates of each feature point come directly from the measurement results of the scanner;

[0119] , and : The coefficient is used to adjust the weight of the three-dimensional coordinates in the encoding, which can be optimized based on a specific facial recognition algorithm or manually set according to the variability and importance in the dataset to improve the discrimination and accuracy of the encoding;

[0120] :The scaling factor is used to adjust the scale of the final encoding to ensure that the encoded value falls within the appropriate numerical range. The factor can be set manually according to application requirements, such as requiring a larger value to avoid loss of precision.

[0121] Calculation example:

[0122] Assume that the coordinates of the feature point P are , and set the adjustment coefficient , , and scaling factor ;

[0123] Compute weighted coordinates:

[0124] ,

[0125] Apply a scaling factor: ,

[0126] Encoding of feature point P is 8.6, which will be used to construct the feature encoding sequence for the entire face.

[0127] See also Figure 4 , the method to extract audio features is:

[0128] S211: based on the real-time audio input of the user, recording the audio data stream and intercepting the audio clip;

[0129] S212: performing denoising and volume normalization processing based on the audio clip to obtain a preprocessed audio signal;

[0130] S213: Based on the preprocessed audio signal, the audio signal is converted into continuous frames, the frequency domain features of each frame are calculated by fast Fourier transform, and the Mel spectrum is extracted by Mel filter bank, referring to the formula:

[0131] ,

[0132] Calculate the short-time frequency domain features of the audio clip to obtain audio feature information;

[0133] in, represents the extracted Mel spectrum features, is the amplitude of the kth frequency component after fast Fourier transform, N is the number of fast Fourier transform points, is the filter response at frequency k in Mel scale f.

[0134] formula: ,

[0135] Parameter meaning and acquisition method:

[0136] : is the amplitude of the kth frequency component obtained after the fast Fourier transform. The process of applying the fast Fourier transform to the original audio data to convert the time domain signal into the frequency domain signal can be directly calculated by mathematical algorithms;

[0137] N: represents the number of fast Fourier transform points, that is, the total number of data points included in the fast Fourier transform processing. The value is preset, such as 1024 or 2048 points, depending on the required frequency resolution and time resolution;

[0138] : is the filter response of frequency k at the Mel scale f. The Mel filter is a set of triangular bandpass filters used to convert normal frequencies to Mel frequencies, which is a frequency scale close to human ear perception. The design of the filter is adjusted according to the sensitivity of the human ear to different frequencies. The specific response value of the filter can be calculated using the known Mel filter design formula.

[0139] Calculation example:

[0140] Assume that the number of points for fast Fourier transform processing is N=1024, consider the specific frequency component k=300, and assume that the amplitude calculated by the fast Fourier transform corresponding to the frequency is ,

[0141] Select a Mel frequency f, such as f=20, and the corresponding Mel filter response is assumed to be ,

[0142] Mel spectrum for frequency component k=300 It can be calculated as follows:

[0143] Compute the energy of a frequency component:

[0144] Calculate the energy of the frequency component with k=300 according to the formula ,get .

[0145] Apply a Mel filter:

[0146] Multiplying the calculated energy with the Mel filter response yields .

[0147] Calculate the final value of the Mel spectrum:

[0148] The resulting value is multiplied by 1000 to increase the dynamic range, and the logarithm is calculated to convert to dB units, that is: ,

[0149] Mel spectrum for frequency component k=300 at Mel frequency f=20 .

[0150] See also Figure 5 , the method of converting into speech coding and tone coding is:

[0151] S221: Based on the audio feature information, the audio features are decomposed through Transformer to obtain the voice and tone features of the audio;

[0152] S222: Based on the voice and tone features of the audio, the formula is:

[0153] ,

[0154] ,

[0155] Mapping speech features to speech coding vectors, and mapping tone features to tone coding vectors, to obtain speech coding and tone coding;

[0156] in, is the speech coding vector, is the tone encoding vector, and is an adjustment factor used to improve the dynamic range of the encoding, and are the conversion coefficients of voice and tone features, is the value of the i-th speech feature, is the value of the i-th tone feature, p and q are powers used to enhance the nonlinear expression of features, and n is the total number of features, representing the number of features extracted from the audio.

[0157] formula:

[0158] ,

[0159] ,

[0160] The meaning and acquisition method of parameters:

[0161] and : It is the adjustment coefficient used to adjust the dynamic range of feature weights during the encoding process, which is usually pre-set according to the distribution of feature data or specific application requirements;

[0162] and :The conversion coefficient is determined by the training data and is usually adjusted by an optimization algorithm during the preprocessing or training phase to maximize the model's fit to the training data;

[0163] and :The feature values ​​are extracted from the audio signal through the Transformer network, including speech features and tone features , each feature is a numerical representation of the audio data;

[0164] p and q: The powers are set to enhance the nonlinear expression of features in the encoding and are pre-set according to the importance and characteristics of the features;

[0165] n: The total number of features, which depends on the model design and the complexity of the audio data, and represents the number of features that will be considered during the encoding process.

[0166] Calculation Example

[0167] Assumptions: Adjustment Factor , ,

[0168] Conversion Factor , ,

[0169] Eigenvalue , ,

[0170] Power p=2, q=3,

[0171] The total number of features n=1,

[0172] According to the formula: , ,

[0173] Perform the calculation:

[0174] Calculate speech coding:

[0175] ,

[0176] ,

[0177] Calculate the mood code:

[0178] ,

[0179] ,

[0180] The steps show the process from audio feature extraction to encoding, which takes into account the nonlinear processing of features and adjusts the dynamic range of the results through preset coefficients.

[0181] See also Figure 6 , the steps to obtain lip shape and expression information are:

[0182] S231: Based on the speech coding vector and the tone coding vector, apply mapping logic:

[0183] ,

[0184] ,

[0185] Convert the speech coding vector into lip shape parameters, and convert the tone coding vector into expression parameters;

[0186] in, is the lip shape parameter, is the expression parameter, and is the overall adjustment coefficient, which is used to adjust the overall output strength of lip shape and expression parameter mapping. and is the mapping adjustment coefficient, which is used to optimize the accuracy and dynamic expression of lip shape and expression parameters. L and E are mapping functions that convert speech and tone encoding into parameters of visual expression. It is a key parameter to control the mapping process and is used to adjust the flexibility and adaptability of the mapping. and is an additional parameter, and is the speech and tone encoding vector extracted from the audio data, m is the number of mapping functions, which determines the complexity and detail of the generated lip shape and expression parameters, j is the index of the currently processed mapping function, which identifies the specific mapping function used when accumulating the mapping results to ensure that all functions are properly processed;

[0187] S232: Based on the lip shape parameters and the expression parameters, the expression parameters corresponding to the lip shape parameters are integrated, the timestamps of the parameters are unified, and the lip shape and expression information is obtained.

[0188] Mapping logic:

[0189] ,

[0190] ,

[0191] The meaning and acquisition method of parameters:

[0192] and :The overall adjustment coefficient is used to control the overall influence of the output parameters, which is usually set based on previous test or calibration data to ensure that the output parameters match the expected animation effect;

[0193] and : Mapping adjustment coefficients are used to optimize the accuracy and dynamic expression of lip shape and expression parameters. The coefficients are pre-adjusted based on the needs of specific animations to ensure the naturalness and consistency of expressions;

[0194] L and E: mapping functions that convert voice and tone encoding into parameters of visual expression. The function is usually trained by a machine learning model to map audio features to visual output in the best way;

[0195] : Controls the key parameters of the mapping process, regulating the flexibility and adaptability of the mapping, determined experimentally to ensure that a variety of audio inputs receive appropriate visual responses;

[0196] and : Additional parameters used to fine-tune the details of the mapping results. The parameters are usually pre-adjusted according to the complexity of different expressions to improve the accuracy and realism of the expressions;

[0197] and : The voice and tone encoding vectors extracted from the audio data, obtained in the previous step;

[0198] m: the number of mapping functions, which reflects the complexity of the model and is determined according to the required output accuracy and expressiveness;

[0199] j: Iterates over the index of each mapping function during the accumulation process to ensure that all mapping functions are properly processed.

[0200] Calculation example:

[0201] Assume the following parameters:

[0202] , , , ;

[0203] Assuming that the mapping functions L and E are linear functions,

[0204] , ;

[0205] , , ;

[0206] , , m=1;

[0207] Calculated according to the formula:

[0208] Calculation of lip shape parameters:

[0209] Apply the mapping function L: ,

[0210] Apply adjustment factor : ,

[0211] Applying an overall adjustment factor : ,

[0212] Final lip shape parameters ,

[0213] Calculation of expression parameters:

[0214] Apply the mapping function E: ,

[0215] Apply adjustment factor : ,

[0216] Applying an overall adjustment factor : ,

[0217] Final expression parameters ;

[0218] The calculation example demonstrates the conversion from voice and tone encoding vectors to specific lip shape and expression parameters, reflecting the detailed steps and logic of the entire parameterization process.

[0219] See also Figure 7 , the steps to obtain the expression adaptability parameter encoding are:

[0220] S311: Collecting real-time feedback from external audiences and extracting audience satisfaction information on the current animated expressions;

[0221] S312: Based on the satisfaction information, statistical analysis is performed to identify the missing points of the naturalness and adaptability of the feedback expression, and the lip shape and expression information are combined through the formula:

[0222] ,

[0223] Adjusting lip shape and expression parameters to obtain adjusted lip shape and expression parameters;

[0224] in, is the adjusted parameter, is the original parameter, is the adjustment coefficient, which is used to control the sensitivity of the adjustment. is the satisfaction score, T is the desired satisfaction target, is the time decay factor, e is the base of the natural logarithm, approximately equal to 2.71828, which is used to determine the decay rate of the feedback effect, t is the current time, is the evaluation start time;

[0225] S313: Based on the adjusted lip shape and expression parameters, the speech expression mapping code is updated to obtain the expression adaptability parameter code.

[0226] formula: ,

[0227] The meaning and acquisition method of parameters:

[0228] : The original lip shape and expression parameters, obtained from the original animation or previous steps;

[0229] : Audience satisfaction ratings, collected through audience feedback systems, such as online surveys or real-time feedback devices, provide direct satisfaction evaluations for the current animated expressions;

[0230] T: Desired satisfaction target, which is preset based on the needs of the project or product and may be based on market research or historical data to define the desired satisfaction level;

[0231] : Adjustment coefficient, which is used to determine the influence of satisfaction feedback on the adjustment range of expression. The coefficient is determined through experiments in the initial testing stage to find the optimal response sensitivity;

[0232] : Time decay factor, which defines the rate at which satisfaction feedback decreases over time. It is adjusted according to the actual application scenario of the animation and the duration of the audience's feedback to prevent outdated data from affecting the current expression adjustment;

[0233] e: the base of the natural logarithm, a mathematical constant used to calculate exponential decay;

[0234] t and : t is the current time, which can be the timestamp when the animation is played or feedback is collected. It is the evaluation start time, that is, the starting time point for collecting feedback.

[0235] Calculation example:

[0236] Assume the following parameters: Indicates that the original parameter is 50, The feedback score is 70, which indicates a relatively high satisfaction level. The expected satisfaction level is 60 at T=60. The adjustment coefficient is 0.5, and the adjustment range is moderate. Time decay factor, smaller decay, e=2.71828 natural logarithm base, t=10, Assume that the current time is 10 minutes and the evaluation start time is 0 minutes.

[0237] According to the formula: ,

[0238] Calculate the difference in satisfaction: ,

[0239] Apply a time decay factor: ,

[0240] Calculate adjustments: ,

[0241] The final adjusted parameters are: ,

[0242] The calculations show that the lip shape and expression parameters in the animation can be adjusted according to real-time audience feedback to improve the naturalness and adaptability of the expression and ensure that audience satisfaction is effectively responded to.

[0243] See also Figure 8 ,The steps to obtain 3D virtual expression animation are:

[0244] S411: Based on expression adaptive parameter encoding and facial feature encoding, the stacked causal Transformer network is used to process the encoding, through the formula:

[0245] ,

[0246] Adjust the vertex position of the 3D model and get the animation command of the 3D model:

[0247] in, is the new vertex position, is the original vertex position, It is a global adjustment coefficient that controls the sensitivity of the overall animation. is the weight factor, is the i-th feature value in the facial feature encoding, is the i-th adjustment value in the expression adaptability parameter encoding, is a dynamic adjustment factor, which is adjusted according to real-time feedback to provide a smoother animation effect, and n is the total number of features;

[0248] S412: Applying the animation command to the 3D model, controlling the expression of the 3D virtual character, and generating a 3D virtual expression animation.

[0249] formula: ,

[0250] The meaning and acquisition method of parameters:

[0251] : Original vertex position, i.e. the basic position state of the 3D model before the animation starts; the vertex position is determined by the design of the 3D model and is usually defined when the model is created;

[0252] : Global adjustment coefficient, used to control the sensitivity of the entire animation to parameter changes; the coefficient is adjusted based on animation needs and prediction experiments to ensure that the animation response is moderate, not too drastic or weak;

[0253] : Weight factor, which determines the proportion of each feature's influence on the animation; the weight factor is usually calculated based on the animator's experience or through animation optimization algorithms to balance the expression of each facial feature;

[0254] : Feature value in facial feature encoding, the value describes the specific facial action to be simulated, obtained through the previous step;

[0255] : The adjustment value in the expression adaptability parameter encoding is used to adjust the feature value to better adapt to the audience feedback. The adjustment value is dynamically adjusted based on user feedback and animation test results;

[0256] : Dynamically adjust factors to optimize the smoothness and naturalness of animation based on real-time feedback. Factors are usually calculated through a real-time rendering feedback loop to adjust animation details in real time;

[0257] n: The total number of features, which indicates the dimension of the feature vector involved in the calculation. This is determined based on the complexity and precision requirements of the animation.

[0258] Calculation example:

[0259] Assume the following parameters for animation calculations:

[0260] The initial vertex position is 0. Choose an appropriate global adjustment factor, Assuming that we are focusing on the movements around the eyes, we give them a higher weight. Facial feature encoding values, indicating that the eyes need to move upward to form an expression, Adjust the expression adaptability parameter value to enhance the amplitude of eye expression. Dynamically adjust the factor to ensure the natural and smooth animation in the current environment. In this example, n=1, only one feature is considered for calculation.

[0261] Calculate according to the formula:

[0262] ,

[0263] ,

[0264] ,

[0265] New vertex position , which indicates that during the animation, the position of the eyes is slightly adjusted upward relative to the original position to produce the desired expression animation.

[0266] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for generating virtual human facial expression animation based on deep neural network, characterized in that: The following steps are involved: Based on the 3D scanning environment, the facial 3D data of the user's virtual character model is captured, and the facial texture and structure information of the 3D model are adjusted through edge detection and texture mapping to generate facial feature coding; Based on the user's real-time audio input, extract the audio features, decompose the audio features through the Transformer network, identify and extract the voice and tone features in the audio, encode the extracted results, convert them into voice codes and tone codes, and generate matching lip shape and expression parameters to obtain lip shape and expression information; The method for extracting audio features is: Based on the user's real-time audio input, record the audio data stream and capture the audio clips; Based on the audio clip, perform denoising and volume normalization processing to obtain a preprocessed audio signal; Based on the preprocessed audio signal, the audio signal is converted into continuous frames, the frequency domain features of each frame are calculated using fast Fourier transform, and the Mel spectrum is extracted through the Mel filter bank, referring to the formula: ， Calculate the short-time frequency domain features of the audio clip to obtain audio feature information; in, represents the extracted Mel spectrum features, is the amplitude of the kth frequency component after fast Fourier transform, N is the number of fast Fourier transform points, is the filter response at frequency k at Mel scale f; The method for converting into speech coding and tone coding is: Based on the audio feature information, the audio features are decomposed through Transformer to obtain the voice and tone features of the audio; Based on the voice and tone characteristics of the audio, the formula is: ， ， Mapping speech features to speech coding vectors, and mapping tone features to tone coding vectors, to obtain speech coding and tone coding; in, is the speech coding vector, is the tone encoding vector, and is the adjustment factor, and are the conversion coefficients of voice and tone features, is the value of the i-th speech feature, is the value of the i-th tone feature, p and q are powers, and n is the total number of features, representing the number of features extracted from the audio; The steps for obtaining the lip shape and expression information are as follows: Based on the speech encoding vector and the tone encoding vector, a mapping logic is applied: ， ， Convert the speech coding vector into lip shape parameters, and convert the tone coding vector into expression parameters; in, is the lip shape parameter, is the expression parameter, and is the overall adjustment factor, and is the mapping adjustment coefficient, L and E are the mapping functions, is the key parameter controlling the mapping process. and is an additional parameter, and is the speech and tone encoding vector, m is the number of mapping functions, and j is the index of the mapping function currently processed; Based on the lip shape parameters and the expression parameters, the expression parameters corresponding to the lip shape parameters are integrated, the timestamps of the parameters are unified, and the lip shape and expression information is obtained; Based on the lip shape and expression information, combined with real-time feedback from external audiences, including audience satisfaction information, the lip shape and expression parameters are adjusted to optimize the naturalness and adaptability of the expression, and the expression adaptability parameter encoding is obtained; Based on the expression adaptive parameter coding and facial feature coding, the synthesis of lip movements and expressions is processed by stacking causal Transformer networks, the facial expressions of 3D virtual characters are controlled, and 3D virtual expression animations are generated.

2. The method for generating virtual human facial expression animation based on deep neural network according to claim 1, characterized in that: The method for capturing the three-dimensional facial data is: Based on the 3D scanning environment, the 3D data information of the user's face is captured and integrated into a set of 3D data points. The position data of facial feature points in the 3D space is converted into numerical coordinate information. Each point is represented by a spatial coordinate. express; Based on the three-dimensional data point set, by formula: ， Filter the captured data to obtain three-dimensional facial data; in, is the cleaned data set, represents a point in three-dimensional space, P is a set of three-dimensional data points, For point The average distance to the set P, is the standard deviation of the set P, is the preset scaling factor, is the adjustment factor, is the coefficient of variation of the point set P.

3. The method for generating virtual human facial expression animation based on deep neural network according to claim 2, characterized in that: The steps of obtaining the facial feature code are: Based on the three-dimensional facial data, the three-dimensional facial data is adjusted through edge detection and texture mapping to extract key facial feature points, where the key facial feature points include the corners of the eyes, the tip of the nose, and the corners of the mouth; Based on the key facial feature points, by formula: ， Encode each feature point, convert the three-dimensional coordinates of each point into a corresponding numerical code, and integrate the numerical codes of the feature points to obtain the feature code of the face; in, is the encoding of feature point i, , and is the three-dimensional coordinate of feature point i, , and Coordinate weight coefficient, is the scaling factor, and i is the index of the feature point.

4. The method for generating virtual human facial expression animation based on deep neural network according to claim 1, characterized in that: The steps of obtaining the expression adaptability parameter encoding are: Collect real-time feedback from external audiences and extract information about their satisfaction with the current animated expressions; Based on the satisfaction information, statistical analysis is performed to identify the missing points of the naturalness and adaptability of the feedback expression, and the lip shape and expression information are combined through the formula: ， Adjusting lip shape and expression parameters to obtain adjusted lip shape and expression parameters; in, is the adjusted parameter, is the original parameter, is the adjustment coefficient, is the satisfaction score, T is the desired satisfaction target, is the time decay factor, e is the base of the natural logarithm, t is the current time, is the evaluation start time; Based on the adjusted lip shape and expression parameters, the speech expression mapping code is updated to obtain the expression adaptability parameter code.

5. The method for generating virtual human facial expression animation based on deep neural network according to claim 1, characterized in that: The steps of obtaining the 3D virtual expression animation are as follows: Based on the expression adaptation parameter encoding and facial feature encoding, the stacked causal Transformer network is used to process the encoding, through the formula: ， Adjust the vertex position of the 3D model and get the animation command of the 3D model: in, is the new vertex position, is the original vertex position, is the global adjustment coefficient, is the weight factor, is the i-th feature value in the facial feature encoding, is the i-th adjustment value in the expression adaptability parameter encoding, is the dynamic adjustment factor, n is the total number of features; The animation command is applied to the 3D model to control the expression of the 3D virtual character and generate a 3D virtual expression animation.

Images