Methods and systems for generating facial expressions for film and animation characters using motion capture

Abstract

This invention relates to the field of film and animation production technology, specifically providing a method and system for generating facial expressions for film and animation characters using motion capture. First, a set of real-time motion data of the target actor is collected using motion capture equipment, including subsets of facial and body motion data. The facial motion data consists of facial key point displacement data with multiple consecutive timestamps. Next, a preset expression mapping model is invoked to decouple the facial motion data, obtaining a set of basic and personalized expression features. Then, the basic expression feature set is adjusted according to preset emotional parameters of the animated character to generate an emotion-adaptive expression feature set, which is then fused with the personalized expression feature set to obtain the target expression feature set. Finally, the target expression feature set is used to drive a 3D facial model to generate facial animation, while simultaneously binding body motion data to the 3D body model to generate a complete character animation, thus improving the realism and expressiveness of the animated character's expressions.

Description

Technical Field

[0001] This invention relates to the field of film and animation production technology, and more specifically, to a method and system for generating facial expressions for film and animation characters by combining motion capture. Background Technology

[0002] In the field of film and animation production, character facial expression generation has always been a crucial and challenging aspect. Traditional methods for generating facial expressions in film and animation primarily rely on animators manually drawing keyframes and adjusting them frame by frame to achieve changes in the character's expression. This method not only requires animators to possess extremely high drawing skills and rich experience, but the production process is also extremely time-consuming, labor-intensive, and inefficient, making it difficult to meet the demands of large-scale, fast-paced film and television production.

[0003] Some existing methods have begun to incorporate motion capture technology to assist in character expression generation. However, most existing motion capture-based expression generation technologies can only collect overall facial movement data, failing to perform fine-grained differentiation and processing of the movement data, and unable to deeply explore the rich expressive information contained behind facial movements. Furthermore, existing technologies often do not fully consider the differences in character type and preset emotional parameters when generating expressions, resulting in expressions that lack specificity and realism, making it difficult for animated characters to express unique personalities and subtle emotional changes.

[0004] Furthermore, existing methods typically focus solely on generating facial expressions, neglecting the coordination between body movements and facial expressions. This results in a disconnect between facial expressions and body movements in the generated character animation, failing to present a coherent and natural overall effect. These limitations of existing technologies make it difficult to effectively improve the quality and efficiency of facial expression generation for film and television animation characters. Summary of the Invention

[0005] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, the present invention provides a method for generating facial expressions for film and television animation characters by combining motion capture, the method comprising:

[0006] The real-time motion data set of the target actor is collected by motion capture equipment. The real-time motion data set includes a facial motion data subset and a body motion data subset. The facial motion data subset consists of facial key point displacement data with multiple consecutive timestamps.

[0007] The facial motion data subset is decoupled by calling a preset expression mapping model to obtain a basic expression feature set and a personalized expression feature set. The basic expression feature set contains general expression parameters that are independent of the character type, and the personalized expression feature set contains differentiated expression parameters that are bound to the character type.

[0008] The basic expression feature set is dynamically adjusted based on the preset emotional parameters of the animated character to generate an emotion-adaptive expression feature set.

[0009] The set of emotion-adapted facial expression features is fused with the set of personalized facial expression features to obtain the target facial expression feature set;

[0010] The target facial feature set drives the 3D facial model of the animated character to generate continuous facial animation, and the body motion data subset is synchronously bound to the 3D body model to generate complete character animation.

[0011] In one possible implementation of the first aspect, the acquisition of a set of real-time motion data of the target actor via motion capture equipment includes:

[0012] A first array of markers is placed on the face of the target actor, covering the movement area of ​​the facial muscle groups, and each marker corresponds to the synchronous monitoring of at least three optical capture sensors.

[0013] The optical capture sensor captures the three-dimensional spatial coordinate change data of the first marker point array at a preset sampling frequency to generate the facial motion data subset;

[0014] A second array of markers is placed at the limb joints of the target actor. The second array of markers contains at least twenty markers to cover the rotation angle monitoring area of ​​the key movement joints.

[0015] The acceleration and angular velocity data of the second marker array are collected in real time by the inertial measurement unit, and the body motion data subset is generated by combining the inverse kinematics algorithm.

[0016] The facial motion data subset and the body motion data subset are time-stamp aligned to ensure that the facial key point displacement data and limb joint rotation data at the same timestamp are time-synchronized.

[0017] In one possible implementation of the first aspect, the step of invoking a preset expression mapping model to perform feature decoupling processing on the facial action data subset to obtain a basic expression feature set and a personalized expression feature set includes:

[0018] The subset of facial motion data is input into a trained feature decoupling neural network, which includes a shared encoder layer, a basic feature decoder, and a personalized feature decoder.

[0019] The shared encoder layer extracts spatiotemporal features from the time series of facial motion data to obtain a hybrid expression feature vector.

[0020] The hybrid facial expression feature vector is input into the basic feature decoder and the personalized feature decoder, respectively. The basic feature decoder outputs general facial expression parameters that are independent of the character type, and the personalized feature decoder outputs differentiated facial expression parameters that are related to the character type.

[0021] The general expression parameters and the differential expression parameters are subjected to orthogonal constraint processing so that the projection directions of the general expression parameters and the differential expression parameters in the feature space remain orthogonal.

[0022] The differentiated expression parameters are dynamically weighted based on preset role type identifiers to generate the personalized expression feature set, wherein different role type identifiers correspond to different combinations of weighting coefficients.

[0023] In one possible implementation of the first aspect, the step of dynamically adjusting the basic expression feature set according to preset emotional parameters of the animated character to generate an emotion-adapted expression feature set includes:

[0024] Obtain the preset emotional parameters of the current scene in the animation script, wherein the preset emotional parameters include emotional intensity values ​​and emotional category identifiers;

[0025] According to the emotion category identifier, the corresponding expression adjustment strategy is matched from the preset emotion rule base. The expression adjustment strategy includes the eye area weight coefficient, the mouth area amplitude threshold and the eyebrow area phase offset.

[0026] Based on the emotional intensity value, the parameters in the expression adjustment strategy are dynamically scaled to generate a set of dynamic adjustment parameters that are adapted to the current emotional intensity.

[0027] The set of dynamically adjusted parameters is applied to the corresponding regional feature data in the set of basic facial expression features, wherein the weight coefficient is adjusted for the eye region features, the amplitude threshold is truncated for the mouth region features, and a phase offset is added to the eyebrow region features.

[0028] The adjusted feature data of each region are processed to ensure that the rate of change of facial features between adjacent timestamps does not exceed the preset visual coherence threshold, thereby generating an emotion-adapted facial feature set.

[0029] In one possible implementation of the first aspect, the step of fusing the emotion-adapted expression feature set with the personalized expression feature set to obtain the target expression feature set includes:

[0030] The general expression parameters in the emotion-adaptive expression feature set are normalized to align their numerical range with the parameter range in the personalized expression feature set.

[0031] The fusion weight coefficient is obtained from the preset fusion rule base based on the character type identifier. The fusion weight coefficient includes the basic expression weight and the personalized expression weight.

[0032] The general expression parameters and the differential expression parameters are linearly superimposed according to the corresponding fusion weight coefficients to generate a preliminary fusion feature vector;

[0033] The preliminary fused feature vector is subjected to nonlinear feature enhancement processing to obtain an enhanced feature vector. The feature enhancement processing includes nonlinear mapping of the feature values ​​of the key expression region through the hyperbolic tangent function.

[0034] The enhanced feature vectors are subjected to temporal consistency checks to ensure that feature changes between adjacent frames meet the preset animation smoothness standards.

[0035] In one possible implementation of the first aspect, generating continuous facial animation based on the target facial feature set driving the 3D facial model of the animated character includes:

[0036] The expression parameters in the target expression feature set are mapped to the hybrid shape weight space of the three-dimensional facial model to generate an initial expression-driven weight set;

[0037] The physical rationality of the initial expression-driven weight set is verified based on the facial anatomical constraints of the dynamic partition. The facial anatomical constraints include the threshold of muscle contraction direction and the safe range of skin deformation in each region, and the constraint intensity is dynamically adjusted based on the expression type.

[0038] The driving weights that do not meet the facial anatomical constraints are iteratively adjusted until all driving weights meet the preset biomechanical feasibility threshold. The adjusted driving weights are then input into the real-time rendering engine to drive the vertex displacement of the 3D facial model to generate continuous facial expression animation.

[0039] The generated continuous facial expression animation is subjected to visual quality evaluation. When a region of facial distortion is detected in the continuous facial expression animation, a local weight fine-tuning process is automatically triggered to optimize the performance of the distorted region.

[0040] In one possible implementation of the first aspect, the iterative adjustment of the driving weights that do not conform to the facial anatomical constraints includes:

[0041] A musculoskeletal simulation model of a three-dimensional facial model is established, the musculoskeletal simulation model including skin elasticity parameters, muscle traction force parameters and bone motion constraint parameters;

[0042] The initial facial expression driving weights are converted into corresponding muscle activation intensity parameters and input into the musculoskeletal simulation model for physical simulation calculations.

[0043] Detecting areas in physical simulation calculations that violate preset biomechanical rules, including areas of excessive skin stretching and areas of conflicting muscle contractions;

[0044] For each violation area, a corresponding constraint adjustment amount is generated, which includes the maximum stretching compensation coefficient and the muscle co-activation coefficient;

[0045] The constraint adjustment amount is fed back to the driving weight adjustment module, which attenuates the driving weight of the violation area and redistributes it to the adjacent compliant area.

[0046] In one possible implementation of the first aspect, the step of performing a visual quality assessment on the generated continuous facial expression animation, and automatically triggering a local weight fine-tuning process to optimize the performance of the distorted areas when a region of facial distortion is detected in the continuous facial expression animation, includes:

[0047] Extract a reference expression sequence corresponding to the current emotion category from a preset standard expression library. The reference expression sequence contains standard expression keyframes that have been reviewed and approved by experts.

[0048] The generated continuous facial expression animation is spatiotemporally aligned with the reference facial expression sequence so that the comparison has the same time reference and spatial coordinate system;

[0049] Calculate the feature difference between the spatiotemporally aligned continuous facial animation and each corresponding frame in the reference facial animation sequence. The feature difference includes key point position deviation, motion trajectory similarity, and facial expression intensity difference.

[0050] When the feature difference of N consecutive frames exceeds the preset visual tolerance threshold, it is determined to be a region of facial expression distortion, and the facial anatomical regions corresponding to the distorted regions are identified, including the eyelid region, the corner of the mouth region, and the nasal alar region.

[0051] Load the corresponding fine-tuning strategy template according to the partition type. The fine-tuning strategy template includes partition boundary constraints and adjacent region influence coefficients.

[0052] While keeping the driving weights of the non-distorted regions unchanged, the driving weights of the distorted regions are subjected to constrained optimization calculations. The objective function of the constrained optimization calculations includes a visual smoothness term and a biomechanical constraint term.

[0053] The optimized driving weights are progressively blended with the original driving weights to generate a naturally transitioning facial expression correction animation.

[0054] A second quality assessment was performed on the corrected facial animation until the feature difference in all regions was below the visual tolerance threshold.

[0055] In one possible implementation of the first aspect, the step of synchronously binding the subset of body motion data to a 3D body model to generate a complete character animation includes:

[0056] Establish a time synchronization relationship between facial expression animation and body motion data, wherein the time synchronization relationship includes key event alignment points and motion response delay parameters;

[0057] Based on the skeleton-driven character model topology, body-expression association parameters matching the character type are obtained from a preset binding rule library. The body-expression association parameters include the dynamic adjustment coefficient of facial skin weight when the head rotates and the micro-expression intensity mapping table triggered by limb movements. Different model topologies are adapted through the skin binding weight adaptive algorithm.

[0058] The body motion data in the aforementioned subset of body motion data is input into the physics engine for kinematic calculation, generating skeletal drive data that conforms to the character's body shape characteristics;

[0059] The facial expression features are dynamically adjusted based on the body-expression correlation parameters to simulate the physical effects of body movement on facial expressions.

[0060] The adjusted facial expression features and skeletal driving data are synthesized using rendering channels to output a complete character animation sequence with physical realism.

[0061] In another aspect, the present invention also provides a system for generating facial expressions of film and animation characters in conjunction with motion capture, including a processor and a machine-readable storage medium connected to the processor. The machine-readable storage medium is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the machine-readable storage medium to implement the above-mentioned method.

[0062] Based on the above, this invention collects real-time motion datasets of target actors, merging and subdividing them into facial and body motion data subsets. It then uses a preset expression mapping model for feature decoupling, splitting facial motion data into a basic expression feature set and a personalized expression feature set. This effectively separates general and differentiated expression parameters, reflecting not only the universality of expressions but also highlighting the uniqueness of different character types. The basic expression feature set is dynamically adjusted according to the preset emotional parameters of the animated character, generating an emotion-adaptive expression feature set. This ensures that the generated expressions accurately match the character's emotional needs, enhancing the emotional expressiveness and realism of the expressions. The emotion-adaptive expression feature set is then fused with the personalized expression feature set to obtain the target expression feature set, further integrating the emotional attributes and character characteristics of the expressions. Finally, the target expression feature set drives the 3D facial model of the animated character to generate continuous facial animation, and the body motion data subset is synchronously bound to the 3D body model to generate a complete character animation. This achieves synchronized presentation of facial expressions and body movements, making the generated character animation more coherent, vivid, and natural. While improving animation production efficiency, this significantly enhances the realism and expressiveness of the animated characters. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the execution flow of the method for generating facial expressions of film and animation characters by combining motion capture, provided in an embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram of exemplary hardware and software components of the film and animation character facial expression generation system that combines motion capture, provided in an embodiment of the present invention. Detailed Implementation

[0065] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating a method for generating facial expressions for film and animation characters using motion capture, provided in one embodiment of the present invention. The following is a detailed description of this method for generating facial expressions for film and animation characters using motion capture.

[0066] Step S110: Collect a set of real-time motion data of the target actor using motion capture equipment. The set of real-time motion data includes a subset of facial motion data and a subset of body motion data. The subset of facial motion data consists of facial key point displacement data with multiple consecutive timestamps.

[0067] In the process of generating facial expressions for film and animation characters, the real-time motion data set of the target actor is crucial for the subsequent generation of realistic animated character expressions and movements. In this embodiment, a professional motion capture device can be used to accurately record the movement information of various parts of the actor's body. Different types of motion capture devices are suitable for capturing the movements of different body parts.

[0068] The real-time motion dataset consists of two key subsets: facial motion data and body motion data. The facial motion data subset focuses on recording subtle facial movements, comprised of facial keypoint displacement data at multiple consecutive timestamps. This data reflects the morphological changes of the actor's face at different times, such as the changes in keypoint positions caused by the contraction and relaxation of facial muscles. This data can accurately capture the details of the actor's expressions, such as the displacement of facial keypoints corresponding to frowning, smiling, and surprise. The body motion data subset records the movements of other parts of the actor's body, such as limb movements and changes in body posture.

[0069] Step S111: Arrange a first marker array on the face of the target actor. The first marker array covers the movement area of ​​the facial muscle group, and each marker corresponds to the synchronous monitoring of at least three optical capture sensors.

[0070] To accurately capture a subset of facial motion data, a first array of markers needs to be placed on the target actor's face. The areas of facial muscle movement are key to the generation of facial expressions; therefore, the first array of markers needs to fully cover these areas, including but not limited to the forehead, around the eye sockets, cheeks, and around the lips. By placing markers in these areas, the positional changes caused by facial muscle movements can be captured more precisely.

[0071] Each marker point corresponds to simultaneous monitoring by at least three optical capture sensors. An optical capture sensor is a device capable of detecting the position of a marker point. Simultaneous monitoring by multiple sensors improves the accuracy and reliability of marker point position detection. When an actor makes a facial expression, the marker point shifts with the movement of facial muscles, and the optical capture sensors record this displacement information in real time. Since a single sensor may have measurement errors, using at least three sensors for simultaneous monitoring can improve measurement accuracy through methods such as triangulation. For example, the three sensors can monitor the marker point from different angles. By calculating the position of the marker point in different sensor coordinate systems and then fusing the data, a more accurate three-dimensional position information of the marker point can be obtained.

[0072] Step S112: Capture the three-dimensional spatial coordinate change data of the first marker point array at a preset sampling frequency using the optical capture sensor to generate the facial motion data subset.

[0073] After the first array of marker points is set up and the optical capture sensor is configured, the optical capture sensor captures the three-dimensional spatial coordinate change data of the first array of marker points at a preset sampling frequency. The preset sampling frequency is set according to actual needs and equipment performance, and it determines the number of times data is collected per unit time. A higher sampling frequency can record the movement trajectory of the marker points more accurately, but it will also generate more data; a lower sampling frequency may lose some detailed information.

[0074] At each sampling moment, the optical capture sensor measures the coordinates of the marker points in three-dimensional space. As the actor's facial expressions change, the coordinates of the marker points also change accordingly. By continuously recording these coordinate changes, a subset of facial motion data can be generated. For example, over a period of time, the optical capture sensor continuously records the three-dimensional coordinates of the marker points, forming a dataset containing multiple timestamps and corresponding coordinate values. This dataset reflects the changes in the actor's facial movements during that time period.

[0075] Step S113: Arrange a second array of markers at the limb joints of the target actor. The second array of markers contains at least twenty markers to cover the rotation angle monitoring area of ​​the key motion joints.

[0076] To collect a subset of body movement data, a second array of markers needs to be placed at the limb joints of the target actor. Limb joints are key parts of body movement; monitoring the movement of these joints allows for accurate recording of the actor's body movements. The second array of markers needs to contain at least twenty markers, which will be placed in the rotation angle monitoring areas of the key movement joints. Key movement joints include the shoulder, elbow, wrist, hip, knee, and ankle joints. The movement of each joint affects the overall posture and movement of the body; therefore, markers need to be placed around these joints to accurately monitor their rotation angles.

[0077] For example, placing markers at the shoulder joint can monitor movements such as raising, lowering, and rotating the arm; placing markers at the knee joint can monitor movements such as bending and extending the leg. By properly placing markers, the rotation angle monitoring area of ​​key joints can be fully covered, thereby accurately recording the movement of various parts of the actor's body.

[0078] Step S114: Acceleration and angular velocity data of the second marker array are collected in real time by the inertial measurement unit, and the body motion data subset is generated by combining the inverse kinematics algorithm.

[0079] After the second array of marker points is set up, an inertial measurement unit (IMU) is used to collect the acceleration and angular velocity data of the second array of marker points in real time. An IMU is a device that can measure the acceleration and angular velocity of an object; it typically consists of an accelerometer and a gyroscope. An accelerometer measures the acceleration of an object along three coordinate axes, reflecting changes in the object's velocity; a gyroscope measures the angular velocity of an object around three coordinate axes, reflecting the object's rotation.

[0080] Acceleration and angular velocity data collected in real time by the inertial measurement unit (IMU) can provide dynamic information about the second marker array during its movement. However, this data alone cannot directly represent the actor's body movements and requires processing using inverse kinematics (IK) algorithms. IK algorithms calculate joint angles based on the position and orientation of end effectors (such as hands and feet). In this embodiment, by combining the acceleration and angular velocity data of the second marker array with IK algorithms, the rotation angles and position information of each joint can be calculated, thereby generating a subset of body movement data. For example, by analyzing the acceleration and angular velocity data of the hand markers and combining them with IK algorithms, the rotation angles of the shoulder, elbow, and wrist joints can be calculated, thus determining the arm's posture.

[0081] Step S115: Perform timestamp alignment processing on the facial motion data subset and the body motion data subset to ensure that the facial key point displacement data and limb joint rotation data under the same timestamp are time-synchronized.

[0082] The collected facial and body motion data subsets may have inconsistent timestamps due to different acquisition devices and frequencies. To ensure accurate matching of facial expressions and body movements in subsequent processing, these two subsets need to be timestamped.

[0083] The purpose of timestamp alignment is to ensure that facial keypoint displacement data and limb joint rotation data at the same timestamp are synchronized. For example, at a specific point in time, an actor may simultaneously make a facial expression and a body movement. Timestamp alignment ensures that the facial keypoint displacement data and limb joint rotation data at that time point are accurately correlated. This can be achieved through methods such as interpolation and synchronous sampling. Interpolation methods can insert new data points on the timeline based on existing data points, aligning the timestamps of the two subsets; synchronous sampling methods can adjust the sampling frequency of the acquisition devices to ensure that they acquire data at the same time points.

[0084] Step S120: Call the preset expression mapping model to perform feature decoupling processing on the facial action data subset to obtain a basic expression feature set and a personalized expression feature set. The basic expression feature set contains general expression parameters that are independent of the character type, and the personalized expression feature set contains differentiated expression parameters that are bound to the character type.

[0085] After collecting the real-time motion data set of the target actor, the next step is to further process the collected facial motion data subset. This involves invoking a pre-defined expression mapping model, whose main function is to decouple features from the facial motion data subset. The purpose of feature decoupling is to separate the different types of expression features contained in the facial motion data, resulting in a basic expression feature set and a personalized expression feature set.

[0086] The basic facial expression feature set contains generic facial expression parameters that are independent of character type. These generic parameters are basic and universally present features of human expressions, such as a raised corner of the mouth when happy and a furrowed brow when sad. These basic facial expression features are similar regardless of the character type. The personalized facial expression feature set, on the other hand, contains differentiated facial expression parameters that are tied to the character type. Different character types may have different facial expression characteristics; for example, anime characters may have more exaggerated expressions, while realistic characters' expressions are closer to those of real humans. These differentiated facial expression parameters can make the expressions of animated characters more consistent with their character settings.

[0087] Step S121: Input the facial motion data subset into the trained feature decoupling neural network, which includes a shared encoder layer, a basic feature decoder, and a personalized feature decoder.

[0088] To decouple features from a subset of facial motion data, a trained feature decoupling neural network is used. A feature decoupling neural network is a neural network model specifically designed to separate different types of features. It mainly consists of a shared encoder layer, a basic feature decoder, and personalized feature decoders.

[0089] The shared encoder layer is the first module of the feature decoupling neural network. Its function is to extract spatiotemporal features from the time series of facial motion data. Facial motion data is a time-varying sequence of data, containing the displacement information of facial key points at different times. The shared encoder layer processes this time series, extracting its spatiotemporal features to obtain a hybrid expression feature vector. This hybrid expression feature vector contains all the expression feature information from the facial motion data, but it has not yet been separated.

[0090] The basic feature decoder and the personalized feature decoder are used to decode basic facial expression features and personalized facial expression features from the mixed facial expression feature vector, respectively. The basic feature decoder outputs general facial expression parameters that are independent of the character type based on the mixed facial expression feature vector, while the personalized feature decoder outputs differentiated facial expression parameters that are related to the character type.

[0091] Step S122: Extract spatiotemporal features from the time series of facial motion data through the shared encoder layer to obtain a hybrid expression feature vector.

[0092] After receiving a subset of facial motion data, the shared encoder layer extracts spatiotemporal features from the time series of facial motion data. The time series of facial motion data is a multidimensional data sequence, where each time point corresponds to a set of displacement data for facial key points. The shared encoder layer processes this time series to extract its spatiotemporal features.

[0093] The spatiotemporal feature extraction process is achieved through a series of neural network layers, such as convolutional layers and recurrent layers. Convolutional layers can extract spatial features from facial motion data, such as the relative positions of different facial parts; recurrent layers can process time-series data, capturing the dynamic changes in facial movements. Through the combination of these layers, the shared encoder layer can transform the time-series facial motion data into a hybrid expression feature vector. This hybrid expression feature vector is a high-dimensional vector that contains all the expression feature information from the facial motion data, serving as the basis for subsequent separation of basic expression features and personalized expression features.

[0094] Step S123: Input the hybrid facial expression feature vector into the basic feature decoder and the personalized feature decoder respectively, wherein the basic feature decoder outputs general facial expression parameters that are independent of the character type, and the personalized feature decoder outputs differentiated facial expression parameters that are related to the character type.

[0095] After obtaining the hybrid facial expression feature vector, it is input into the basic feature decoder and the personalized feature decoder respectively. The task of the basic feature decoder is to extract universal facial expression parameters from the hybrid facial expression feature vector, which are independent of the character type. These universal facial expression parameters reflect the characteristics of basic human expressions, such as the shape and position changes of eyebrows, eyes, mouth, etc. under different expressions. The basic feature decoder decodes the hybrid facial expression feature vector through a series of neural network layers, extracting the universal facial expression feature information and converting it into universal facial expression parameters.

[0096] The personalized feature decoder focuses on extracting differentiated facial expression parameters related to character type. Different character types may have different facial expression styles and characteristics; for example, cartoon characters' expressions may be more exaggerated, while realistic characters' expressions are closer to those of real humans. Based on the character type information, the personalized feature decoder mines facial expression features related to that character type from the mixed facial expression feature vector and converts them into differentiated facial expression parameters.

[0097] Step S124: Perform orthogonal constraint processing on the general expression parameters and the differential expression parameters so that the projection directions of the general expression parameters and the differential expression parameters in the feature space remain orthogonal.

[0098] To ensure the independence between basic facial expression features and personalized facial expression features, orthogonalization constraints need to be applied to the general and differentiated facial expression parameters. In the feature space, the general and differentiated facial expression parameters can be viewed as vectors in different directions. The purpose of the orthogonalization constraint is to ensure that the projection directions of these two vectors in the feature space remain orthogonal, i.e., the angle between them is 90 degrees.

[0099] Orthogonalization constraints can be implemented using optimization algorithms, such as the Gram-Schmidt orthogonalization algorithm. During orthogonalization, general and differentiated expression parameters are adjusted so that their projection directions in the feature space are mutually perpendicular. This avoids interference between basic and personalized expression features, improving the accuracy of subsequent expression generation.

[0100] Step S125: Dynamically weight the differentiated expression parameters based on the preset role type identifier to generate the personalized expression feature set, wherein different role type identifiers correspond to different combinations of weighting coefficients.

[0101] After obtaining the differentiated expression parameters, they need to be dynamically weighted based on preset character type identifiers to generate a personalized expression feature set. The preset character type identifiers are codes or tags used to identify different character types, such as anime characters, realistic characters, and science fiction characters. Different character types may require different degrees of personalized expression features; therefore, the differentiated expression parameters need to be dynamically weighted according to the character type identifiers.

[0102] Each character type identifier corresponds to a different combination of weighting coefficients. These weighting coefficients are used to adjust the importance of differentiated facial expression parameters. For example, for anime characters, it may be necessary to emphasize more exaggerated facial features, so the corresponding weighting coefficients will be larger; while for realistic characters, it may be necessary to more closely resemble the facial features of real humans, so the corresponding weighting coefficients will be relatively smaller. Through dynamic weighting, the set of personalized facial expression features can be made more consistent with the character's design.

[0103] Step S130: Dynamically adjust the basic expression feature set according to the preset emotional parameters of the animated character to generate an emotion-adapted expression feature set.

[0104] After obtaining the basic set of facial expression features, it is necessary to dynamically adjust them according to the preset emotional parameters of the animated character to generate an emotionally adapted set of facial expression features. The preset emotional parameters of the animated character are set in advance during the animation production process, reflecting the character's emotional state in the current scene. Different emotional states correspond to different facial expressions, therefore, the basic set of facial expression features needs to be adjusted according to the preset emotional parameters.

[0105] Step S131: Obtain the preset emotional parameters of the current scene in the animation script. The preset emotional parameters include emotional intensity values ​​and emotional category identifiers.

[0106] First, the preset emotional parameters for the current scene need to be obtained from the animation script. The animation script is the foundation of animation production, containing the emotional settings of characters in different scenes. The preset emotional parameters include emotional intensity values ​​and emotional category identifiers. The emotional intensity value represents the strength of the character's emotion, such as slight joy or extreme anger; the emotional category identifier is used to identify the category of the character's emotion, such as joy, sadness, anger, or surprise.

[0107] Step S132: Match the corresponding expression adjustment strategy from the preset emotion rule base according to the emotion category identifier. The expression adjustment strategy includes the eye region weight coefficient, the mouth region amplitude threshold and the eyebrow region phase offset.

[0108] After obtaining the emotion category identifier, the system matches the corresponding expression adjustment strategy from a pre-defined emotion rule base. This pre-defined emotion rule base is a database storing expression adjustment strategies for different emotion categories. Each emotion category has its unique expression style, thus requiring different expression adjustment strategies to implement it.

[0109] The facial expression adjustment strategy includes parameters such as the eye region weighting coefficient, the mouth region amplitude threshold, and the eyebrow region phase offset. The eye region weighting coefficient is used to adjust the importance of the eye region in the expression. For example, in a surprised expression, the eyes are more prominent, so the eye region weighting coefficient will be larger. The mouth region amplitude threshold is used to limit the amplitude of mouth movements. For example, in a sad expression, the mouth movements may be smaller, so the mouth region amplitude threshold will be smaller. The eyebrow region phase offset is used to adjust the position and shape of the eyebrows. For example, in an angry expression, the eyebrows may furrow downwards, so the eyebrow region phase offset will be adjusted accordingly.

[0110] Step S133: Dynamically scale the parameters in the expression adjustment strategy based on the emotional intensity value to generate a set of dynamic adjustment parameters that are adapted to the current emotional intensity.

[0111] After obtaining the facial expression adjustment strategy, it is necessary to dynamically scale its various parameters based on the emotional intensity value. The emotional intensity value reflects the strength of the character's emotions, and different emotional intensities require different facial expressions. By dynamically scaling the various parameters in the facial expression adjustment strategy, a set of dynamically adjusted parameters adapted to the current emotional intensity can be generated.

[0112] For example, if the emotional intensity value is large, parameters such as the weight coefficient of the eye region, the amplitude threshold of the mouth region, and the phase offset of the eyebrow region will be increased accordingly to express a stronger emotion; if the emotional intensity value is small, these parameters will be decreased accordingly.

[0113] Step S134: Apply the set of dynamically adjusted parameters to the corresponding region feature data in the set of basic facial expression features, wherein the weight coefficient is adjusted for the eye region features, the amplitude threshold is truncated for the mouth region features, and a phase offset is added to the eyebrow region features.

[0114] After generating the set of dynamically adjusted parameters, they are applied to the corresponding regional feature data in the basic expression feature set. Specifically, weight coefficients are applied to the eye region features to enhance or weaken the expression of the eye region in facial expressions by multiplying by the eye region weight coefficient; amplitude threshold truncation is applied to the mouth region features to limit the amplitude of mouth region movements within the mouth region amplitude threshold range; and phase offsets are added to the eyebrow region features to express different emotional states by adjusting the position and shape of the eyebrows.

[0115] Step S135: Perform smooth transition processing on the adjusted feature data of each region so that the rate of change of facial expression features between adjacent timestamps does not exceed the preset visual coherence threshold, and generate an emotion-adapted facial expression feature set.

[0116] After adjusting the feature data of each region in the basic facial expression feature set, a smooth transition process needs to be performed on the adjusted feature data of each region. The purpose of the smooth transition process is to ensure that the rate of change of facial features between adjacent time stamps does not exceed a preset visual coherence threshold. If the facial features change too quickly, it will make the expressions of the animated characters look unnatural and affect the viewer's visual experience.

[0117] By employing smooth transition processing, changes in facial features between adjacent time stamps can be made more natural and fluid. For example, interpolation algorithms can be used to process the adjusted feature data of each region, allowing facial features to change gradually between adjacent time stamps rather than jumping abruptly. The resulting set of emotion-adapted facial features can better reflect the emotional state of the animated character in the current scene.

[0118] Step S140: The emotion-adapted expression feature set and the personalized expression feature set are fused to obtain the target expression feature set.

[0119] After obtaining the set of emotion-adapted facial expression features and the set of personalized facial expression features, they need to be fused to obtain the target facial expression feature set. The purpose of the fusion process is to combine the basic facial expression features that are adapted to the character's emotions and the personalized facial expression features that are related to the character's type, so as to make the animated character's expressions richer and more realistic.

[0120] Step S141: Normalize the general expression parameters in the emotion-adaptive expression feature set so that their numerical range is consistent with the parameter range in the personalized expression feature set.

[0121] Since general facial expression parameters and personalized facial expression parameters may originate from different calculation processes or feature extraction methods, their numerical ranges may differ. This difference can affect subsequent fusion operations, as directly fusing parameters from different ranges may lead to the overemphasis or neglect of certain features.

[0122] Normalization can be achieved using various methods, such as linear normalization. The basic idea of ​​linear normalization is to map the values ​​of general facial expression parameters to a specific range, typically between 0 and 1. Specifically, the maximum and minimum values ​​of the general facial expression parameters are first identified. Then, for each general facial expression parameter, the minimum value is subtracted and the result is divided by the difference between the maximum and minimum values, transforming it to the range of 0 to 1. In this way, all general facial expression parameters are adjusted to a uniform numerical range, facilitating comparison and integration with parameters in a personalized facial expression feature set.

[0123] Step S142: Obtain the fusion weight coefficient from the preset fusion rule library according to the character type identifier. The fusion weight coefficient includes the basic expression weight and the personalized expression weight.

[0124] After normalizing the general facial expression parameters, the next step is to retrieve the fusion weight coefficients from a pre-defined fusion rule library based on the character type identifier. The pre-defined fusion rule library is a database storing the fusion weight coefficients corresponding to different character types. The character type identifier uniquely identifies the type of the animated character; different character types may require different fusion ratios for basic and personalized facial expressions.

[0125] The fusion weighting coefficients include basic expression weights and personalized expression weights. Basic expression weights determine the importance of the emotion-adaptive expression feature set in the fusion process, while personalized expression weights determine the importance of the personalized expression feature set. For example, for a cartoon-style character, personalized expressions might be more emphasized, thus the personalized expression weight would be relatively large; while for a realistic-style character, basic expressions might be more important, resulting in a relatively large basic expression weight.

[0126] Step S143: Linearly superimpose the general expression parameters and the differential expression parameters according to the corresponding fusion weight coefficients to generate a preliminary fusion feature vector.

[0127] After obtaining the fusion weight coefficients, the normalized general expression parameters and the differentiated expression parameters are linearly superimposed according to their corresponding fusion weight coefficients. Linear superposition is a simple and effective fusion method. It involves multiplying the general expression parameters by the basic expression weights, multiplying the differentiated expression parameters by the personalized expression weights, and then adding the two together to obtain a preliminary fusion feature vector.

[0128] For example, let the general facial expression parameters be a vector A, the differentiated facial expression parameters be a vector B, the basic facial expression weight be W1, and the personalized facial expression weight be W2. Then, the initial fused feature vector C is calculated as: C = A * W1 + B * W2. In this way, the basic facial expression features and the personalized facial expression features are organically combined to generate an initial fused feature vector containing information from both.

[0129] Step S144: Perform nonlinear feature enhancement processing on the preliminary fused feature vector to obtain an enhanced feature vector. The feature enhancement processing includes nonlinear mapping of the feature values ​​of the key expression region through the hyperbolic tangent function.

[0130] While the initial fused feature vector already contains information on basic and personalized facial expression features, further processing may be needed to enhance the expressiveness of the expressions. Here, a non-linear feature enhancement process will be applied to the initial fused feature vector.

[0131] Nonlinear feature enhancement employs the hyperbolic tangent function to perform a nonlinear mapping on the feature values ​​of key facial expression regions. The hyperbolic tangent function is a commonly used nonlinear activation function that maps input values ​​to a specific interval and has a smooth curve shape. In facial expression feature enhancement, the feature values ​​of key facial expression regions may not be prominent enough for various reasons. The nonlinear mapping using the hyperbolic tangent function can enhance the expressiveness of these feature values, making the expressions more vivid and obvious.

[0132] For example, feature values ​​of key facial expression regions such as the eyes and mouth are input into a hyperbolic tangent function for processing. The hyperbolic tangent function adjusts the input values ​​according to their magnitude, making smaller values ​​smaller and larger values ​​larger, thereby enhancing the feature representation of key facial expression regions.

[0133] Step S145: Perform a temporal consistency check on the enhanced feature vector to ensure that the feature changes between adjacent frames meet the preset animation smoothness standard.

[0134] After obtaining the enhanced feature vectors, it is also necessary to perform a consistency check on them in the temporal dimension. Animation consists of a series of consecutive frames, and the changes in facial features between adjacent frames should meet certain smoothness standards; otherwise, the animation will look unnatural.

[0135] The temporal consistency check examines the changes in feature vectors between adjacent frames. A preset animation smoothness standard defines the maximum range of change in feature vectors between adjacent frames. If the change in feature vectors between adjacent frames exceeds this range, the feature vectors need to be adjusted to make the changes smoother.

[0136] For example, interpolation can be used to adjust the feature vectors. Interpolation is a method of estimating unknown data points using known data points. In temporal consistency verification, the feature vectors of intermediate frames can be calculated using interpolation algorithms based on the feature vectors of adjacent frames, making the feature changes between adjacent frames more continuous and natural. Ultimately, after temporal consistency verification, the resulting feature vectors can ensure the temporal smoothness of the animated character's expressions, generating a target expression feature set.

[0137] Step S150: Based on the target facial feature set, drive the 3D facial model of the animated character to generate continuous facial animation, and synchronously bind the body motion data subset to the 3D body model to generate complete character animation.

[0138] After obtaining the target facial expression feature set, the next step is to use this feature set to drive the 3D facial model of the animated character to generate continuous facial expression animation. Simultaneously, a subset of body motion data is bound to the 3D body model to generate a complete character animation. This step is the ultimate goal of the entire film and television animation character expression generation method, aiming to transform the previously collected and processed data into vivid and realistic animation effects.

[0139] Step S151: Map the expression parameters in the target expression feature set to the hybrid shape weight space of the three-dimensional facial model to generate an initial expression-driven weight set.

[0140] First, the expression parameters in the target expression feature set need to be mapped to the blend shape weight space of the 3D facial model. 3D facial models typically use blend shape deformation technology to achieve expression animation. This technology expresses various different expressions by weighting and combining different basic shapes (i.e., blend shapes).

[0141] The facial expression parameters in the target facial expression feature set reflect the facial expression characteristics of the animated character in the current scene. Mapping these parameters to the blend shape weight space involves assigning a corresponding weight value to each blend shape. These weight values ​​represent the degree of contribution of each blend shape to the current facial expression. Through this mapping, an initial facial expression driving weight set is generated, which contains a set of weight values ​​used to drive the deformation of the 3D facial model.

[0142] Step S152: Perform physical rationality verification on the initial expression-driven weight set according to the facial anatomical constraints of the dynamic partition. The facial anatomical constraints include the muscle contraction direction threshold and the safe range of skin deformation for each region, and dynamically adjust the constraint intensity based on the expression type.

[0143] After generating the initial set of expression-driven weights, their physical plausibility needs to be verified. This is because in actual facial movements, the contraction of facial muscles and the deformation of the skin are subject to certain physiological limitations. To make the generated facial animations more realistic and natural, the initial set of expression-driven weights needs to be checked against the facial anatomical constraints of dynamic partitioning.

[0144] Facial anatomical constraints include regional muscle contraction direction thresholds and safe ranges for skin deformation. Different facial regions have different muscle structures and movement patterns; therefore, it is necessary to set corresponding muscle contraction direction thresholds for each region to ensure that the direction of muscle contraction conforms to physiological laws. The safe range for skin deformation limits the maximum degree of skin deformation during facial expressions, preventing excessive stretching or deformation.

[0145] Furthermore, the constraint strength is dynamically adjusted based on the expression type. Different expression types may require different levels of constraint; for example, some exaggerated expressions may allow for greater muscle contraction and skin deformation, while some subtle expressions require stricter constraints. By dynamically adjusting the constraint strength, facial animation can maintain physical plausibility across different expression types.

[0146] Step S153: Iteratively adjust the driving weights that do not meet the facial anatomical constraints until all driving weights meet the preset biomechanical feasibility threshold. Then, input the adjusted driving weights into the real-time rendering engine to drive the vertex displacement of the 3D facial model to generate continuous facial expression animation.

[0147] If certain driving weights are found to be inconsistent with facial anatomical constraints during the physical rationality verification process, these driving weights need to be iteratively adjusted. The iterative adjustment process is a continuous optimization process, in which the driving weights are repeatedly checked and adjusted until they gradually meet the preset biomechanical feasibility threshold.

[0148] To facilitate iterative adjustments, a 3D facial musculoskeletal simulation model can be established. This model includes information such as skin elasticity parameters, muscle traction parameters, and skeletal motion constraint parameters, enabling the simulation of facial muscle movement and skin deformation. The initial expression-driven weights are converted into corresponding muscle activation intensity parameters and input into the musculoskeletal simulation model for physical simulation calculations.

[0149] During the physical simulation calculations, the system detects any regions that violate preset biomechanical rules, such as areas of excessive skin stretching and areas of conflicting muscle contractions. For each violation region, a corresponding constraint adjustment is generated, which includes information such as the maximum stretching compensation coefficient and the muscle co-activation coefficient. This constraint adjustment is then fed back to the driving weight adjustment module, which attenuates the driving weights of the violation regions and redistributes them to adjacent compliant regions.

[0150] By continuously performing physical simulation calculations and adjusting the driving weights, until all driving weights meet the preset biomechanical feasibility threshold, the adjusted driving weights are input into the real-time rendering engine. The real-time rendering engine then drives the vertex displacement of the 3D facial model according to these weights, thereby generating continuous facial expression animations.

[0151] Step S154: Perform a visual quality assessment on the generated continuous facial expression animation. When a region of facial distortion is detected in the continuous facial expression animation, automatically trigger a local weight fine-tuning process to optimize the performance of the distorted region.

[0152] After generating continuous facial expression animations, a visual quality assessment is required. The purpose of the visual quality assessment is to check for areas of facial distortion in the animation and ensure the overall quality of the animation.

[0153] To perform visual quality assessment, a reference expression sequence corresponding to the current emotion category is extracted from a pre-defined standard expression library. This standard expression library is a database storing standard expression keyframes for various emotion categories; these keyframes have been reviewed by experts and are of high quality and representativeness. The generated continuous expression animation is then spatiotemporally aligned with the reference expression sequence, ensuring they share the same time reference and spatial coordinate system for comparison.

[0154] Then, the feature difference between the spatiotemporally aligned continuous facial expression animation and each corresponding frame in the reference facial expression sequence is calculated. The feature difference includes indicators such as key point position deviation, motion trajectory similarity, and expression intensity difference. When the feature difference of several consecutive frames exceeds a preset visual tolerance threshold, it is determined to be a region of facial distortion, and the corresponding facial anatomical regions are identified, such as eyelid regions, corner of mouth regions, and nasal alar regions.

[0155] Based on the partition type, a corresponding fine-tuning strategy template is loaded. This template includes partition boundary constraints and influence coefficients of adjacent regions. While maintaining the driving weights of non-distorted regions, constrained optimization calculations are performed on the driving weights of distorted regions. The objective function of this constrained optimization calculation includes visual smoothness and biomechanical constraints. By optimizing this objective function, biomechanical requirements can be met while ensuring natural and smooth facial expressions.

[0156] The optimized driving weights are progressively blended with the original driving weights to generate a naturally transitioning facial expression correction animation. A second quality assessment is then performed on the corrected facial expression animation until the feature difference in all regions falls below the visual tolerance threshold, ensuring that the animation's visual quality meets the expected results.

[0157] Step S155: Establish a time synchronization relationship between facial expression animation and body motion data, wherein the time synchronization relationship includes key event alignment points and motion response delay parameters.

[0158] While generating continuous facial expression animations, it is also necessary to synchronously bind a subset of body motion data to a 3D body model to generate complete character animations. First, it is essential to establish a time synchronization relationship between facial expression animations and body motion data.

[0159] The time synchronization relationship includes key event alignment points and action response delay parameters. Key event alignment points refer to the time points when representative events in facial expressions and body movements occur, such as the start time of a facial expression and the start time of a body movement. By determining these key event alignment points, the temporal consistency of facial expressions and body movements can be ensured.

[0160] The motion response delay parameter takes into account the delay in the body's response to facial expressions. In actual human movement, there may be a certain time difference between body movements and facial expressions. For example, after the body performs a certain action, there may be a brief delay before the facial expression changes. By setting the motion response delay parameter, this time difference can be simulated more realistically, making character animation more natural.

[0161] Step S156: Based on the skeleton-driven character model topology, obtain body-expression association parameters that match the character type from the preset binding rule library. The body-expression association parameters include the dynamic adjustment coefficient of facial skinning weight when the head rotates and the micro-expression intensity mapping table triggered by limb movements. Adapt different model topologies through the skinning binding weight adaptive algorithm.

[0162] Next, based on the skeleton-driven character model topology, body-expression association parameters matching the character type are obtained from a pre-defined binding rule library. The skeleton-driven character model controls the character's movements through the motion of bones; different character types have different skeletal structures and movement patterns.

[0163] The body-expression correlation parameters include dynamic adjustment coefficients for facial skin weights during head rotation and a micro-expression intensity mapping table triggered by limb movements. The dynamic adjustment coefficients for facial skin weights during head rotation adjust the degree of deformation of the facial skin during head rotation, ensuring the naturalness of facial expressions during head movements. The micro-expression intensity mapping table triggered by limb movements specifies the micro-expression intensity corresponding to different limb movements; for example, waving an arm may trigger a slight upward turn of the corners of the mouth.

[0164] Since different character models may have different topologies, an adaptive skinning rigging weight algorithm is needed to adapt to these different model topologies. The adaptive skinning rigging weight algorithm automatically adjusts the skinning rigging weights based on the character model's skeletal structure and vertex distribution, so that facial expressions and body movements can be accurately rigging onto the character model.

[0165] Step S157: Input the body motion data in the subset of body motion data into the physics engine for kinematic calculation to generate skeletal drive data that conforms to the character's body shape characteristics.

[0166] The body motion data from the subset of body motion data is input into the physics engine for kinematic calculation. A physics engine is a software tool that can simulate the physical motion of an object; it can calculate the object's trajectory and posture based on the input motion data.

[0167] In this embodiment, the physics engine calculates the motion information of the character's skeleton based on body motion data, generating skeletal drive data that conforms to the character's body shape characteristics. These characteristics include factors such as height, weight, and limb length, which affect the character's movement and posture. The physics engine considers these factors to ensure that the generated skeletal drive data accurately reflects the character's body shape characteristics.

[0168] Step S158: Dynamically adjust facial expression features based on the body-expression correlation parameters to simulate the physical effects of body movement on facial expressions.

[0169] After generating skeletal drive data, facial expression features are dynamically adjusted based on body-expression correlation parameters. There is a certain physical connection between body movement and facial expressions; for example, strenuous physical activity can lead to changes in facial expressions, such as the tension in facial muscles during rapid breathing.

[0170] Based on body-expression correlation parameters, the previously generated facial expression features are adjusted. For example, the deformation of the facial skin during head rotation is adjusted according to the dynamic adjustment coefficient of the facial skin weight; corresponding micro-expressions are added based on the micro-expression intensity mapping table triggered by limb movements. In this way, the physical effects of body movement on facial expressions are simulated, making the character animation more realistic and natural.

[0171] Step S159: Perform rendering channel synthesis on the adjusted facial expression features and skeletal driving data to output a complete character animation sequence with physical realism.

[0172] Finally, the adjusted facial expression features and skeletal driving data are composited using render passes. Render pass compositing is the process of combining information from different render passes (such as facial expressions, body movements, etc.) to generate the final animated image.

[0173] By combining adjusted facial expression features and skeletal driving data through rendering pass-through compositing, a complete character animation sequence with physical realism is output. This complete character animation sequence includes the character's facial expressions and body movements, and both are consistent in time and space, presenting the audience with a vivid and lifelike character image.

[0174] The entire method for generating facial expressions for film and animation characters also involves some additional steps, such as model training.

[0175] In detail, the feature decoupling neural network is the key model for achieving feature decoupling of facial action data. It needs to be trained to accurately separate basic expression features and personalized expression features.

[0176] Specifically, the training dataset is first obtained, which contains a large number of facial motion data samples. Each sample has a corresponding basic expression label and a personalized expression label. The basic expression label is used to identify general expressions in the samples that are unrelated to the character type, while the personalized expression label is used to identify differentiated expressions that are related to the character type.

[0177] Before training begins, the parameters of the feature-decoupled neural network need to be initialized. There are various methods for parameter initialization, such as random initialization and Xavier initialization. Different initialization methods will affect the training speed and convergence of the neural network, and an appropriate initialization method needs to be selected based on the specific situation.

[0178] Training parameters include learning rate, batch size, and number of training epochs. The learning rate controls the step size of parameter updates during neural network training; an excessively large learning rate may lead to training instability, while an excessively small learning rate will slow down training. Batch size refers to the number of samples input into the neural network for each training iteration; an appropriate batch size can improve training efficiency. The number of training epochs indicates the number of times the entire training dataset is trained; the appropriate number of epochs is usually determined based on the size and complexity of the training dataset.

[0179] Then, facial motion data samples from the training dataset are input into the feature decoupling neural network for forward propagation. During this process, the shared encoder layer extracts spatiotemporal features from the time series of facial motion data to obtain a hybrid expression feature vector. Then, the basic feature decoder and the personalized feature decoder decode the basic expression features and personalized expression features from the hybrid expression feature vector, respectively.

[0180] Then, based on the decoded basic and personalized facial expression features, and the true labels of the facial action data samples, a loss function is calculated. The loss function measures the difference between the neural network output and the true labels; common loss functions include mean squared error loss and cross-entropy loss. In this embodiment, the loss function may simultaneously consider the errors of both the basic and personalized facial expression features, as well as the orthogonality constraints between them.

[0181] Based on the calculated loss function, backpropagation is performed. Backpropagation is an algorithm for calculating gradients, which can calculate the gradient of the loss function with respect to each parameter in the neural network. Then, based on the gradients and the learning rate, the parameters of the neural network are updated, causing the loss function to gradually decrease.

[0182] Finally, repeat the above steps until the training termination condition is met. The training termination condition can be reaching a preset number of training epochs, or the loss function converging to a small value. By continuously repeating the processes of forward propagation, loss function calculation, backpropagation, and parameter update, the feature decoupling neural network will gradually learn how to accurately separate basic facial expression features from personalized facial expression features. When the training termination condition is met, the training process ends, and the trained feature decoupling neural network is obtained.

[0183] Furthermore, in the above embodiments, the emotion rule base is an important basis for adjusting the set of basic facial expression features according to the preset emotion parameters of the animated character, and its construction and training process is as follows.

[0184] First, a large number of facial expression data samples containing different emotion categories and intensities were collected. These samples could come from facial expression capture data of real actors or be extracted from animated works through manual annotation. Each sample recorded the corresponding emotion category identifier, emotion intensity value, and corresponding facial expression features.

[0185] Then, the collected emotional data samples are analyzed to identify the patterns of facial expression changes under different emotional categories and intensities. For example, the changes in features of areas such as the eyes, mouth, and eyebrows are analyzed under different emotional categories such as joy, sadness, and anger, and how different emotional intensities affect the feature expression of these areas.

[0186] Based on the analysis of the correlation between emotion and facial expression, an initial facial expression adjustment strategy was formulated. This strategy includes parameters such as the eye region weighting coefficient, the mouth region amplitude threshold, and the eyebrow region phase offset. Corresponding default parameter values ​​were set for different emotion categories. For example, for joy, the eye region weighting coefficient might be increased to emphasize the brightness and widening of the eyes; for sadness, the mouth region amplitude threshold might be decreased to make mouth movements more restrained.

[0187] Based on this, training parameters, such as learning rate and number of training epochs, are set for the emotion rule base. Simultaneously, evaluation metrics are determined to measure the performance of the emotion rule base. Evaluation metrics may include the accuracy of facial expression adjustment and the similarity to real facial expressions.

[0188] The initial facial expression adjustment strategy is trained and optimized using collected sentiment data samples. The sentiment category identifier and sentiment intensity value of the samples are input into the sentiment rule base. Adjusted facial expression features are generated based on the current adjustment strategy and then compared with the real facial expression features of the samples to calculate an evaluation metric. Based on the evaluation metric results, the parameters in the facial expression adjustment strategy are adjusted to make the generated facial expression features closer to real expressions. This process is repeated until the evaluation metric reaches a satisfactory level.

[0189] Finally, the trained sentiment rule base can be validated using an independent validation dataset. The validation dataset contains sentiment data samples that were not used in the training. Validation reveals potential problems in the sentiment rule base, such as inaccurate adjustments for certain sentiment categories or intensities. These issues are then addressed through further refinement and optimization of the sentiment rule base, ultimately resulting in an accurate and reliable one.

[0190] Furthermore, the fusion rule base is used to determine the fusion weight coefficients of the emotion-adapted facial expression feature set and the personalized facial expression feature set. Its construction and maintenance process is as follows.

[0191] First, the animated characters are categorized into types such as anime characters, realistic characters, and science fiction characters. Different character types have different facial expressions and styles, requiring different blending weight coefficients.

[0192] For each character type, a large number of facial expression data samples were collected. These samples include corresponding data for the emotion-adapted facial expression feature set and the personalized facial expression feature set. By analyzing these samples, we can understand the importance and expression methods of basic and personalized facial expressions under different character types.

[0193] Based on the collected facial expression data for different character types, an initial fusion weight coefficient is determined for each character type. The initial fusion weight coefficient includes a base expression weight and a personalized expression weight. For example, for anime characters, due to their more exaggerated and personalized expression style, a larger personalized expression weight may be assigned; for realistic characters, more emphasis may be placed on the realism of the base expression, resulting in a larger base expression weight.

[0194] Then, the initial fusion weight coefficients are adjusted and optimized using the collected facial expression data samples. The emotionally adapted facial expression feature set and the personalized facial expression feature set of the samples are fused according to the initial fusion weight coefficients to generate fused facial expression features. Then, the fused facial expression features are evaluated to determine whether they conform to the characteristics and requirements of the role type through manual or automated evaluation metrics. Based on the evaluation results, the fusion weight coefficients are adjusted to make the fused facial expression features more ideal. This process is repeated until satisfactory fusion weight coefficients are obtained.

[0195] With the development of animation production technology and the emergence of new character types, the blending rule library needs to be updated and maintained regularly. This involves collecting new facial expression data samples, re-evaluating and adjusting the blending weight coefficients of existing character types, and adding new character types and their corresponding blending weight coefficients. Through regular updates and maintenance, the blending rule library can be ensured to always adapt to different animation production needs.

[0196] Furthermore, the standard emoji library is an important reference for evaluating the visual quality of continuous facial animation, and its construction and updating process is as follows.

[0197] First, standard facial expression samples are collected across various emotional categories. These samples can be derived from professional actors' performances, classic expressions from animated works, etc. Each sample undergoes expert review to ensure its accuracy and representativeness.

[0198] Then, the collected standard facial expression samples were categorized according to emotion type, and each standard facial expression sample was annotated in detail. The annotation information included emotion category identifiers, feature descriptions of facial expression keyframes, etc.

[0199] Based on the categorized and labeled standard facial expression samples, a standard facial expression sequence corresponding to each emotion category is constructed. The standard facial expression sequence contains standard facial expression keyframes approved by experts. These keyframes are arranged in chronological order to form a complete facial expression change process.

[0200] With advancements in animation technology and evolving aesthetic sensibilities, standard emoji libraries require regular updates. This involves collecting new standard emoji samples, optimizing and adjusting existing standard emoji sequences, and adding new emotion categories and corresponding standard emoji sequences. Regular updates ensure that the standard emoji library consistently reflects current emoji production standards and aesthetic requirements.

[0201] The data collection process involves collecting facial and body movement data from actors. This data is sensitive to privacy and requires appropriate privacy protection and anti-leakage technologies.

[0202] For example, after acquiring real-time motion data from actors, the data is immediately encrypted. Either symmetric or asymmetric encryption algorithms can be used. Symmetric encryption algorithms use the same key for both encryption and decryption, offering the advantage of fast encryption speed; asymmetric encryption algorithms use public and private keys for encryption and decryption, providing higher security. Encryption ensures the security of data during transmission and storage, preventing unauthorized access and tampering.

[0203] Implement a strict access control mechanism for the database storing action data. Only authorized personnel should be able to access and process this data. Role-based access control can be used to assign different access permissions to different personnel; for example, data acquisition personnel can only perform data acquisition operations, and data processing personnel can only perform data processing operations. Access control restricts the scope of data access and reduces the risk of data leakage.

[0204] When using motion data for training and processing, the data is anonymized. This involves removing personally identifiable information about the actors, such as names and ID numbers, and retaining only feature data related to facial expressions and movements. Anonymization protects the actors' privacy and prevents the data from being used for unauthorized purposes.

[0205] Regularly back up action data and store it on secure storage devices. Simultaneously, establish a data recovery mechanism to promptly recover data in the event of loss or corruption. Through data backup and recovery, ensure data integrity and availability, and prevent data loss due to unforeseen events.

[0206] Furthermore, a security audit mechanism can be established to audit the data access and processing processes. All operations on the action data can be recorded, including access time, operator, and operation content. Through security auditing, abnormal access and processing behaviors can be detected promptly, allowing for appropriate preventative and handling measures to ensure data security.

[0207] Through the detailed implementation steps described above, those skilled in the art can seamlessly implement a method for generating facial expressions for film and animation characters by combining motion capture, including data acquisition, feature decoupling, expression adjustment, feature fusion, and animation generation, while ensuring data security and privacy protection, and generating high-quality and realistic facial expressions and movements for film and animation characters.

[0208] Figure 2 The diagram illustrates exemplary hardware and software components of a motion-capture-integrated film and animation character facial expression generation system 100, which can implement the inventive concept, according to some embodiments of the present invention. For example, a processor 120 may be used in the motion-capture-integrated film and animation character facial expression generation system 100 and to perform the functions described in the present invention.

[0209] The motion capture-based character facial expression generation system 100 can be a general-purpose server or a special-purpose server; both can be used to implement the motion capture-based character facial expression generation method of this invention. Although only one server is shown in this invention, for convenience, the functions described in this invention can be implemented in a distributed manner on multiple similar platforms to balance the load.

[0210] For example, a motion capture-based film and animation character facial expression generation system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the motion capture-based film and animation character facial expression generation system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present invention can be implemented according to these program instructions. The motion capture-based film and animation character facial expression generation system 100 also includes an I / O interface 150 between the computer and other input / output devices.

[0211] For ease of explanation, only one processor is described in the motion capture-integrated film and animation character facial expression generation system 100. However, it should be noted that the motion capture-integrated film and animation character facial expression generation system 100 of the present invention may also include multiple processors. Therefore, the steps executed by one processor described in the present invention may also be executed jointly by multiple processors or individually. For example, if the processor of the motion capture-integrated film and animation character facial expression generation system 100 executes steps A and B, it should be understood that steps A and B may also be executed jointly by two different processors or individually by one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor jointly execute steps A and B.

[0212] Furthermore, this embodiment of the invention also provides a readable storage medium, wherein computer-executable instructions are preset in the readable storage medium, and when the processor executes the computer-executable instructions, the above-described method for generating facial expressions of film and animation characters in conjunction with motion capture is implemented.

[0213] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A method for generating facial expressions for film and animation characters using motion capture, characterized in that, The method includes: The real-time motion data set of the target actor is collected by motion capture equipment. The real-time motion data set includes a facial motion data subset and a body motion data subset. The facial motion data subset consists of facial key point displacement data with multiple consecutive timestamps. A preset expression mapping model is invoked to perform feature decoupling processing on the facial motion data subset, resulting in a basic expression feature set and a personalized expression feature set, including: The subset of facial motion data is input into a trained feature decoupling neural network, which includes a shared encoder layer, a basic feature decoder, and a personalized feature decoder. The shared encoder layer extracts spatiotemporal features from the time series of facial motion data to obtain a hybrid expression feature vector. The hybrid facial expression feature vector is input into the basic feature decoder and the personalized feature decoder, respectively. The basic feature decoder outputs general facial expression parameters that are independent of the character type, and the personalized feature decoder outputs differentiated facial expression parameters that are related to the character type. The general expression parameters and the differential expression parameters are subjected to orthogonal constraint processing so that the projection directions of the general expression parameters and the differential expression parameters in the feature space remain orthogonal. The differentiated expression parameters are dynamically weighted based on preset role type identifiers to generate the personalized expression feature set, wherein different role type identifiers correspond to different combinations of weighting coefficients; The basic expression feature set contains general expression parameters that are independent of the character type, while the personalized expression feature set contains differentiated expression parameters that are bound to the character type. The basic expression feature set is dynamically adjusted based on the preset emotional parameters of the animated character to generate an emotion-adaptive expression feature set. The set of emotion-adapted facial expression features is fused with the set of personalized facial expression features to obtain the target facial expression feature set; The target facial feature set drives the 3D facial model of the animated character to generate continuous facial animation, and the body motion data subset is synchronously bound to the 3D body model to generate complete character animation.

2. The method for generating facial expressions for film and animation characters by combining motion capture according to claim 1, characterized in that, The set of real-time motion data of the target actor collected by motion capture equipment includes: A first array of markers is placed on the face of the target actor, covering the movement area of ​​the facial muscle groups, and each marker corresponds to the synchronous monitoring of at least three optical capture sensors. The optical capture sensor captures the three-dimensional spatial coordinate change data of the first marker point array at a preset sampling frequency to generate the facial motion data subset; A second array of markers is placed at the limb joints of the target actor. The second array of markers contains at least twenty markers to cover the rotation angle monitoring area of ​​the key movement joints. The acceleration and angular velocity data of the second marker array are collected in real time by the inertial measurement unit, and the body motion data subset is generated by combining the inverse kinematics algorithm. The facial motion data subset and the body motion data subset are time-stamp aligned to ensure that the facial key point displacement data and limb joint rotation data at the same timestamp are time-synchronized.

3. The method for generating facial expressions for film and animation characters by combining motion capture according to claim 1, characterized in that, The step of dynamically adjusting the basic facial expression feature set according to the preset emotional parameters of the animated character to generate an emotion-adapted facial expression feature set includes: Obtain the preset emotional parameters of the current scene in the animation script, wherein the preset emotional parameters include emotional intensity values ​​and emotional category identifiers; According to the emotion category identifier, the corresponding expression adjustment strategy is matched from the preset emotion rule base. The expression adjustment strategy includes the eye area weight coefficient, the mouth area amplitude threshold and the eyebrow area phase offset. Based on the emotional intensity value, the parameters in the expression adjustment strategy are dynamically scaled to generate a set of dynamic adjustment parameters that are adapted to the current emotional intensity. The set of dynamically adjusted parameters is applied to the corresponding regional feature data in the set of basic facial expression features, wherein the weight coefficient is adjusted for the eye region features, the amplitude threshold is truncated for the mouth region features, and a phase offset is added to the eyebrow region features. The adjusted feature data of each region are processed to ensure that the rate of change of facial features between adjacent timestamps does not exceed the preset visual coherence threshold, thereby generating an emotion-adapted facial feature set.

4. The method for generating facial expressions for film and animation characters by combining motion capture according to claim 1, characterized in that, The process of fusing the emotion-adapted facial expression feature set with the personalized facial expression feature set to obtain the target facial expression feature set includes: The general expression parameters in the emotion-adaptive expression feature set are normalized to align their numerical range with the parameter range in the personalized expression feature set. The fusion weight coefficient is obtained from the preset fusion rule base based on the character type identifier. The fusion weight coefficient includes the basic expression weight and the personalized expression weight. The general expression parameters and the differential expression parameters are linearly superimposed according to the corresponding fusion weight coefficients to generate a preliminary fusion feature vector; The preliminary fused feature vector is subjected to nonlinear feature enhancement processing to obtain an enhanced feature vector. The feature enhancement processing includes nonlinear mapping of the feature values ​​of the key expression region through the hyperbolic tangent function. The enhanced feature vectors are subjected to temporal consistency checks to ensure that feature changes between adjacent frames meet the preset animation smoothness standards.

5. The method for generating facial expressions for film and animation characters by combining motion capture according to claim 1, characterized in that, The step of generating continuous facial expression animation based on the target facial expression feature set to drive the 3D facial model of the animated character includes: The expression parameters in the target expression feature set are mapped to the hybrid shape weight space of the three-dimensional facial model to generate an initial expression-driven weight set; The physical rationality of the initial expression-driven weight set is verified based on the facial anatomical constraints of the dynamic partition. The facial anatomical constraints include the threshold of muscle contraction direction and the safe range of skin deformation for each region, and the constraint strength is dynamically adjusted based on the expression type. The driving weights that do not meet the facial anatomical constraints are iteratively adjusted until all driving weights meet the preset biomechanical feasibility threshold. The adjusted driving weights are then input into the real-time rendering engine to drive the vertex displacement of the 3D facial model to generate continuous facial expression animation. The generated continuous facial expression animation is subjected to visual quality evaluation. When a region of facial distortion is detected in the continuous facial expression animation, a local weight fine-tuning process is automatically triggered to optimize the performance of the distorted region.

6. The method for generating facial expressions for film and animation characters by combining motion capture according to claim 5, characterized in that, The iterative adjustment of driving weights that do not meet the facial anatomical constraints includes: A musculoskeletal simulation model of a three-dimensional facial model is established, the musculoskeletal simulation model including skin elasticity parameters, muscle traction force parameters and bone motion constraint parameters; The initial facial expression driving weights are converted into corresponding muscle activation intensity parameters and input into the musculoskeletal simulation model for physical simulation calculations. Detecting areas in physical simulation calculations that violate preset biomechanical rules, including areas of excessive skin stretching and areas of conflicting muscle contractions; For each violation area, a corresponding constraint adjustment amount is generated, which includes the maximum stretching compensation coefficient and the muscle co-activation coefficient; The constraint adjustment amount is fed back to the driving weight adjustment module, which attenuates the driving weight of the violation area and redistributes it to the adjacent compliant area.

7. The method for generating facial expressions for film and animation characters by combining motion capture according to claim 5, characterized in that, The step involves performing a visual quality assessment on the generated continuous facial expression animation. When a region of facial distortion is detected in the continuous facial expression animation, a local weight fine-tuning process is automatically triggered to optimize the performance of the distorted region, including: Extract a reference expression sequence corresponding to the current emotion category from a preset standard expression library. The reference expression sequence contains standard expression keyframes that have been reviewed and approved by experts. The generated continuous facial expression animation is spatiotemporally aligned with the reference facial expression sequence so that the comparison has the same time reference and spatial coordinate system; Calculate the feature difference between the spatiotemporally aligned continuous facial animation and each corresponding frame in the reference facial expression sequence. The feature difference includes key point position deviation, motion trajectory similarity, and facial expression intensity difference. When the feature difference of N consecutive frames exceeds the preset visual tolerance threshold, it is determined to be a region of facial expression distortion, and the facial anatomical regions corresponding to the distorted regions are identified, including the eyelid region, the corner of the mouth region, and the nasal alar region. Load the corresponding fine-tuning strategy template according to the partition type. The fine-tuning strategy template includes partition boundary constraints and adjacent region influence coefficients. While keeping the driving weights of the non-distorted regions unchanged, the driving weights of the distorted regions are subjected to constrained optimization calculations. The objective function of the constrained optimization calculations includes a visual smoothness term and a biomechanical constraint term. The optimized driving weights are progressively blended with the original driving weights to generate a naturally transitioning facial expression correction animation. A second quality assessment was performed on the corrected facial animation until the feature difference in all regions was below the visual tolerance threshold.

8. The method for generating facial expressions for film and animation characters by combining motion capture according to claim 1, characterized in that, The step of synchronously binding the subset of body motion data to a 3D body model to generate a complete character animation includes: Establish a time synchronization relationship between facial expression animation and body motion data, wherein the time synchronization relationship includes key event alignment points and motion response delay parameters; Based on the skeleton-driven character model topology, body-expression association parameters matching the character type are obtained from a preset binding rule library. The body-expression association parameters include the dynamic adjustment coefficient of facial skin weight when the head rotates and the micro-expression intensity mapping table triggered by limb movements. Different model topologies are adapted through a skin binding weight adaptive algorithm. The body motion data in the aforementioned subset of body motion data is input into the physics engine for kinematic calculation, generating skeletal drive data that conforms to the character's body shape characteristics; The facial expression features are dynamically adjusted based on the body-expression correlation parameters to simulate the physical effects of body movement on facial expressions. The adjusted facial expression features and skeletal driving data are synthesized using rendering channels to output a complete character animation sequence with physical realism.

9. A system for generating facial expressions for film and animation characters using motion capture, characterized in that, The method includes a processor and a memory, the memory and the processor being connected. The memory is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the memory to implement the method for generating facial expressions of film and television animation characters in combination with motion capture as described in any one of claims 1-8.

Images