A humanoid robot facial expression mapping and calibration method

Abstract

The application provides a humanoid robot facial expression mapping and calibration method, the facial expression mapping method constructs a self-supervised discrete expression data set, trains a multilayer perceptron model based on the data set, realizes preliminary mapping from expression parameters to rudder control signals, extracts expression parameters in a real person expression data set, generates robot rudder control signals through the model, constructs a self-supervised time sequence expression data set based on a redirection method, trains a long short-term memory network model, and realizes time sequence mapping of continuous expressions. The facial expression calibration method uses a visual capture device to obtain robot facial expression parameters in real time, gradually optimizes the rudder control signal, and makes the expression parameter response approach the target value. Through data-driven modeling and visual feedback optimization, the application realizes high-fidelity mapping from semantic expression parameters to multi-rudder collaborative control and systematic calibration, effectively improving the naturalness, accuracy and long-term stability of the robot facial expression.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to a method for mapping and calibrating facial expressions in a humanoid robot. Background Technology

[0002] With advancements in materials science, servo control technology, and 3D printing, biomimetic robots have made significant progress in achieving human-like appearances, and artificial intelligence technology has made generative facial expression control possible. Robots with biomimetic appearances and human-like expressions have become a cutting-edge research direction in the field of human-computer interaction. Facial expressions can effectively convey emotions and intentions, enhancing the realism and immersion of interactions. However, the current presentation of robot expressions is still constrained by the following factors:

[0003] In terms of hardware limitations, the number of servo motors and degrees of freedom are limited, and the elastic skin material directly affects the accuracy of facial expressions, making it difficult to accurately reproduce complex facial movements. Modeling difficulties are prominent. Due to the complexity of facial muscle movements, it is difficult to establish an inverse kinematic model from the target expression to the collaborative control of multiple servo motors, resulting in deviations in expression mapping. The lack of calibration is a key bottleneck. The lack of a systematic expression calibration method leads to poor expression repeatability and easy deviation. Furthermore, factors such as mechanical assembly tolerances, part wear, and material creep will further exacerbate the drift in expression output. Therefore, there is an urgent need for a technical solution that balances high-fidelity mapping, temporal coherence, and systematic calibration to solve the technical problems of unnatural, inaccurate, and poorly consistent temporal expressions in existing robots, as well as low long-term repeatability. Summary of the Invention

[0004] This invention provides a method for mapping and calibrating facial expressions of humanoid robots, which solves the technical problems of unnatural, inaccurate, poor temporal continuity and low long-term repeatability of robot expressions caused by the limited number of facial actuators (servo motors), the influence of elastic skin material properties and the lack of systematic modeling and calibration methods in the prior art.

[0005] This invention proposes a facial expression mapping and calibration method for humanoid robots. The method includes a mapping part and a calibration part. The core of the mapping part is to establish a high-fidelity mapping model between high-dimensional, semantic facial expression parameters (represented by Blendshape coefficients) and low-dimensional, physical multi-servo motor cooperative control signals, thereby achieving accurate and smooth reproduction of the target expression on the robot hardware. The mapping method is as follows:

[0006] S1 collects facial expression data of the robot within its physical reach, generates paired data of servo control signals and blendshape parameters through random motor excitation, and obtains a self-supervised discrete expression dataset. The robot's face is equipped with at least 17 servos (4 for eyebrows, 4 for eyelids, 1 for jaw, 4 for lips, and 4 for corners of the mouth), with each servo corresponding to 3-4 blendshape parameters. The random motor excitation program generates 10,000 sets of drive signal combinations (0-1 normalized values), injects hardware safety constraints (such as EyebrowLeftOuter servo drive value ≥ 0.75, UpperEyelidLeft servo drive value ≤ 0.75) and symmetry constraints, and collects the corresponding blendshape coefficients through Apple ARKitLiveLinkFace, which are then paired and stored to form the dataset.

[0007] S2 trains a multilayer perceptron (MLP) model based on a self-supervised discrete facial expression dataset to obtain a preliminary mapping from the Blendshape parameters to the servo control signal;

[0008] S3 extracts the Blendshape parameters from the real human facial expression dataset, generates robot servo control signals through the multilayer perceptron (MLP) model, collects the actual Blendshape sequence generated by the robot, and obtains a self-supervised temporal facial expression dataset.

[0009] S4 trains a Long Short-Term Memory (LSTM) network model based on a self-supervised temporal facial expression dataset. The LSTM network model adopts a structure that combines a temporal feature extraction module and a nonlinear regression module. After training, it obtains the temporal mapping of continuous facial expressions.

[0010] The calibration method aims to address the issue of facial expression output drift caused by factors such as mechanical assembly tolerances, part wear, and material creep. Through a systematic search and optimization process, it determines the precise drive value for each facial servo when achieving a baseline expression, thereby ensuring the long-term repeatability and consistency of the facial expression system. The specific steps are as follows:

[0011] The S5 uses a high-precision facial motion capture device to collect the Blendshape coefficients of the robot's face in real time based on time-series mapping, and obtains the optimization target feedback signal for the calibration process. The device is Apple ARKit LiveLink Face, which is equipped on iPhone 13 and above models, and is connected to the robot control terminal via a wired network (latency ≤50ms), and collects ambient light intensity of 500-1000 lux.

[0012] S6 calibrates the servo motors in the order of facial regions: eyebrows, eyelids, jaw, lips, and corners of the mouth. The calibration order for the mouth region is jaw, lips, and corners of the mouth, in order to obtain a motion-decoupled calibration.

[0013] S7 employs a search strategy that combines coarse and fine granularity to progressively optimize the servo control signal and obtain a Blendshape response that approaches the target value.

[0014] S8 sets corresponding Blendshape response targets for different facial motion servos, calibrates motion range nodes in the eyebrow unit, eyelid unit, jaw unit, lip unit, and corner of mouth unit, and coordinates the adjustment of multiple servos to achieve calibration of complex expressions.

[0015] Furthermore, the facial expression data collected by the robot itself within its physical reach is used to generate paired data of servo control signals and Blendshape parameters through random motor excitation, thereby obtaining a self-supervised discrete facial expression dataset, including:

[0016] S11 generates multiple sets of servo drive signal combinations randomly through a random motor excitation program, under the premise of ensuring mechanical safety. Each signal combination is a drive signal vector containing the normalized position values ​​of all servos to be controlled in the range of 0-1.

[0017] S12 synchronously injects and enforces preset hardware safety constraints and symmetry constraints during the signal combination generation process to exclude combinations of mechanical damage and non-human facial expression signals;

[0018] S13 sends the obtained driving signal vector to the robot. After the robot's motion stabilizes, it collects the response vector of the Blendshape coefficient corresponding to the robot's face. The driving signal vector and the Blendshape response vector are then paired and stored. This process is repeated until enough data samples covering the robot's expression space are collected to form a self-supervised discrete expression dataset.

[0019] Furthermore, a multilayer perceptron (MLP) model is trained based on a self-supervised discrete facial expression dataset to achieve a preliminary mapping from Blendshape parameters to servo control signals, including:

[0020] S21 trains a multilayer perceptron (MLP) model based on a self-supervised discrete facial expression dataset. The input layer of this model consists of 52-dimensional blendshape coefficients, and the output layer dimension is consistent with the number of facial servo motors on the robot. The hidden layers are set to 2-3 layers, with 256-512 neurons in each layer. The ReLU activation function is used, and the Adam optimizer is used for training with mean squared error (MSE) as the loss function. The initial learning rate of the Adam optimizer is 0.001, β=(0.9,0.999), and the weight decay is 1e-4. The model is initially trained for 200 epochs, with a validation set split ratio of 20%. The early stopping strategy is to stop if there is no decrease in the validation set loss after 30 epochs. The model learns the complex nonlinear relationship between the single-frame blendshape coefficient vector and the single-frame normalized servo drive vector, and obtains a mapping model from single-frame static facial expression parameters to single-frame servo control signals.

[0021] Furthermore, the Blendshape parameters are extracted from the real-person facial expression dataset, and robot servo control signals are generated through the Multilayer Perceptron (MLP) model. The actual Blendshape sequences generated by the robot are collected to obtain a self-supervised temporal facial expression dataset, including:

[0022] S31 randomly selects 10 consecutive speaking videos (30 seconds each, 30fps) from the publicly available real-person facial expression audio and video dataset and 8 emotional expression videos (5 videos per class, 15 seconds each) from the HDTF dataset and the RAVDESS dataset, and extracts a Blendshape coefficient sequence of a consecutive speaking video. , convert the sequence The data is input frame by frame into a trained Multilayer Perceptron (MLP) model, which then generates a sequence of robot servo control signals. ;

[0023] S32 will control the servo motor sequence The data is sent to the robot for execution. A computer vision-based facial motion capture system (such as LiveLinkFace with ARKit) is used to synchronously capture the Blendshape response generated by the robot's face when executing the sequence, thus obtaining the actual Blendshape parameter sequence of the response. Obtain a self-supervised temporal facial expression dataset .

[0024] Furthermore, a Long Short-Term Memory (LSTM) network model is trained based on a self-supervised temporal facial expression dataset. The LSTM model employs a structure combining a temporal feature extraction module and a nonlinear regression module. After training, the temporal mapping of continuous facial expressions is obtained, including:

[0025] The temporal feature extraction module consists of two layers of unidirectional Long Short-Term Memory (LSTM) networks with a hidden state dimension of 256 and an inter-layer dropout rate of 0.05-0.2, used to extract high-level temporal features from the input sequence. The nonlinear regression module is a two-layer fully connected network with residual connections, which maps the hidden states of the LSTM network to the servo control signal space, and finally outputs normalized driving values ​​through a Sigmoid layer. The training adopts a sliding window combined with a strategy of passing hidden states across windows, with a window length of 20-40 frames and a stride of 5-15 frames. Within the same sentence, the hidden states and cell states of the LSTM network are passed to the next window in consecutive windows, and the state is reset only at the sentence boundary. After training, a Blendshape sequence of a target expression is input, and the model directly outputs a smooth and temporally coordinated servo control sequence to obtain the temporal mapping of consecutive expressions.

[0026] Furthermore, a search strategy combining coarse-grained and fine-grained methods is employed to progressively optimize the servo control signal, obtaining calibrated drive values ​​that approximate the Blendshape response, including:

[0027] For each servo or servo group to be calibrated, the coarse search stage traverses the entire working range in steps of 10%–20% to quickly locate the driving value range that makes the target blendshape response close to the expected value. The fine-tuning stage searches within the optimal range in steps of 1%–5% to find the driving value that minimizes the target blendshape response error.

[0028] Furthermore, corresponding Blendshape response targets are set for different facial motion servos. Motion range nodes are calibrated in the eyebrow, eyelid, jaw, lip, and corner of the mouth units, respectively. Multiple servos are coordinated and adjusted to achieve the calibration of complex facial expressions. This calibration method defines a clear and quantifiable Blendshape response target for each motion unit, where the English abbreviations and coefficients represent blendshape parameters, as detailed below:

[0029] Eyebrow Unit: By co-optimizing the outer and inner servos of the eyebrow, the combined values ​​of the BrowDown coefficient, BrowInnerUp coefficient, and BrowOuterUp coefficient are minimized to determine the "neutral" position of the eyebrow; through independent optimization, the drive value that makes the BrowDown coefficient stable at about 0.2 (indicating that the eyebrow is pressed down) is found as the "lower limit" of the servo, and the drive value that makes the BrowInnerUp coefficient and BrowOuterUp coefficient stable at about 0.2 (indicating that the eyebrow tip is raised) is found as the "upper limit" of the EyebrowInner servo and EyebrowOuter servo.

[0030] Eyelid Unit: Optimize the UpperEyelid servo to find the drive value that minimizes the sum of the EyeWide and EyeBlink coefficients as the "neutral" position; find the UpperEyelid servo value that stabilizes the EyeWide coefficient at approximately 0.25 as the "maximum open eye" position; find the UpperEyelid servo value that stabilizes the EyeBlink coefficient at approximately 0.25 as the "maximum closed eye" position. Optimize the lower eyelid servo, targeting the EyeSquint coefficient, and determine both ends of its range of motion. Find the LowerEyelid servo value that stabilizes the EyeSquint coefficient at approximately 0.1 as the "maximum squint" position; find the LowerEyelid servo value that stabilizes the EyeSquint coefficient at approximately 0.05 as the "minimum squint" position.

[0031] Jaw unit: Targeting the JawOpen coefficient, find Jaw servo values ​​that stabilize at close to 0 (closed) and approximately 0.3 (fully open).

[0032] Lip unit (with jaw closed): Optimize the upper and lower lip servos separately, using MouthUpperUp and MouthLowerDown coefficients as targets, and determine the drive values ​​corresponding to their minimum and maximum displacements. Find the UpperLip servo value that stabilizes the MouthUpperUp coefficient at approximately 0.1 as the maximum "upper lip lift" position; find the UpperLip servo value that stabilizes the MouthUpperUp coefficient at approximately 0 as the minimum "upper lip lift" position; find the LowerLip servo value that stabilizes the MouthLowerUp coefficient at approximately 0.1 as the maximum "lower lip pull" position; find the LowerLip servo value that stabilizes the MouthLowerUp coefficient at approximately 0 as the minimum "lower lip pull" position.

[0033] Mouth corner unit (with jaw and lips closed): Cooperatively optimize the X and Y direction servos controlling the corner of the mouth. First, minimize the sum of MouthSmile and MouthFrown coefficients to find the "neutral" corner of the mouth, and record the neutral values ​​of CornerX and CornerY servos. Then, cooperatively adjust the servos to make the MouthSmile coefficient reach about 0.3 to determine the "maximum smile" position, and record the maximum values ​​of CornerX and CornerY servos. Cooperatively adjust the servos to make the MouthFrown coefficient reach about 0.1 to determine the "maximum pout" position, and record the minimum values ​​of CornerX and CornerY servos.

[0034] Furthermore, the collected Blendshape data is denoised, and the calibrated servo drive values ​​are written into the robot's non-volatile memory for use in the expression mapping model.

[0035] Furthermore, the present invention also includes a humanoid robot, comprising a facial expression mapping system and a facial expression calibration system, wherein the facial expression mapping system adopts the mapping method described in any of the technical solutions, and the facial expression calibration system adopts the calibration method described in any of the technical solutions.

[0036] Beneficial effects of this invention:

[0037] 1. An end-to-end high-fidelity facial expression mapping pipeline was established: Through a progressive data-driven approach of "self-supervised data acquisition, static MLP mapping and temporal LSTM modeling", the inverse calculation problem from semantic facial expression parameters to complex control of multiple actuators was effectively overcome, solving the problem of difficulty in traditional inverse kinematics modeling. The generated facial expressions have both single-frame accuracy and sequence fluency.

[0038] 2. Achieved systematic, automated, and quantitative facial expression calibration: Through "partitioned sequential calibration + coarse-grained search", automated quantitative calibration is achieved, transforming the debugging process that originally relied on human experience into an automatic optimization process with clear objectives, clear steps, and visual feedback. This greatly improves calibration efficiency and accuracy, fundamentally ensuring the repeatability and long-term stability of facial expression output, and breaking through the bottleneck of reliance on human experience.

[0039] 3. Improved naturalness of facial expressions through motion decoupling design: A strict sequential calibration process was designed specifically for the mouth, a highly coupled area, which effectively decouples the movements of the jaw, lips, and corners of the mouth at the mechanical level, enabling the robot to present coordinated and natural facial expressions that are closer to human physiological characteristics.

[0040] 4. High versatility and robustness: The entire methodological framework does not depend on specific mechanical structures, actuator models or skin materials. It is based on data-driven modeling and visual feedback optimization, and can be widely applied to biomimetic robot facial systems with different configurations and degrees of freedom, making it highly universal.

[0041] 5. This invention establishes a high-fidelity mapping from Blendshape parameters to servo control signals through a “mapping + calibration” linkage design. At the same time, it solves the problem of facial expression drift through automated calibration, thereby achieving a synergistic improvement in the naturalness, accuracy and long-term stability of facial expressions. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the overall framework of the facial expression mapping system;

[0043] Figure 2This is a schematic diagram of an LSTM network structure;

[0044] Figure 3 This is a schematic diagram of the calibration process. Detailed Implementation

[0045] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0046] Numerous specific details are set forth in the following description to provide a thorough understanding of the invention. The described embodiments are only a part of, and not all, of the embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0048] One embodiment of the present invention includes a method for mapping facial expressions of a humanoid robot and a calibration method for linkage. The method for mapping facial expressions of a humanoid robot includes the following steps:

[0049] Step 1: Construct a self-supervised discrete facial expression dataset;

[0050] Blendshape (also often translated as "blending deformation" or "shape key") is a core parameter in 3D graphics used to drive the deformation of model vertices. Essentially, it's a set of weights that control the shape changes of the model. Blendshape coefficients include coefficients for various parts of the face, including eye expression coefficients, eyebrow expression coefficients, mouth and nose expression coefficients, cheek and chin coefficients, and tongue coefficients. The mouth and nose expression coefficients include basic lip movements, smiling and pursing, jaw and mouth opening and closing, and nose movements. Each Blendshape coefficient name uses a mainstream naming convention, and their core logic is that one base model (BaseMesh) corresponds to multiple target deformation models (TargetShapes), and each TargetShape corresponds to one Blendshape parameter (weight), typically ranging from 0 to 1. When the parameter value is 0, the model retains its basic shape; when the value is 1, the model completely transforms into the target shape; values ​​between 0 and 1 represent a linear blend of the basic and target shapes, typically used in scenarios such as facial expression driving and character lip-syncing. Blendshape parameters are divided into control layer parameters and underlying data parameters: control layer parameters include name, weight, and index, which are commonly used "switches" to drive deformation; underlying data parameters include vertex offset, normal offset, and frame / sample count, which are the raw data for model deformation and generally do not require manual modification. In programming, the corresponding weight values ​​can be accurately obtained through "Blendshape name" or "index".

[0051] Under the premise of ensuring mechanical safety and the rationality of facial expressions, a combination of drive signals covering the range of motion of each servo motor is generated in a programmed and random manner to drive the robot's face to produce a large number of physically achievable facial expressions. At the same time, using a computer vision-based facial motion capture system (such as LiveLinkFace with ARKit), the Blendshape coefficients generated by the robot's face when performing these actions are collected synchronously to form the original paired dataset of "drive signal - facial expression response".

[0052] During the generation process, the following constraints are injected simultaneously: Pre-defined hardware safety constraints and symmetry constraints are enforced to prevent damage to the mechanism or the generation of non-human expressions. Hardware safety constraints include mutual exclusion restrictions between the eyebrow and eyelid servos; when the EyebrowLeftOuter value is ≥0.75, the UpperEyelidLeft value is ≤0.75. Linkage restrictions between the jaw and lip servos are also enforced; for example, when the Jaw value is ≥0.80, the UpperLipLeft / Right and LowerLipLeft / Right values ​​are both ≤0.80. Symmetry constraints mandate that the drive values ​​of paired servos with the same function on the left and right sides (EyebrowLeftOuter and EyebrowRightOuter) remain consistent to ensure that the expression conforms to biological characteristics and aesthetic requirements.

[0053] The implementation steps are as follows: Fix the humanoid robot and set up an iPhone device running the Apple ARKit LiveLink Face application in front of it; write a random motor excitation program to randomly generate multiple sets of servo drive signal combinations (each signal is a normalized value in the range of 0-1) under the premise of forcibly executing the above constraints; send the drive vector to the robot, and after its movement stabilizes, synchronously collect the corresponding Blendshape coefficients through ARKit and store the "drive signal vector-Blendshape response vector" in pairs; repeat this process thousands to tens of thousands of times until enough data samples covering the robot's expression space are collected to form a discrete expression dataset.

[0054] Step 2: Train the static redirection model (MLP);

[0055] Based on the aforementioned self-supervised discrete facial expression dataset, a multilayer perceptron (MLP) model is trained. This MLP model learns the complex nonlinear relationship between the single-frame Blendshape coefficient vector and the single-frame normalized servo drive vector. This model achieves a preliminary, frame-level mapping from the facial expression parameter space to the actuator control space, laying the foundation for the "redirection" from human facial expressions to robot facial expressions.

[0056] The input layer dimension of this MLP model is the same as the coefficient dimension of Blendshape (52-dimensional, compatible with ARKit standards), the output layer dimension is the same as the number of servos, and the hidden layers are set to 2-3 layers, each with 256-512 neurons, using the ReLU activation function. The mathematical expression of the ReLU function is as follows:

[0057]

[0058] The ReLU function achieves sparse activation and alleviates the vanishing gradient problem through simple calculation, making it suitable for deep network training. When the input x is greater than 0, the output equals the input, and the gradient is 1; when the input x is less than or equal to 0, the output is 0, and the gradient is 0. To address the neuron death problem, LeakyReLU or PReLU can be used instead, with the following expression: ( (Learnable) The LeakyReLU function preserves small gradients in the negative interval. The PReLU function, with a value between 0.01 and 0.1, should only be used when there is sufficient data. These are learnable parameters.

[0059] This MLP model uses mean squared error (MSE) as the loss function. MSE quantifies the error by calculating the average squared difference between the predicted and actual values. The formula is as follows: For the first The true label of each sample For the model to the first The predicted value for each sample, Given the sample size, its continuous and differentiable property makes it suitable for gradient descent-type optimization algorithms. If there are many outliers in the data, mean absolute error (MAE) or threshold selection (HuberLoss) can be used. Mean absolute error (MAE): Set a threshold ,when When to choose , When using MAE, A threshold selection (HuberLoss) is performed: MSE is used for small errors, and MAE is used for large errors. δ is typically set to 1.0 to ensure continuous differentiability and smooth gradients. The HuberLoss threshold δ needs to be manually adjusted; a smaller δ is closer to MAE, and a larger δ is closer to MSE. Preprocessing is performed based on the data. Since the self-supervised discrete facial expression dataset is centralized for normalization, MSE can be used directly. If there are many outliers, MAE or HuberLoss can be used first, or outliers can be truncated.

[0060] The Adam optimizer is used for training. The main hyperparameters tuned for the Adam optimizer are the learning rate (lr) and momentum parameter. With weight decay, the Adam optimizer achieves adaptive learning rate through first-order momentum (mean gradient) and second-order momentum (mean squared gradient), while addressing the issue of initial momentum approaching zero through bias correction. Parameter configuration: initial learning rate (lr) set to 0.001, momentum parameter... The parameters are (0.9, 0.999). It is a tuple containing two values. , ), The first-order moment decay coefficient is used to control the weighted average (momentum) of historical gradients, making parameter updates smoother and reducing oscillations; first-order momentum (gradient mean): , Default is 0.9. The gradient mean, The current gradient; The second-order moment decay coefficient is used to control the weighted average of the squared historical gradients, which is used to adaptively adjust the learning rate for each parameter. Second-order momentum (mean of squared gradients): , The mean of the gradient, ( (Default 0.999), where The closer to 1: the stronger the momentum, the smoother the update, but the slower the response to the latest gradient. The closer the second moment is to 1, the longer the memory of the second moment is, and the more stable the adaptive learning rate. Corrected first-order momentum: Corrected second momentum: Parameter update: Based on the learning rate, To prevent the denominator from being 0, The default value is 1e-08, which is used to prevent small constants from having a denominator of 0 after parameter updates.

[0061] The Adam optimizer's weight decay (weight_decay) is set to 1e-4 to achieve L2 regularization to prevent overfitting, forcing the weight parameters of the MLP model to be as small as possible, but without letting the parameters become 0. The final result is a sparse model with a relatively uniform weight distribution, which improves the model's noise resistance.

[0062] The priority for hyperparameter tuning is: learning rate (lr) > weight decay (weight_decay) > momentum parameter. > The learning rate can be adjusted according to the task. For simple regression tasks (feature dimension <100), 1e-3 to 5e-3 is recommended, and for complex regression tasks (such as time series prediction), 1e-4 to 5e-4 is recommended. Generally, no major adjustments are needed; only minor adjustments are made during training oscillations. The weight_decay value is increased when the model is complex and the amount of data is small.

[0063] The training loop is as follows: set the number of training rounds, switch the model to training mode; sequentially perform forward propagation to calculate predicted values, calculate loss, clear gradients to zero (to avoid gradient accumulation), backpropagation to calculate parameter gradients, and Adam optimizer to update parameters; during training, hyperparameters can be adjusted by observing the loss curve; after training, switch to evaluation mode and disable gradient calculation for verification. This model learns the complex nonlinear relationship between a single-frame Blendshape coefficient vector and a single-frame normalized servo drive vector, achieving a preliminary, frame-level mapping from the facial expression parameter space to the actuator control space, laying the foundation for the "redirection" from human facial expressions to robot facial expressions.

[0064] Step 3: Construct a self-supervised temporal facial expression dataset;

[0065] Continuous and expressive Blendshape coefficient sequences are extracted from publicly available real-person facial expression audio and video datasets (such as HDTF and RAVDESS). The HDTF dataset, a high-resolution audiovisual face dataset, contains 362 subjects and 15.8 hours of 720P-1080P video, focusing on providing accurate lip shape and head pose data. The RAVDESS dataset contains labeled data for 8 emotions (with normal and strong intensities) from 24 professional actors, suitable for training emotional expression templates. Combining these two datasets can improve the emotional richness and realistic movement of robot expressions through transfer learning.

[0066] In the data preprocessing stage, the video sets in the HDTF and RAVDESS datasets are processed frame by frame using tools such as Mediapipe to extract Blendshape coefficient values. At the same time, the audio in the videos is extracted to construct a high-quality speech-Blendshape dataset that is consistent with the semantics of real human facial expressions, providing support for the semantic association of facial expressions.

[0067] The extracted blendshape coefficient sequences are input frame-by-frame into a trained MLP model, which is then converted into a sequence of robot servo drive signals. This sequence drives the robot to execute the signals, and the actual generated blendshape sequences are recorded by a facial capture system. This results in a time-aligned, self-supervised temporal facial expression dataset of "target blendshape - drive signal," which fully encompasses the nonlinear response characteristics arising from the robot's dynamics and facial skin elasticity. Specifically, blendshape coefficient sequences for continuous speech are extracted from datasets such as HDTF and RAVDESS. The predicted servo control signal sequence is obtained by inputting the MLP model frame by frame. The signal sequence is sent to the robot for execution, and ARKit is used to record the robot's facial Blendshape response sequence throughout the process. Finally, a time series dataset was obtained. .

[0068] Step 4: Train the temporal facial expression generation model (LSTM);

[0069] The model employs a Long Short-Term Memory (LSTM) network to model temporal mapping relationships, using a blendshape coefficient sequence of sequential associations. As input, it directly outputs a smooth and continuous sequence of servo drive signals of the same length. Where T represents the sequence length. The network structure can include an LSTM module for extracting temporal context features and a fully connected module for high-dimensional nonlinear regression. The hidden state dimension of the LSTM module is set to 256, and the dropout rate between layers is usually set to 0.05-0.2. Experiments have shown that 0.1 yields the best results. The nonlinear regression module is a two-layer fully connected network with residual connections, which maps the hidden state of the LSTM to the servo control signal space, and finally outputs the normalized drive value through the Sigmoid layer.

[0070] Inter-layer dropout is a dropout operation added between LSTM layers. It randomly "shuts down" some neurons in the output of the previous LSTM layer with a set probability (e.g., 0.1), breaking redundant feature dependencies and preventing overfitting. LSTM itself has some anti-overfitting ability through gating mechanisms. Excessive dropout (e.g., >0.3) will disrupt long-range dependencies. 0.1 is a "lightweight regularization," which prevents overfitting while preserving effective temporal features to the greatest extent, making it particularly suitable for medium-sized LSTMs with a hidden dimension of 256. In practical parameter tuning, you can first fix the training value at 0.1. If overfitting occurs, you can gradually increase it to 0.15, then 0.2. If underfitting occurs, you can decrease it to 0.05 or set it to 0. Inter-layer dropout can be combined with L2 regularization (weight_decay=1e-4), and it needs to be distinguished from the LSTM's built-in dropout. The sum of the two should not exceed 0.3.

[0071] The Sigmoid function passes through Mapping unbounded inputs to the 0-1 range, its centrally symmetric nature (output 0.5 when input is 0) naturally suits the mid-position control of servos, and its smooth and differentiable characteristics support backpropagation training. The physical control of servos has a defined range (e.g., angle 0-180°, PWM signal 500-2500μs), while the values ​​of the LSTM hidden states after transformation by the fully connected layer are unbounded. The Sigmoid function can achieve normalization, improve interpretability, and adapt to the nonlinear response characteristics of servos. The fully connected layer first reduces the 256-dimensional LSTM hidden states to the number of servo units, constrains the input range (-2,2) using batch normalization (BN) to prevent the Sigmoid layer from entering the saturation region, then outputs the 0-1 normalized drive value for each servo unit through the Sigmoid layer, and finally back-normalizes it to the physical control range of the servo unit using the formula "physical value = minimum value + normalized value × (maximum value - minimum value)".

[0072] During training, a sliding window (window length W=30, stride S=5) combined with a strategy of passing hidden states across windows is employed. Conventional sliding window training cuts off temporal dependencies between windows, while the core of passing hidden states across windows is using the LSTM hidden state (h / c) at the end of the previous window's training as the initial hidden state for the next window, allowing the model to remember long-term temporal information outside the window. Specifically, at the start of each training round, the hidden state (h,c) is initialized to all zeros; during each window's training, the final hidden state of the previous window is used as the initial value, and the gradient flow of the previous window's hidden state is cut off to avoid gradient explosion. LSTM hidden states and cell states from consecutive windows within the same sentence are passed to the next window, simulating the ability to process long sequences. The state is only reset at sentence boundaries, ensuring the coherence and naturalness of the generated driving signals over long time.

[0073] After training, the model can directly convert the input target Blendshape sequence into a smooth servo control sequence, driving the robot to make natural facial expression animations. During the inference phase, the hidden state needs to be kept continuous. When a new frame (or starting frame) is input, the hidden state of the previous frame is used as the initial state of the current input, continuously generating coherent temporal frames.

[0074] In this embodiment, there is also a calibration method linked to the humanoid robot facial expression mapping method. This method uses a systematic search and optimization process to determine the precise drive value of each facial servo when realizing the baseline expression. Based on the division of facial regions, there are at least 17 servos, and the specific distribution of the servos is as follows:

[0075] Eyebrow: EyebrowLeftOuter servo, EyebrowRightOuter servo, EyebrowLeftInner servo, EyebrowRightInner servo;

[0076] Eyelids: Upper Eyelid Left (Right) servo, Lower Eyelid Left (Right) servo;

[0077] Jaw: Jaw servo;

[0078] Lip: Upper LipLeft (Right) servo, Lower LipLeft (Right) servo;

[0079] Mouth corner: LeftCornerX servo, RightCornerX servo, LeftCornerY servo, RightCornerY servo;

[0080] Calibration specifically includes the following steps:

[0081] Step 1: Visual feedback acquisition;

[0082] High-precision facial motion capture equipment (such as Apple ARKit LiveLink Face) is used to collect the robot's facial blendshape coefficients in real time, which are then used as optimization feedback signals in the calibration process. This device, using a camera, can track more than 50 specific facial muscle movements and transmits data in real time over the network via the LiveLink protocol. It supports multicast networking and timecode synchronization with tools such as Unreal Engine, ensuring low latency and high accuracy in data transmission.

[0083] When using the device, pay attention to device compatibility and ensure that the device and the robot control terminal are on the same network (wired or high-quality Wi-Fi preferred) to avoid tracking errors caused by poor lighting conditions such as backlighting or direct strong light.

[0084] Step 2: Partition order determination;

[0085] A "functional partitioning and sequential execution" strategy was adopted to calibrate the motion units of eyebrows, eyelids, jaw, lips, and corners of the mouth in turn. Among them, the mouth area required a strict mechanical decoupling sequence: first, the opening and closing motion of the jaw was calibrated; then, with the jaw fixed in the closed position, the independent motions of the upper and lower lips were calibrated; finally, with the jaw and lips fixed in the reference position, the horizontal and vertical motions of the corners of the mouth were calibrated, effectively isolating the motion coupling between the degrees of freedom.

[0086] Step 3: Optimization of combining coarse and fine granularity;

[0087] For each servo or servo group to be calibrated, a two-stage optimization strategy of "global coarse search and local fine-tuning" is adopted. The first stage traverses the entire working range with a step size of 10%-20% to quickly locate the driving value range where the target blendshape response is close to the expected value. The second stage performs a fine search within the optimal range with a step size of 1%-5% to find the precise driving value that minimizes the error of the target blendshape response.

[0088] To improve the reliability of calibration data, noise reduction processing can be performed on the collected Blendshape data. A moving average algorithm (taking the average value of multiple consecutive frames) can be used to filter high-frequency noise, or Kalman filtering can be used to suppress noise and complete data, avoiding calibration deviations caused by jitter in the original data.

[0089] Step 4: Quantify target setting;

[0090] This step involves calibration in the following order: eyebrows, eyelids, jawline, lips, and corners of the mouth. The details are as follows:

[0091] Eyebrow area

[0092] Neutral Position: Simultaneously traverse the EyebrowLeftOuter servo (linked with the EyebrowRightOuter servo) and the EyebrowLeftInner servo (linked with the EyebrowRightInner servo), searching within a reasonable range with a relatively large step size (e.g., 10%). The optimization goal is to minimize the sum of the BrowDownLeft(Right) coefficient, BrowInnerUp coefficient, and BrowOuterUpLeft(Right) coefficient measured by ARKit. After finding the minimum value, record the value of the two servos at this point as the neutral value.

[0093] Downward limit: Starting from the neutral value, simultaneously decrease the values ​​of the two linked servo motors in small steps (e.g., 2%). When the BrowDownLeft(Right) coefficient value stabilizes at around 0.2, record the minimum value of the two servo motors at this point.

[0094] The upper limit is reached by fixing the EyebrowInnerLeft(Right) servo at a neutral value, iterating through the EyebrowOuterLeft(Right) servos until the BrowOuterUpLeft(Right) coefficient reaches 0.2, and recording this as the maximum value of the EyebrowOuterLeft(Right) servo. Similarly, fixing the EyebrowOuterLeft(Right) servo at a neutral value, iterating through the EyebrowInnerLeft(Right) servos until the BrowInnerUp coefficient reaches 0.2, and recording this as the maximum value of the EyebrowInnerLeft(Right) servo.

[0095] eyelid area

[0096] Upper eyelid neutral value: Traverse the UpperEyelidLeft(Right) servos, find the servo position with the smallest sum of EyeWideLeft(Right) coefficient and EyeBlinkLeft(Right) coefficient, and record it as the neutral value of the UpperEyelidLeft(Right) servo.

[0097] Upper eyelid limit: Starting from the neutral value, increase the servo value to make the EyeWideLeft(Right) coefficient reach about 0.25, which is recorded as the maximum value of the UpperEyelidLeft(Right) servo; decrease the servo value to make the EyeBlinkLeft(Right) coefficient reach about 0.25, which is recorded as the minimum value of the UpperEyelidLeft(Right) servo.

[0098] Lower eyelid limit: Iterate through the LowerEyelidLeft(Right) servo. Increase its value until the EyeSquintLeft(Right) coefficient reaches approximately 0.1, which is recorded as the maximum value of the LowerEyelidLeft(Right) servo; decrease its value until the EyeSquintLeft(Right) coefficient drops to approximately 0.05, which is recorded as the minimum value of the LowerEyelidLeft(Right) servo.

[0099] mandibular region

[0100] Iterate through the Jaw servos. When the JawOpen coefficient approaches 0, record it as the minimum value of the Jaw servo; when the JawOpen coefficient approaches 0.3, record it as the maximum value of the Jaw servo.

[0101] Lip area (must be performed when Jaw servo is at its minimum value):

[0102] Upper lip calibration: Iterate through the UpperLipLeft(Right) servos. Record the minimum servo value when the MouthUpperUpLeft(Right) coefficient is 0, and record the maximum servo value when the MouthUpperUpLeft(Right) coefficient approaches 0.1.

[0103] Lower lip calibration: Iterate through the LowerLipLeft(Right) servos. Record the minimum servo value when the MouthLowerDownLeft(Right) coefficient is 0; record the maximum servo value when the MouthLowerDownLeft(Right) coefficient approaches 0.1.

[0104] Mouth area (must be performed when Jaw servo, Upper LipLeft (Right) servo, and Lower LipLeft (Right) servo are all at their minimum values):

[0105] Neutral position: Collaboratively traverse the two sets of servos, LeftCornerX (linked with RightCornerX) and LeftCornerY (linked with RightCornerY), to find the position where the sum of MouthSmileLeft(Right) coefficient and MouthFrownLeft(Right) coefficient is minimized, and record the neutral value of the two sets of servos at this time.

[0106] Smile Limit: Starting from the neutral value, simultaneously increase the values ​​of the Left (Right) CornerX and Left (Right) CornerY servo groups until the MouthSmileLeft (Right) coefficient reaches 0.3, and record it as the maximum value of the servo.

[0107] Mouth Frown Left (Right) Limit: Starting from the neutral value, simultaneously decrease the values ​​of the Left (Right) CornerX and Left (Right) CornerY servo groups until the MouthFrownLeft (Right) coefficient reaches 0.1, which is recorded as the minimum value of the servo.

[0108] By calibrating in the above order, the precise drive values ​​of each servo motor under neutral, extreme, and key facial expressions are obtained. These drive values ​​are written into the robot's non-volatile memory for subsequent use by the facial expression mapping model, ensuring long-term repeatability accuracy.

[0109] Example 2

[0110] This invention provides a humanoid robot that includes a facial expression mapping system and an automatic facial expression calibration system using the methods described in the above embodiments. Through the collaborative work of the two systems, it achieves highly natural, highly accurate and long-term stable facial expression output.

[0111] This invention achieves high-fidelity mapping and systematic calibration from semantic expression parameters to multi-servo motor collaborative control through data-driven modeling and visual feedback optimization, effectively improving the naturalness, accuracy and long-term stability of robot facial expressions.

[0112] The method described in this invention has been verified on actual humanoid robot prototypes, significantly improving the expressiveness and reliability of facial expressions. For those skilled in the art, any adaptive modifications and improvements made to the data acquisition method, neural network structure, calibration search algorithm, etc., without departing from the principles of this invention, should be considered to fall within the protection scope of this invention.

Claims

1. A method for mapping and calibrating facial expressions of a humanoid robot, characterized in that, The humanoid robot facial expression mapping method is linked with the calibration method. The humanoid robot facial expression mapping method includes the following steps: S1 collects facial expression data of the robot within its physical reach, generates paired data of servo control signals and Blendshape parameters through random motor excitation, and obtains a self-supervised discrete expression dataset. S2 trains a multilayer perceptron (MLP) model based on a self-supervised discrete facial expression dataset to obtain a preliminary mapping from the Blendshape parameters to the servo control signal; S3 extracts the Blendshape parameters from the real human facial expression dataset, generates robot servo control signals through the multilayer perceptron (MLP) model, collects the actual Blendshape sequence generated by the robot, and obtains a self-supervised temporal facial expression dataset. S4 trains a Long Short-Term Memory (LSTM) network model based on a self-supervised temporal facial expression dataset. The LSTM network model adopts a structure that combines a temporal feature extraction module and a nonlinear regression module. After training, it obtains the temporal mapping of continuous facial expressions. The method for facial expression calibration of humanoid robots includes the following steps: S5 uses a high-precision facial motion capture device to collect the Blendshape coefficients of the robot's face in real time based on time-series mapping, and obtains the optimization target feedback signal for the calibration process. S6 calibrates the servo motors in the order of facial regions: eyebrows, eyelids, jaw, lips, and corners of the mouth. The calibration order for the mouth region is jaw, lips, and corners of the mouth, in order to obtain a motion-decoupled calibration. S7 employs a search strategy that combines coarse and fine granularity to progressively optimize the servo control signal and obtain a Blendshape response that approaches the target value. S8 sets corresponding Blendshape response targets for different facial motion servos, calibrates motion range nodes in the eyebrow unit, eyelid unit, jaw unit, lip unit, and corner of mouth unit, and coordinates the adjustment of multiple servos to achieve calibration of complex expressions.

2. The method for mapping and calibrating facial expressions of a humanoid robot according to claim 1, characterized in that, The process involves collecting facial expression data from the robot itself within its physical reach, generating paired data of servo control signals and Blendshape parameters through random motor excitation, and obtaining a self-supervised discrete facial expression dataset, including: S11 generates multiple sets of servo drive signal combinations randomly through a random motor excitation program, under the premise of ensuring mechanical safety. Each signal combination is a drive signal vector containing the normalized position values ​​of all servos to be controlled in the range of 0-1. S12 synchronously injects and enforces preset hardware safety constraints and symmetry constraints during the signal combination generation process to exclude combinations of mechanical damage and non-human facial expression signals; S13 sends the obtained driving signal vector to the robot. After the robot's motion stabilizes, it collects the response vector of the Blendshape coefficient corresponding to the robot's face. The driving signal vector and the Blendshape response vector are then paired and stored. This process is repeated until enough data samples covering the robot's expression space are collected to form a self-supervised discrete expression dataset.

3. The method for mapping and calibrating facial expressions of a humanoid robot according to claim 1, characterized in that, This study trains a multilayer perceptron (MLP) model based on a self-supervised discrete facial expression dataset to achieve a preliminary mapping from blendshape parameters to servo control signals. The process involves training the MLP model with a 52-dimensional blendshape coefficient input layer, an output layer dimension consistent with the number of servo motors on the robot's face, and 2-3 hidden layers with 256-512 neurons per layer. The ReLU activation function is used, and the mean squared error (MSE) is used as the loss function. The Adam optimizer is employed for training to learn the complex nonlinear relationship between the single-frame blendshape coefficient vector and the single-frame normalized servo drive vector, thus obtaining a mapping model from single-frame static facial expression parameters to single-frame servo control signals.

4. The method for mapping and calibrating facial expressions of a humanoid robot according to claim 1, characterized in that, Extract the Blendshape parameters from the real-person facial expression dataset, generate robot servo control signals through the Multilayer Perceptron (MLP) model, collect the actual Blendshape sequences generated by the robot, and obtain a self-supervised temporal facial expression dataset, including: S31 extracts a sequence of Blendshape coefficients from a publicly available dataset of real-person facial expression audio and video recordings, representing a continuous speech sequence. , convert the sequence The frames are fed into the trained multilayer perceptron (MLP) model to obtain the corresponding predicted servo control signal sequence. ; S32 will sequence The data is sent to the robot for execution, and the facial blendshape response is collected when the robot executes the sequence to obtain the actual blendshape parameter sequence. Obtain a self-supervised temporal facial expression dataset .

5. The method for mapping and calibrating facial expressions of a humanoid robot according to claim 1, characterized in that, A Long Short-Term Memory (LSTM) network model is trained based on a self-supervised temporal facial expression dataset. The LSTM model employs a structure combining a temporal feature extraction module and a nonlinear regression module. After training, it obtains the temporal mapping of continuous facial expressions. The model includes: a temporal feature extraction module consisting of two unidirectional LSTM layers with a hidden state dimension of 256 and an inter-layer dropout rate of 0.05-0.2, used to extract high-level temporal features from the input sequence; and a nonlinear regression module, a two-layer fully connected network with residual connections, which integrates the LSTM data with the temporal features. The hidden states of the network (LSTM) are mapped to the servo control signal space, and finally the normalized driving values ​​are output through the Sigmoid layer. The training adopts a sliding window strategy combined with the strategy of passing the hidden states across windows. The window length is 20-40 frames and the step size is 5-15 frames. Within the same sentence, the hidden states and cell states of the Long Short-Term Memory Network (LSTM) are passed to the next window in consecutive windows, and the state is reset only at the sentence boundary. After training, a Blendshape sequence of a target expression is input, and the model directly outputs a smooth and temporally coordinated servo control sequence to obtain the temporal mapping of consecutive expressions.

6. A method for mapping and calibrating facial expressions of a humanoid robot, characterized in that, A search strategy combining coarse-grained and fine-grained methods is employed to progressively optimize the servo control signal, obtaining calibrated drive values ​​that approximate the Blendshape response, including: For each servo or servo group to be calibrated, the coarse search stage traverses the entire working range in steps of 10%–20% to quickly locate the driving value range that makes the target blendshape response close to the expected value. The fine-tuning stage searches within the optimal range in steps of 1%–5% to find the driving value that minimizes the target blendshape response error.

7. The method for mapping and calibrating facial expressions of a humanoid robot according to claim 6, characterized in that, The collected Blendshape data is denoised, and the calibrated servo drive values ​​are written into the robot's non-volatile memory for use in the expression mapping model.

8. A humanoid robot, characterized in that, The system includes a facial expression mapping system and a facial expression calibration system. The facial expression mapping system uses the mapping method described in any one of claims 1-7, and the facial expression calibration system uses the calibration method described in any one of claims 1-7.

Images