Bionic expression control system and method based on 3D self-supervised learning

Abstract

The invention discloses a bionic expression control system based on 3D self-supervised learning, which comprises an extended expression standard module, a 3D physical sensing module, a forward physical model, a reverse control model and a self-supervised training framework, and relates to the technical field of artificial intelligence. According to the bionic expression control system and method based on 3D self-supervised learning, through 3D physical feedback and self-supervised training of a bidirectional model, the system can autonomously learn a driving rule and a control strategy without manual labeling, the development cost is greatly reduced, a forward physical model accurately models dynamic characteristics of a driving unit, and the driving efficiency is improved. The reverse control model learns the optimal strategy, the control precision is remarkably improved through bidirectional self-supervised learning and the physical twinning technology, compared with a traditional method, the method has obvious advantages, physical rules are learned by the system through the physical twinning technology instead of simple mapping, and good generalization ability is achieved for unseen expression combinations.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a biomimetic facial expression control system and method based on 3D self-supervised learning. Background Technology

[0002] Artificial intelligence (AI) is a transformative technology that enables machines to solve problems just like humans, from recognizing images and generating creative content to making data-driven predictions. AI empowers businesses to make smarter decisions at scale. In today's digital environment, organizations generate massive amounts of data through sensors, user interactions, and system logs. AI uses this data to optimize operations, including automating customer support, enhancing marketing strategies, and providing actionable insights through advanced analytics. With AWS, businesses can seamlessly integrate AI to accelerate innovation, optimize customer experiences, and solve complex problems. AWS's AI solutions help businesses achieve personalized interactions, automated decision-making, and unlock new growth opportunities in the ever-changing digital world—while also benefiting from AWS's commitment to privacy, security, and responsible AI.

[0003] Existing facial expression robots mainly employ rule-based control methods or simple mapping relationships, which have the following problems: Low accuracy: It cannot accurately model the nonlinear characteristics and physical constraints of the driving unit; Poor generalization: It does not perform well in controlling unfamiliar combinations of facial expressions; Unable to adapt: ​​Material aging and changes in temperature and humidity lead to a decline in control performance; Some studies employ supervised learning methods, training the model using manually labeled "facial expression parameters - driving signals," but this method: High annotation costs: It requires a large amount of manual annotation and is difficult to cover all expression combinations; Hardware dependency: The labeled data is bound to specific hardware and cannot be migrated; Unable to handle dynamic changes: Unable to adapt to long-term changes in material properties; Existing facial expression standards are primarily designed for virtual characters; when applied to physical robots: Insufficient dimensions: lack of key micro-expressions; Inconsistent driving methods: Some parameters are bidirectional, which is not suitable for unidirectional physical actuators. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a biomimetic facial expression control system and method based on 3D self-supervised learning, which solves the problems mentioned in the background.

[0005] To achieve the above objectives, this invention provides the following technical solution: a bionic facial expression control system based on 3D self-supervised learning, comprising an extended expression standard module, a 3D physical perception module, a forward physics model, an inverse control model, and a self-supervised training framework. The extended expression standard module includes basic facial expression parameters, eye movements, neck movements, micro-expressions, etc., all parameters are normalized to [0,1] mapping values ​​on the control interface, and unidirectional or bidirectional physical actions are achieved through device mapping functions. The 3D physical perception module employs a facial capture device based on depth cameras or structured light technology, enabling real-time... The system acquires 400-500 facial 3D key points and multi-dimensional expression parameters as physical ground truth feedback. The forward physical model is based on a recurrent neural network, establishing a forward mapping from the driving signal sequence to the expression parameters and learning the dynamic response characteristics of the driving unit, including hysteresis, nonlinearity, and coupling effect. The inverse control model is based on a recurrent neural network, establishing an inverse mapping from the target expression parameter sequence to the driving signal and learning the optimal control strategy. The self-supervised training framework jointly trains the forward and inverse models through physical consistency loss, recurrent consistency loss, smoothness constraints, and physical constraints, without the need for manual annotation.

[0006] Preferably, the system supports hybrid drive of servo motors and electronic muscles, and intelligently allocates drive tasks according to the frequency characteristics and accuracy requirements of facial expression parameters. High-frequency, high-precision tasks are assigned to servo motors, while low-frequency, continuous tasks are assigned to electronic muscles.

[0007] Preferably, the loss function of the self-supervised training framework is: L_total=α·L_forward+β·L_cycle+γ·L_smooth+δ·L_physics, where each weight coefficient can be adjusted according to the actual situation, preferably α=0.8-1.2, β=0.3-0.7, γ=0.05-0.2, δ=0.1-0.3.

[0008] Preferably, the 3D physical perception module uses a high-precision 3D facial capture device to acquire the 3D coordinates and multi-dimensional expression parameters of 400-500 facial key points in real time as physical truth feedback.

[0009] Preferably, all parameters of the extended expression standard module are normalized to [0,1] mapping values ​​on the control interface, and unidirectional or bidirectional physical movements are realized through the device mapping function; paired muscles are controlled independently on the left and right sides.

[0010] Preferably, the self-supervised training framework achieves unsupervised learning through the following means: the difference between the facial expression parameters predicted by the forward model and the real facial expression parameters captured by 3D; the driving signal output by the inverse model should restore the original facial expression after being predicted by the forward model; the temporal smoothness and spatial continuity of the driving signal; and the range limitation, speed limitation, and energy minimization of the driving signal.

[0011] Preferably, the control method of the bionic facial expression control system based on 3D self-supervised learning specifically includes the following steps: S1. Self-supervised data acquisition: Generates exploratory driving signal sequences, executes them, and records real facial expression feedback through a 3D capture device. Data acquisition is conducted with the explicit consent of the user and in compliance with applicable laws and regulations. S2. Bidirectional model joint training: Simultaneously train the forward physics model and the inverse control model, and jointly optimize them through physical consistency loss and cycle consistency loss; S3. Real-time facial expression control: Receives the target facial expression sequence, the inverse model calculates the optimal driving signal and executes it, the 3D capture device acquires the actual facial expression, and the forward model predicts the facial expression parameters and calculates the prediction error. S4. Online adaptive update: When the cumulative prediction error exceeds the threshold, an update is triggered. The model is fine-tuned on an independent thread or replica. After completion, the main model is updated through weight hot replacement or moving average update strategy to adapt to material aging and environmental changes. Beneficial effects

[0012] This invention provides a biomimetic facial expression control system and method based on 3D self-supervised learning. Compared with existing technologies, it has the following advantages: This biomimetic facial expression control system and method based on 3D self-supervised learning expands the facial expression standard module to include basic facial expression parameters, eye movements, neck movements, micro-expressions, etc. All parameters are normalized to [0,1] mapping values ​​on the control interface. Unidirectional or bidirectional physical actions are realized through device mapping functions. The 3D physical perception module uses a facial capture device based on depth camera or structured light technology to acquire 400-500 facial 3D key points and multidimensional expression parameters in real time as physical truth feedback. The forward physical model is based on a recurrent neural network, establishing a forward mapping from the driving signal sequence to expression parameters. It learns the dynamic response characteristics of the driving unit, including hysteresis, nonlinearity, and coupling effects. Through 3D physical feedback and the self-supervised learning of the bidirectional model, the system achieves the desired physical effects. Supervised training enables the system to autonomously learn driving laws and control strategies without manual annotation, significantly reducing development costs. The forward physical model accurately models the dynamic characteristics of the driving unit, while the inverse control model learns the optimal strategy. Through bidirectional self-supervised learning and physical twin technology, control accuracy is significantly improved, showing clear advantages over traditional methods. Through physical twin technology, the system learns physical laws rather than simple mappings, exhibiting good generalization ability for unseen facial expression combinations. The online learning module can adapt to long-term changes such as material aging, temperature and humidity variations, and mechanical wear without manual recalibration. The unified framework supports hybrid drive of servos and eMuscles, intelligently allocating control tasks according to their respective characteristics, fully leveraging the advantages of fast servo response and smooth motion of eMuscles. Attached Figure Description

[0013] Figure 1 This is a diagram of the overall system architecture of the present invention; Figure 2 This is a structural principle block diagram of the system of the present invention; Figure 3 This is a flowchart of the self-supervised training process of the present invention; Figure 4 This is a flowchart of the 3D capture process of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Please see Figure 1-4 This invention provides a technical solution: a biomimetic expression control system based on 3D self-supervised learning, comprising an extended expression standard module, a 3D physical perception module, a forward physical model, an inverse control model, and a self-supervised training framework. The extended expression standard module includes basic facial expression parameters, eye movements, neck movements, micro-expressions, etc., and all parameters are normalized to [0,1] mapping values ​​on the control interface. Unidirectional or bidirectional physical actions are realized through device mapping functions. The 3D physical perception module uses a facial capture device based on depth camera or structured light technology to acquire 400-500 facial 3D key points and multidimensional expression parameters in real time as physical truth feedback. The forward physical model is based on a recurrent neural network, establishing a forward mapping from the driving signal sequence to expression parameters, and learning the dynamic response characteristics of the driving unit, including hysteresis, nonlinearity, and coupling effect. The inverse control model is based on a recurrent neural network, establishing an inverse mapping from the target expression parameter sequence to the driving signal, and learning the optimal control strategy. The self-supervised training framework jointly trains the forward and inverse models through physical consistency loss, recurrent consistency loss, smoothness constraints, and physical constraints, without the need for manual annotation.

[0016] In this invention, the system supports hybrid drive of servo motors and electronic muscles. Drive tasks are intelligently allocated according to the frequency characteristics and accuracy requirements of facial expression parameters. High-frequency, high-precision tasks are assigned to servo motors, while low-frequency, continuous tasks are assigned to electronic muscles.

[0017] In this invention, the loss function of the self-supervised training framework is: L_total=α·L_forward+β·L_cycle+γ·L_smooth+δ·L_physics, where each weight coefficient can be adjusted according to the actual situation, preferably α=0.8-1.2, β=0.3-0.7, γ=0.05-0.2, δ=0.1-0.3.

[0018] In this invention, the 3D physical perception module uses a high-precision 3D facial capture device to acquire the 3D coordinates and multi-dimensional expression parameters of 400-500 facial key points in real time as physical truth feedback.

[0019] In this invention, all parameters of the extended facial expression standard module are normalized to [0,1] mapping values ​​on the control interface, and unidirectional or bidirectional physical movements are realized through the device mapping function; paired muscles are controlled independently on the left and right sides.

[0020] In this invention, the self-supervised training framework achieves unsupervised learning through the following means: the difference between the facial expression parameters predicted by the forward model and the real facial expression parameters captured by 3D; the driving signal output by the inverse model should restore the original facial expression after being predicted by the forward model; the temporal smoothness and spatial continuity of the driving signal; and the range limitation, speed limitation, and energy minimization of the driving signal.

[0021] This invention also discloses a control method for a biomimetic facial expression control system based on 3D self-supervised learning, specifically including the following steps: S1. Self-supervised data acquisition: Generates exploratory driving signal sequences, executes them, and records real facial expression feedback through a 3D capture device. Data acquisition is conducted with the explicit consent of the user and in compliance with applicable laws and regulations. S2. Bidirectional model joint training: Simultaneously train the forward physics model and the inverse control model, and jointly optimize them through physical consistency loss and cycle consistency loss; S3. Real-time facial expression control: Receives the target facial expression sequence, the inverse model calculates the optimal driving signal and executes it, the 3D capture device acquires the actual facial expression, and the forward model predicts the facial expression parameters and calculates the prediction error. S4. Online adaptive update: When the cumulative prediction error exceeds the threshold, an update is triggered. The model is fine-tuned on an independent thread or replica. After completion, the main model is updated through weight hot replacement or moving average update strategy to adapt to material aging and environmental changes.

[0022] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0023] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0024] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A biomimetic expression control system based on 3D self-supervised learning, comprising an extended expression standard module, a 3D physical perception module, a forward physical model, an inverse control model and a self-supervised training framework, characterized in that: The extended expression standard module includes facial expression basic parameters, eye movement, neck movement, micro-expression, etc. All parameters are normalized to [0, 1] mapping values on the control interface, and one-way or two-way physical actions are realized through a device mapping function. The 3D physical perception module adopts a face capture device based on a depth camera or a structured light technology to obtain 400-500 facial 3D key points and multi-dimensional expression parameters as physical true value feedback in real time. The forward physical model is based on a recurrent neural network to establish a forward mapping from a driving signal sequence to expression parameters and learn dynamic response characteristics of a driving unit including hysteresis, nonlinearity and coupling effect. The inverse control model is based on a recurrent neural network to establish an inverse mapping from a target expression parameter sequence to a driving signal and learn an optimal control strategy. The self-supervised training framework jointly trains the forward and inverse models through physical consistency loss, cycle consistency loss, smoothness constraint and physical constraint without manual annotation.

2. The bionic expression control system based on 3D self-supervised learning according to claim 1, characterized in that: The system supports mixed driving of a rudder and an electronic muscle, intelligently allocates driving tasks according to frequency characteristics and accuracy requirements of expression parameters, and allocates high-frequency and high-precision tasks to the rudder and low-frequency continuous tasks to the electronic muscle.

3. The bionic expression control system based on 3D self-supervised learning according to claim 1, characterized in that: The loss function of the self-supervised training framework is L_total=α·L_forward+β·L_cycle+γ·L_smooth+δ·L_physics, wherein each weight coefficient can be adjusted according to actual conditions, and α=0.8-1.2, β=0.3-0.7, γ=0.05-0.2 and δ=0.1-0.3 are preferred.

4. The bionic expression control system based on 3D self-supervised learning according to claim 1, characterized in that: The 3D physical perception module adopts a high-precision 3D face capture device to obtain 3D coordinates of 400-500 facial key points and multi-dimensional expression parameters in real time as physical true value feedback.

5. The 3D self-supervised learning based bionic expression control system according to claim 1, wherein: All parameters of the extended expression standard module are normalized to [0, 1] mapping values on the control interface, and one-way or two-way physical actions are realized through a device mapping function. Paired muscles are independently controlled on the left and right sides.

6. The 3D self-supervised learning based bionic expression control system according to claim 1, wherein: The self-supervised training framework realizes unsupervised learning in the following ways: a difference between expression parameters predicted by the forward model and real expression parameters captured by the 3D device; driving signals output by the inverse model should restore the original expression after being predicted by the forward model; time smoothness and spatial continuity of the driving signals; range limitation, speed limitation and energy minimization of the driving signals.

7. The bionic expression control system based on 3D self-supervised learning according to any one of claims 1-6, wherein the control method specifically comprises the following steps: S1, self-supervised data acquisition: generating an exploratory driving signal sequence, executing and recording real expression feedback through a 3D capture device, and data acquisition being performed on the premise of obtaining explicit consent of a user and complying with applicable laws and regulations; S2, joint training of bidirectional models: simultaneously training a forward physical model and an inverse control model, and jointly optimizing through physical consistency loss and cycle consistency loss; S3, Real-time expression control: receive target expression sequence, inverse model calculate optimal driving signal and execute, 3D capture device get actual expression, forward model predict expression parameters and calculate prediction error; S4, Online adaptive update: trigger update when cumulative prediction error exceeds threshold, fine-tune model on independent thread or copy, update main model through weight hot replacement or sliding average update strategy after completion, adapt to material aging and environmental changes.

Images