A robotic expression mimicking method and system

Abstract

This invention proposes a method and system for robot facial expression imitation. The method includes: establishing a jaw and eye model, constructing a matching facial surface model and dividing it into meshes; selecting n points as nodes to be driven, and setting three shape memory alloy drive rods at each node, one end of which drives the node to move according to facial expression data, and the other end is connected to a human face backplate model, with a controllable current connected to the drive rods; building a database to record the correspondence between facial muscle movements, facial expressions, jaw and eye model states, and drive rod displacements; and driving the nodes and models to move according to the facial expression data and the database to imitate facial expressions. This invention uses an elastic mesh to construct the model and optimizes the node distribution; it sets 13 key regions as nodes to be driven to capture the core features of facial expressions; and it uses nickel-titanium shape memory alloy drive rods to achieve fine multi-directional displacement of nodes, which has the advantages of smaller size, faster response, and smoother movements compared to traditional servo motor drives.

Description

Technical Field

[0001] This invention relates to the field of facial expression imitation technology, and in particular to a method and system for robot facial expression imitation. Background Technology

[0002] Humans express emotions in various ways, such as facial expressions, speech, and body language. Among these, facial expressions contain rich emotional information and are the most direct and important channel for conveying emotions. The renowned American psychologist Mehrabian conducted in-depth research on facial expressions and found that more than half of the information we convey in daily life is through facial expressions. Even the same verbal expression and body language can convey different meanings depending on the facial expression used. However, unlike other forms of emotional expression, the emotional definition of facial expressions is largely a human consensus. Therefore, facial expressions occupy an extremely important part of human emotional expression and have become a major research topic for scholars. Humanoid facial design has demonstrated enormous application potential in multiple fields, including human-computer interaction, film and television production, virtual reality, and medical rehabilitation. In human-computer interaction, the highly realistic rendering of human-like facial expressions can significantly enhance the emotional interaction between humans and machines, making the interaction process more natural and smooth, thereby improving the quality and satisfaction of user experience. In virtual reality scenarios, highly realistic facial images help enhance the user's immersive experience, allowing them to more truly feel as if they are there. In the field of medical rehabilitation, accurate facial models can provide solid technical support for maxillofacial surgery simulation and facial plastic surgery repair effect evaluation, effectively reducing surgical risks and improving treatment effects and rehabilitation quality.

[0003] With the increasing maturity of robotics technology, in order to reduce the mechanical feel of human-computer interaction, improve the naturalness of emotion transmission, and optimize the application benefits of service and medical robots, scholars are focusing on the research of multi-degree-of-freedom humanoid facial emotion structures. The core is to break through expression-driven, mechanical design, and control technologies. In the early stages, with the goal of realizing basic expressions, scholars relied on experience and simple actuation, using a small number of servo motors to control key parts such as eyebrows and jaws, such as the WE-3R-II from Waseda University in Japan. Although it can complete basic expressions, it is limited by experience-based design and low-biomimetic skin, resulting in stiff expressions, limited variety, and susceptibility to interference from lighting and occlusion, making it unable to cope with complex scenarios. Summary of the Invention

[0004] This invention aims to at least solve the technical problems existing in the prior art, and in particular, it innovatively proposes a method and system for robot facial expression imitation.

[0005] To achieve the above-mentioned objectives of the present invention, the present invention provides a method for robot facial expression imitation, the method comprising: S1. Create a mandibular model and an eye model; S2. Construct a facial surface model that matches the jaw model and eye model based on the elastic mesh, and perform mesh generation; S3. Select n points on the facial surface model as nodes to be driven; S4. Three driving rods are set for each of the nodes to be driven. One end of the three driving rods is used to drive the node to be driven to move in different directions according to the facial expression data, and the other end is connected to the face back panel model respectively. The driving rods are made of shape memory alloy and connected to a controllable current. S5. Establish a database of the state of the jaw model and eye model under different facial expressions and the displacement of the drive rod based on the correspondence between facial muscle movements and different expressions; S6. Based on the facial expression data, drive the movement of the node to be driven, the jaw model, and the eye model according to the database to imitate facial expressions.

[0006] As an optional embodiment of the present invention, the shape memory alloy in step S4 may be a nickel-titanium shape memory alloy material.

[0007] As an optional embodiment of the present invention, optionally, constructing a facial surface model matching the jaw model and eye model based on an elastic mesh in step S2 includes: S201. Collect basic three-dimensional contour data of the robot's face, including the initial morphological parameters of the jaw model and eye model; S202. Import the basic three-dimensional contour data into the three-dimensional modeling system to generate the initial facial surface framework; S203. Based on the deformation characteristics of the elastic mesh, the distribution of mesh nodes in the initial curved surface frame is optimized so that the mesh density matches the surface change requirements of different areas of the face, and the mesh edges fit the edges of the jaw model and eye model to obtain the facial curved surface model.

[0008] As an optional embodiment of the present invention, n may be equal to 13 in step S3.

[0009] As an optional embodiment of the present invention, optionally, the nodes to be driven in step S3 include: determining the positions of the nodes to be driven on the facial surface model according to the regional distribution of facial expression movement, wherein: 1 node each in the middle and on both sides of the forehead, 1 node each below the left and right brow bones near the eyelids, 1 node each at the starting point of the apple cheek below the left and right eye sockets, 1 node each at the apple cheek protrusion on the left and right cheeks, 1 node each above the nostrils on both sides of the nose, and 1 node at the tip of the chin.

[0010] As an optional embodiment of the present invention, optionally, the database for establishing the states of the jaw model and eye model under different facial expressions and the displacement of the drive rod in step S5, based on the correspondence between facial muscle movements and different expressions, includes: S501. Collect facial muscle movement data under various typical human facial expressions; S502. Based on the muscle movement data, use a three-dimensional motion capture system to synchronously record the actual displacement of the corresponding nodes to be driven on the face, as well as the real-time state parameters of the jaw model and eye model. S503. Establish a database based on the actual displacement of the node to be driven and the real-time state parameters of the mandibular model and the eye model.

[0011] As an optional embodiment of the present invention, optionally, the expression imitation in step S6, which involves driving the movement of the node to be driven, the jaw model, and the eye model based on the expression data according to the database, includes: S601, Receive the facial expression data to be imitated; S602. Preprocess the facial expression data to extract facial expression feature information, including expression category, intensity and regional distribution of facial muscle movements; S603. Based on the extracted facial expression feature information, perform matching and retrieval in the database to obtain the displacement of the node to be driven, the real-time state parameters of the jaw model and the eye model corresponding to the target facial expression. S604. Facial expression mimicry is performed based on the displacement of the node to be driven, the real-time state parameters of the jaw model and the eye model.

[0012] In another aspect, the present invention also provides a robot facial expression mimicry system, the system comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the robot facial expression mimicry method when executing the executable instructions.

[0013] The beneficial effects of this invention are as follows: This invention constructs a facial surface model that matches the jaw and eye models using an elastic mesh, and optimizes the mesh node distribution to adapt to changing facial expression requirements. Combined with 13 nodes covering key areas such as the forehead, brow bone, cheekbones, nose, and chin, it accurately captures the core features of facial expression movements. Using a nickel-titanium shape memory alloy drive rod, leveraging its shape memory effect and controllable current adjustment characteristics, it achieves multi-directional, delicate displacement of the driven nodes. Compared to traditional servo motor drives, it has advantages such as small size, fast response, and smooth movement. By establishing a database of typical human expressions and drive parameters, it achieves rapid matching and precise driving of expression data, significantly improving the realism and naturalness of robot expression imitation and effectively reducing the mechanical feel of emotions in human-computer interaction. This invention provides a practical technical path for the design of humanoid robot expression systems and can be widely applied in fields such as human-computer interaction, virtual reality, and medical rehabilitation, helping to improve user experience and technical support capabilities in related scenarios.

[0014] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0015] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a robot facial expression imitation method according to the present invention; Figure 2 This is a schematic diagram of the structure of the mandible model and eye model of the present invention; Figure 3 This is a schematic diagram of the structure of the mandible model and eye model of the present invention; Figure 4 This is a schematic diagram of the small deformation region model structure of the present invention; Figure 5 This is a schematic diagram of the small deformation region model structure of the present invention; Figure 6 This is a schematic diagram of the installation structure of the drive rod of the present invention; Figure 7 This is a schematic diagram of the installation structure of the drive rod of the present invention; Figure 8 This is a schematic diagram of the structure of the present invention, which adds corresponding forces to the outer corner nodes of the face elastic mesh at the corner of the eye and the corner nodes of the mouth. Figure 9 This is a schematic diagram of the total deformation of the deformable part of the model obtained by solving the present invention; Figure 10 This is a diagram of a human face model after establishing a spatial coordinate system according to the present invention; Figure 11 This is a comparison chart of the mean values ​​of each frame of the data after interpolation of the symmetrical nodes of the "happy" expression of the present invention; Figure 12 This is a scatter plot of the key nodes of the "happy" expression in this invention; Figure 13 This is the x-axis feature point fitting data for the "happy" expression of the present invention. Detailed Implementation

[0016] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0017] Example 1 like Figure 1As shown, a method for robot facial expression imitation includes: S1. Establish the mandibular model and eye model (the construction methods for the mandibular model and eye model are existing technologies and will not be described in detail here). like Figure 2 and 3 As shown, this is a basic model for constructing facial features with significant deformation during facial expressions. Based on the analysis of facial movement mechanisms, areas of the face that deform considerably during expression changes are identified, such as the area around the eyes and mouth. The basic geometric models of these areas are constructed using the 3D modeling software Solidworks. These models primarily include a mouth that can open and close (the jaw model); an eye socket model that can blink; and an eyeball model that can move up, down, left, and right. The eye socket and eyeball models are collectively referred to as the eye model. The model of the areas with significant deformation is shown below. Figure 2 and 3 As shown.

[0018] The mandible model includes upper and lower lip modules. The upper lip module is fixed to the base model via two long, column-type support bases, using threaded fastening. A servo motor mounting device is located between the upper and lower lip modules, with two servo motors mounted symmetrically on the left and right, their output axes pointing outwards. The servo motor mounting device is fixed to the upper lip module via a flange-type mounting shaft and threaded fastening, and to the base model via a long, column-type support base. The lower lip module is connected to the servo motor mounting device via a rotating shaft, and to the servo motor output shaft via a crank-rocker. This mandible model allows the mouth model to open and close by rotating the servo motors on both sides simultaneously.

[0019] The eye model stands atop a composite mounting base, which is secured to the upper lip module via four short column supports and threaded connections. The composite mounting base contains two protruding brackets and multiple threaded holes to provide positioning, support, and fixation for the servo motor mount. The servo motor is threadedly fixed to the servo motor mount. Two eyeball supports, including the lower eye socket, are symmetrically fixed to the front of the composite mounting base. The upper eye socket can rotate around its support frame, and is fixed to the composite mounting base via a rotating support located inside the eye socket. The upper eye socket is connected to the servo motor output shaft via a crank-rocker mechanism. Rotation of the servo motor drives the crank, which in turn drives the connecting rod and the upper eye socket connected via a revolute joint, thus opening and closing the eye socket model. The eyeball model is bolted to the eye socket support frame, and the servo motor is mounted on the support frame with its output shaft pointing vertically downwards. The servo motor and the eyeball are connected via a crank-rocker mechanism. When the servo motor's output shaft rotates, the crank and connecting rod are driven to rotate the eyeball left and right. The upward and downward rotation of the eyeball mechanism is achieved by a crank-connecting rod mechanism connected to another servo motor, which drives the rotation of the eye socket support frame, thus enabling the eyeball to look up and down. The eye socket model can open and close to simulate human eye opening and closing, while the eyeball model can achieve combined movements of looking up, down, left, and right.

[0020] S2. Construct a facial surface model that matches the jaw model and eye model based on the elastic mesh, and perform mesh generation; S3. Select n points on the facial surface model as nodes to be driven; S4. Three driving rods are set for each of the nodes to be driven. One end of the three driving rods is used to drive the node to be driven to move in different directions according to the facial expression data, and the other end is connected to the face back panel model respectively. The driving rods are made of shape memory alloy and connected to a controllable current. S5. Establish a database of the state of the jaw model and eye model under different facial expressions and the displacement of the drive rod based on the correspondence between facial muscle movements and different expressions; S6. Based on the facial expression data, drive the movement of the node to be driven, the jaw model, and the eye model according to the database to imitate facial expressions.

[0021] The principle of the robot facial expression mimicry method in this embodiment is as follows: First, a jaw model and an eye model are constructed using 3D modeling software. The jaw model uses servo motors to drive the opening and closing of the mouth, while the eye model uses servo motors and crank-rocker mechanisms to control the opening and closing of the eye sockets and the rotation of the eyeballs in all directions, thus simulating the basic movements of areas with large facial deformations. Next, based on elastic mesh technology, basic 3D contour data of the robot's face is collected and imported into the 3D modeling system to generate an initial curved surface framework. Then, the distribution of mesh nodes is optimized according to the deformation characteristics of the elastic mesh to match the mesh density with the facial expression changes in different areas, while ensuring that the mesh edges fit the edges of the jaw and eye models, thereby constructing a facial curved surface model that matches both and completing the mesh division. Subsequently, 13 key nodes to be driven are selected on this facial curved surface model. These nodes are distributed in areas such as the middle and sides of the forehead, below the left and right brow bones near the eyelids, below the left and right eye sockets at the origin of the apple cheeks, the apple cheek bulges on the left and right cheeks, above the sides of the nose and the tip of the chin. Three drive rods made of nickel-titanium shape memory alloy are set at each node to be driven. One end of the drive rod is used to drive the node to move in different directions according to the expression data, and the other end is connected to the face backplate model and connected to a controllable current. The shape memory effect and controllable current adjustment characteristics of the shape memory alloy are used to achieve delicate displacement of the node. Then, facial muscle movement data under various typical human expressions are collected. The actual displacement of the corresponding nodes to be driven on the face and the real-time state parameters of the jaw and eye models are recorded simultaneously using a 3D motion capture system to establish a database for different expressions. Finally, the expression data to be imitated is received, preprocessed to extract expression feature information, and based on this feature information, the corresponding drive parameters are matched and retrieved from the database, thereby driving the nodes to be driven, the jaw model and the eye model to move, so as to achieve accurate imitation of the target expression by the robot. This method effectively improves the realism and naturalness of robot expression imitation by optimizing model construction, node selection and driving method, combined with database matching mechanism.

[0022] As an optional embodiment of the present invention, the shape memory alloy in step S4 may be a nickel-titanium shape memory alloy material.

[0023] As an optional embodiment of the present invention, optionally, constructing a facial surface model matching the jaw model and eye model based on an elastic mesh in step S2 includes: S201. Collect basic three-dimensional contour data of the robot's face, including the initial morphological parameters of the jaw model and eye model; S202. Import the basic three-dimensional contour data into the three-dimensional modeling system to generate the initial facial surface framework; S203. Based on the deformation characteristics of the elastic mesh, the distribution of mesh nodes in the initial curved surface frame is optimized so that the mesh density matches the surface change requirements of different areas of the face, and the mesh edges fit the edges of the jaw model and eye model to obtain the facial curved surface model.

[0024] As an optional embodiment of the present invention, n may be equal to 13 in step S3.

[0025] As an optional embodiment of the present invention, optionally, the nodes to be driven in step S3 include: determining the positions of the nodes to be driven on the facial surface model according to the regional distribution of facial expression movement, wherein: 1 node each in the middle and on both sides of the forehead, 1 node each below the left and right brow bones near the eyelids, 1 node each at the starting point of the apple cheek below the left and right eye sockets, 1 node each at the apple cheek protrusion on the left and right cheeks, 1 node each above the nostrils on both sides of the nose, and 1 node at the tip of the chin.

[0026] As an optional embodiment of the present invention, optionally, the database for establishing the states of the jaw model and eye model under different facial expressions and the displacement of the drive rod in step S5, based on the correspondence between facial muscle movements and different expressions, includes: S501. Collect facial muscle movement data under various typical human facial expressions; S502. Based on the muscle movement data, use a three-dimensional motion capture system to synchronously record the actual displacement of the corresponding nodes to be driven on the face, as well as the real-time state parameters of the jaw model and eye model. S503. Establish a database based on the actual displacement of the node to be driven and the real-time state parameters of the mandibular model and the eye model.

[0027] As an optional embodiment of the present invention, optionally, the expression imitation in step S6, which involves driving the movement of the node to be driven, the jaw model, and the eye model based on the expression data according to the database, includes: S601, Receive the facial expression data to be imitated; S602. Preprocess the facial expression data to extract facial expression feature information, including expression category, intensity and regional distribution of facial muscle movements; S603. Based on the extracted facial expression feature information, perform matching and retrieval in the database to obtain the displacement of the node to be driven, the real-time state parameters of the jaw model and the eye model corresponding to the target facial expression. S604. Facial expression mimicry is performed based on the displacement of the node to be driven, the real-time state parameters of the jaw model and the eye model.

[0028] The purpose of this invention is to provide a method for mimicking robot facial expressions, including modeling the large deformation region of facial expressions and the small deformation region of facial expressions of a humanoid robot, and a method for controlling the reproduction of facial expressions by utilizing the properties of shape memory alloys.

[0029] 1. Modeling of large deformation regions To construct basic models of areas with significant facial deformation during facial expressions, the analysis of facial movement mechanisms identifies areas of substantial deformation, such as the area around the eyes and mouth. The basic geometric models of these areas are constructed using the 3D modeling software Solidworks. These models primarily include a mouth that can open and close (the jaw model), an eye socket model that can blink, and eyeball models that can move up, down, left, and right. The eye socket and eyeball models are collectively referred to as the eye models. The models of these large deformation areas are shown below. Figure 2 and 3 As shown.

[0030] The mandible model includes upper and lower lip modules. The upper lip module is fixed to the base model via two long, column-type support bases, using threaded fastening. A servo motor mounting device is located between the upper and lower lip modules, with two servo motors mounted symmetrically on the left and right, their output axes pointing outwards. The servo motor mounting device is fixed to the upper lip module via a flange-type mounting shaft and threaded fastening, and to the base model via a long, column-type support base. The lower lip module is connected to the servo motor mounting device via a rotating shaft, and to the servo motor output shaft via a crank-rocker. This mandible model allows the mouth model to open and close by rotating the servo motors on both sides simultaneously.

[0031] The eye model stands atop a composite mounting base, which is secured to the upper lip module via four short column supports and threaded connections. The composite mounting base contains two protruding brackets and multiple threaded holes to provide positioning, support, and fixation for the servo motor mount. The servo motor is threadedly fixed to the servo motor mount. Two eyeball supports, including the lower eye socket, are symmetrically fixed to the front of the composite mounting base. The upper eye socket can rotate around its support frame, and is fixed to the composite mounting base via a rotating support located inside the eye socket. The upper eye socket is connected to the servo motor output shaft via a crank-rocker mechanism. Rotation of the servo motor drives the crank, which in turn drives the connecting rod and the upper eye socket connected via a revolute joint, thus opening and closing the eye socket model. The eyeball model is bolted to the eye socket support frame, and the servo motor is mounted on the support frame with its output shaft pointing vertically downwards. The servo motor and the eyeball are connected via a crank-rocker mechanism. When the servo motor's output shaft rotates, the crank and connecting rod are driven to rotate the eyeball left and right. The upward and downward rotation of the eyeball mechanism is achieved by a crank-connecting rod mechanism connected to another servo motor, which drives the rotation of the eye socket support frame, thus enabling the eyeball to look up and down. The eye socket model can open and close to simulate human eye opening and closing, while the eyeball model can achieve combined movements of looking up, down, left, and right.

[0032] 2. Modeling small deformation areas Identify facial areas with relatively small deformations during facial expression changes, such as the area between the eyebrows and the corners of the mouth, and design them parametrically in detail. By defining key dimensional parameters, geometric parameters (such as control point coordinates of curves and curvature parameters of surfaces), and relative position parameters, a parametric model of these small facial deformation areas is established. Elastic meshes, as an effective geometric representation method, play a crucial role in structural design. They can flexibly adapt to the complex and ever-changing geometric shapes of the face, accurately and meticulously depicting the microscopic details of the face, thus providing a solid foundation for the design of facial mesh structures. Through scientifically sound mesh partitioning strategies and deformation mechanisms, it is possible to more effectively simulate the movement patterns of facial muscles, the dynamic changes in expressions, and the specific differences in facial features between individuals. By designing a 3D model of an elastic mesh that conforms to the curvature of the human face and its parametric modeling, small deformation movements during facial expressions can be realized.

[0033] To better reflect reality, a 3D model of the face's curved surface was obtained by scanning a real human face. The facial curved surface model was then appropriately adjusted and repaired to obtain a symmetrical model. Based on the designed large deformation areas such as the eyes and mouth, the face curved surface model was trimmed to create holes to expose key parts like the mouth and eyes. The surface boundaries of the face curved surface model were then converted from a 3D sketch to a solid reference, and a 1mm radius scan was performed on the sketch to obtain the face's elastic mesh. The face curved surface model was thickened, and appropriate holes were added based on the jaw and eye models. It was then fixed to the neck model to form a face backplate module, which was secured to the foundation model via threaded connections. Multiple small holes were also made on the face backplate module to allow for subsequent binding to secure the face's elastic mesh boundaries. Small deformation area models are shown below. Figure 4 and 5 As shown.

[0034] 3. Methods for reproducing and controlling facial expressions Facial expression reproduction mainly includes the movement of large deformation areas such as the jaw and eyes, and the deformation of small deformation areas such as the corners of the mouth and the space between the eyebrows. As mentioned above, the movement of large deformation area modules is mainly achieved by a crank-rocker structure that causes the corresponding parts to rotate. Small deformation areas are mainly reproduced by applying force or displacement to the intersection nodes of the facial elastic mesh. The main tasks involve the distribution and arrangement of nodes in the facial elastic mesh and the driving method of the nodes.

[0035] Human facial muscles are thin, flat cutaneous muscles, located superficially, thin and delicate, mainly distributed around the mouth and eyes, as well as around the nose, jaw, and forehead. Around the eyes, the main muscles include the corrugator supercilii, orbicularis oculi, and depressor supercilii; around the mouth, the risorius, buccinator, masseter, and orbicularis oris muscles; around the nose, the nasalis muscles and levator labii superioris; around the jaw, the masseter and depressor labii inferioris muscles; and around the forehead, the frontalis muscle. This study proposes to select and simplify facial elastic mesh nodes based on 68 facial feature points from the dlib library. Thirteen nodes are selected as driving nodes, including those in the brow area (brow tip, middle, and tail), the eye area (corner of the eye), and the lip area (corner of the mouth, upper lip, and lower lip). Figure 6 and 7 As shown.

[0036] The driving of the nodes is proposed to be achieved through the unique shape memory effect of slender rod-shaped shape memory alloys. Shape memory alloys are a class of smart materials with special functions. After undergoing plastic deformation at low temperatures, they can automatically recover their original shape when heated to a specific phase transformation temperature, exhibiting reversible forms such as single-pass and double-pass deformations. They also possess superelasticity, capable of producing deformations far exceeding those of ordinary metals (up to 5%-10%) above the phase transformation temperature, with no residual deformation after the removal of external force. Mechanically, they exhibit high strength and long fatigue life; chemically, they demonstrate excellent corrosion resistance and biocompatibility. Furthermore, their core martensitic phase transformation can be precisely controlled through temperature, stress, and other conditions, providing a foundation for various engineering applications. The proposed method utilizes the "electrothermal effect" to trigger the shape memory effect of slender rod-shaped shape memory alloys to control the force and displacement of facial expression nodes in different directions. Precise and reversible length changes (deformation amplitude 1%-5%) can be achieved by adjusting the current, with fast response speed and stable output force. When energized, the alloy heats up rapidly due to Joule heating (Q=I²Rt, where I is the current, R is the alloy resistance, and t is the energizing time). When the temperature exceeds the phase transformation point, the internal martensite phase transforms into the more densely packed austenite phase, causing the rod to macroscopically shrink. After the power is turned off, the heat dissipates and the phase transformation reverses, and the rod returns to its original position under elastic action. Essentially, this is the result of a reversible phase transformation triggered by the electrothermal effect.

[0037] To reproduce multiple facial expressions, the driving direction of each node should be multidimensional. Therefore, three forces or displacements in different directions are applied to each node to be driven. For example... Figure 6 and 7 As shown, one end of a slender rod-shaped shape memory alloy is fixed to each selected node of the face elastic mesh, and the other end is fixed to the face back plate model. The three slender rod-shaped shape memory alloy segments under each node are approximately distributed in a circle with three equal parts without interfering with other models.

[0038] 4. Facial expression simulation implementation The motion of facial expressions was simulated using the ANSYS Workbench engineering simulation platform. The constructed 3D model was imported into Workbench, and the necessary engineering data, such as material properties, were created for the simulation. The SpaceClaim tool was then used to repair and simplify the model. Because directly building the face model and generating the elastic mesh for simulation in SolidWorks was too complex and prone to geometric defects such as feature redundancy and unclosed regions, the "Concept" function in the DesignModeler module of ANSYS Workbench was used instead. First, the wireframe geometry of the face was constructed, and then a circular cross-section with a radius of 1mm was assigned to it to generate a line body. This method significantly simplified the model and, combined with the cross-sectional parameters of the line body, accurately simulated the mechanical behavior of slender structures, meeting the simulation requirements. To simplify the meshing process, 13 slender rod-shaped shape memory alloy models were also constructed as line body models. The simplified model was opened in the Mechanical tool, and corresponding geometric materials were added to each geometry; for example, 65Mn spring steel was used for the elastic mesh model of the face, and nickel-titanium shape memory alloy was used for the slender rod-shaped shape memory alloy models. Based on the actual situation, connections and contacts between various components are added, along with appropriate mesh generation and adjustment methods. While ensuring analysis accuracy, to significantly optimize simulation efficiency and simplify model logic, models that do not participate in small deformations, such as the eyes and jaw models, are designated as non-rigid bodies, thus excluding them from mesh generation.

[0039] The goal is to reproduce a "smiling" expression. Fixed constraints are added to certain surfaces of the face background model and the outermost boundary of the face surface model. Corresponding forces are added to the outer corner nodes of the eye and mouth nodes of the elastic mesh of the face, such as... Figure 8 As shown. The total deformation of the deformable parts of the model obtained after solving is as follows. Figure 9 As shown.

[0040] 5. Implementation of common facial expression displacement based on elastic facial mesh nodes Facial expressions are divided into macro-expressions and micro-expressions, both of which involve facial muscle activity. Some scholars encode human faces, but the most mainstream facial coding method is based on the characteristics of facial anatomy, where psychologists such as Ekman divided facial muscles into several independent yet interconnected motor units (AUs). This method describes the correspondence between different facial muscle movements and different facial expressions. Common AU units are shown in Table 1, and the complete AUs associated with common expressions are shown in Table 2.

[0041] Table 1 Common AU Units Table 2. AUs related to facial expressions The nodes required for facial expression reproduction have been simplified, so the AU parts related to common expressions have been omitted. The bilaterally symmetrical nodes and their driving rod-shaped shape memory alloys have been simplified to unilateral. To elaborate on the displacements that each face mesh node should drive in three directions, a system is established as follows: Figure 10 The spatial coordinate system shown has its origin at the node to be driven (e.g., point ag), and its directions point along the axes of three slender rod-shaped shape memory alloys toward the face backplate model. And below... Figure 6 Using the plane of the model shown as a reference, the first quadrant is designated as axis 1, and axes 2 and 3 are established sequentially by rotating counterclockwise. In the case of single-sided nodes, the associated nodes of the aforementioned common expressions are shown in Table 2.

[0042] Table 3 shows the axial displacement of the expression-related nodes, which only includes the nodes and their values ​​on the left half of the model viewed directly. The nodes and their coordinate axes on the right half can be mirrored, and their values ​​are the same as those on the left half.

[0043] Table 3. Axial displacement ( / mm) of the left half node associated with facial expressions 6. Analysis of common facial expression displacement data in a fixed coordinate system To further quantify facial expression displacement characteristics, verify the rationality of the displacement implementation scheme, and accurately grasp the peak displacement, intensity distribution, and displacement differences between feature points of each grid node in the three directions of the fixed coordinate system under different facial expressions, a detailed analysis of the displacement data of common facial expressions in the fixed coordinate system will be conducted next. The fixed coordinate system is defined as follows: x-axis is horizontal to the right, y-axis is vertical upward, and z-axis is perpendicular to the paper and outward.

[0044] The initial data for this experiment came from the BP4D-Spontaneous dataset (containing 3D dynamic spontaneous facial expression data from 41 subjects). Data filtering was performed before code processing: based on the facial motion unit (AU) intensity value corresponding to each frame, the frame with the lowest intensity was selected as the neutral frame, and the frame with the highest intensity as the peak frame. After verification, segments with continuous expression changes and optimal features between the neutral and peak frames were selected. The displacement difference between these segments and the neutral frames was calculated, and then subsequent MATLAB batch processing was performed on these segments. Combining the previously established facial elastic mesh nodes, AU association system, and spatial coordinate system, the expression displacement features were quantified and peak data extracted through code. Only 400-500 frames of data after the neutral frames were retained, and symmetry bias was used to control the fit to the physiological features of real human faces.

[0045] The code performs batch processing on valid participant data throughout the entire process. The core steps are integrated as follows: First, it automatically identifies all participant folders under the base path, loads the displacement data and verifies its completeness and standardization, and filters out valid participants whose neutral frames meet the judgment criteria and whose number of frames after the neutral frame is within the range of 400-500 (this frame number range is set by the user based on the range of typical facial expression changes from the neutral frame to the peak frame for most participants in the BP4D-Spontaneous dataset, which can balance the completeness of facial expression features and data redundancy); Second, it uses all valid participants... The maximum number of frames after the neutral frame is used as the global reference time axis. All data lengths are standardized. Outliers are filtered using the 3σ+MAD dual criterion, and missing values ​​are filled with the mean. Frame completion is performed using pchip interpolation to ensure data continuity. Simultaneously, interpolation error is calculated to quantify frame completion accuracy, and deviation corrections are applied to symmetrical nodes (approximately the opposite value in the X direction, retaining natural errors in the Y / Z directions) to reflect the physiological characteristics of the non-perfect left-right symmetry of real human faces. Subsequently, the mean, deviation, and other core features of the symmetrical node displacement data are statistically analyzed, and a comparison chart of the symmetrical node displacement mean is plotted to visually present the processing effect (e.g., ...). Figure 11 This clearly displays the displacement trends and natural errors of the left / right symmetrical nodes; simultaneously, a third-order polynomial is used to fit the X / Y / Z data of each feature point individually, and the fitting effect is evaluated by the coefficient of determination (R²), and the displacement scatter plot of key feature points is drawn. Figure 12 ), a schematic diagram of the fitted curve (e.g.) Figure 13 The first step is to visually demonstrate the continuity of interpolation and the fit of the fitted curve to the displacement trend. Next, based on the "group consensus + three-dimensional modulus method", the global expression intensity is calculated and the peak frame is located. The displacement values ​​of each feature point in the peak frame are extracted and the symmetry deviation is corrected. After the data is sorted, it is saved to Excel. Finally, the feature point displacement values ​​from the neutral to the peak stage under each expression are obtained (as shown in Tables 4-6). Finally, the average R² value and interpolation error are combined to complete the data processing quality assessment. The statistical report, visualization charts and original data files are saved in batches to ensure that the results are traceable and reusable.

[0046] Table 4 "Happiness" (Peak frame 417) Table 5 "Sadness" (Peak Frame 495) Table 6 "Anger" (Peak Frame 454) The aforementioned X, Y, and Z displacements obtained through batch processing with MATLAB represent the target displacement values ​​that each facial expression node needs to be driven in a fixed coordinate system. These values ​​are also the core basis for controlling the length of the shape memory alloy rod. Theoretically, by matching the initial length and deformation coefficient of the shape memory alloy rod in each direction, the X / Y / Z displacements can be converted into the corresponding deformation of the alloy rod. Further derivation of the required electrothermal effect control parameters, such as current and energizing time, is then possible. Subsequent research will focus on specific implementation schemes for the electrothermal effect around this target displacement: on the one hand, by combining parameters such as the phase transition temperature and resistance characteristics of the shape memory alloy, a quantitative correlation model of "current-temperature-deformation-node displacement" will be established to clarify the current intensity and energizing time thresholds corresponding to different X / Y / Z displacements; on the other hand, a shape memory alloy driving module adapted to facial mesh nodes will be designed to verify whether controlling the extension and retraction of the alloy rod through the electrothermal effect can accurately reproduce the target X / Y / Z displacements, ultimately achieving controllable and reversible driving of the facial expression nodes.

[0047] Example 2 A robot facial expression mimicry system, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement a robot facial expression mimicry method when executing executable instructions.

[0048] It should be noted that the computer device includes a processor, a memory, and may also include one or more of a multimedia component, an input / output (I / O) interface, and a communication component.

[0049] The processor controls the overall operation of the computer device to complete all or part of the steps in the above-mentioned robot facial expression imitation method.

[0050] Memory is used to store various types of data to support the operation of the computer device. This data may include, for example, instructions for any application or method used to operate on the computer device, as well as application-related data. Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0051] The multimedia component may include a screen and an audio component, wherein the screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals; for example, the audio component may include a microphone for receiving external audio signals, the received audio signals may be further stored in memory or transmitted via a communication component; the audio component may also include at least one speaker for outputting audio signals.

[0052] I / O interfaces provide interfaces between the processor and other interface modules, such as keyboards, mice, buttons, etc.; these buttons can be virtual buttons or physical buttons.

[0053] The communication component is used for wired or wireless communication between the computer device and other devices; wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G or 5G, or one or more combinations thereof, and the corresponding communication component may include: Wi-Fi module, Bluetooth module, NFC module, mobile communication module.

[0054] As a preferred embodiment, the computer device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the above-described robot facial expression imitation method.

[0055] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for imitating robot facial expressions, characterized in that, The method includes: S1. Create a mandibular model and an eye model; S2. Construct a facial surface model that matches the jaw model and eye model based on the elastic mesh, and perform mesh generation; S3. Select n points on the facial surface model as nodes to be driven; S4. Three driving rods are set for each of the nodes to be driven. One end of the three driving rods is used to drive the node to be driven to move in different directions according to the facial expression data, and the other end is connected to the face back panel model respectively. The driving rods are made of shape memory alloy and connected to a controllable current. S5. Establish a database of the state of the jaw model and eye model under different facial expressions and the displacement of the drive rod based on the correspondence between facial muscle movements and different expressions; S6. Based on the facial expression data, drive the movement of the node to be driven, the jaw model, and the eye model according to the database to imitate facial expressions.

2. The robot facial expression imitation method as described in claim 1, characterized in that, The shape memory alloy mentioned in step S4 is a nickel-titanium shape memory alloy material.

3. The robot facial expression imitation method as described in claim 1, characterized in that, In step S2, constructing a facial surface model that matches the jaw model and eye model based on an elastic mesh includes: S201. Collect basic three-dimensional contour data of the robot's face, including the initial morphological parameters of the jaw model and eye model; S202. Import the basic three-dimensional contour data into the three-dimensional modeling system to generate the initial facial surface framework; S203. Based on the deformation characteristics of the elastic mesh, the distribution of mesh nodes in the initial curved surface frame is optimized so that the mesh density matches the surface change requirements of different areas of the face, and the mesh edges fit the edges of the jaw model and eye model to obtain the facial curved surface model.

4. The robot facial expression imitation method as described in claim 1, characterized in that, In step S3, n equals 13.

5. The robot facial expression imitation method as described in claim 1, characterized in that, In step S3, the nodes to be driven include: based on the regional distribution of facial expression movement, the positions of the nodes to be driven on the facial surface model are determined, including: 1 node each in the middle and on both sides of the forehead, 1 node each below the left and right brow bones near the eyelids, 1 node each at the starting point of the apple cheek below the left and right eye sockets, 1 node each at the apple cheek protrusion on the left and right cheeks, 1 node each above the nostrils on both sides of the nose, and 1 node at the tip of the chin.

6. The robot facial expression imitation method as described in claim 1, characterized in that, In step S5, a database is established based on the correspondence between facial muscle movements and different expressions, including the states of the mandibular and eye models for different expressions, as well as the displacement of the drive rod. S501. Collect facial muscle movement data under various typical human facial expressions; S502. Based on the muscle movement data, use a three-dimensional motion capture system to synchronously record the actual displacement of the corresponding nodes to be driven on the face, as well as the real-time state parameters of the jaw model and eye model. S503. Establish a database based on the actual displacement of the node to be driven and the real-time state parameters of the mandibular model and the eye model.

7. The robot facial expression imitation method as described in claim 1, characterized in that, In step S6, the expression imitation is performed by driving the movement of the node to be driven, the jaw model, and the eye model based on the expression data and the database. This includes: S601, Receive the facial expression data to be imitated; S602. Preprocess the facial expression data to extract facial expression feature information, including expression category, intensity and regional distribution of facial muscle movements; S603. Based on the extracted facial expression feature information, perform matching and retrieval in the database to obtain the displacement of the node to be driven, the real-time state parameters of the jaw model and the eye model corresponding to the target facial expression. S604. Facial expression mimicry is performed based on the displacement of the node to be driven, the real-time state parameters of the jaw model and the eye model.

8. A robot facial expression mimicry system, characterized in that, The system includes: processor; Memory used to store processor-executable instructions; The processor is configured to implement the robot facial expression imitation method according to any one of claims 1 to 8 when executing the executable instructions.

Images