A method, system, device, and storage medium for controlling facial expressions in a bionic robot.

By updating the target angles of the eyeballs and facial joints using binocular cameras and proportional control laws, and coordinating eyelid closing movements, the problem of visual field occlusion in bionic robots during gaze tracking and facial expression mirroring is solved, achieving natural facial expression display and efficient visual field utilization.

CN123060686APending Publication Date: 2026-11-10QINGNAO (SHANGHAI) INTELLIGENT TECHNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611297731.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-25
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In existing bionic robots, eye movements and eyelid actions can obstruct the camera's field of view during gaze tracking and facial expression mirroring, leading to a higher failure rate in visual tracking and unnatural facial expressions.

Method used

Image frames are acquired using a binocular camera. The target angles of the eyeballs and facial joints are updated through a proportional control law formula to ensure visual field safety and coordinate eyelid closing movements. Closed-loop incremental proportional control and visual field safety travel calibration are used to achieve coordinated operation of gaze tracking and facial expression mirroring.

Benefits of technology

It achieves natural facial expressions and near 100% field of vision utilization in the interaction process of the bionic robot, eliminates the obstruction of the field of vision by eye movements, reduces eye tremors and eccentric swaying, and improves the stability and naturalness of face following.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123060686A_ABST
    Figure CN123060686A_ABST
Patent Text Reader

Abstract

This invention discloses a bionic robot facial expression control method, system, device, and storage medium, relating to the field of bionic robot technology. The bionic robot facial expression control method of this invention includes the following steps: S1, acquiring image frames from a binocular camera; S2, generating a target face bounding box and extracting target face expression features based on the image frames; S3, generating a normalized deviation e(ex, ey) of the center of the target face bounding box relative to the image center, where ex represents the horizontal degree of freedom and ey represents the vertical degree of freedom; S4, incrementally updating the target angles of the vertical and horizontal degrees of freedom of the binocular camera according to the proportional control law formula: Δθ=Kp·e. The bionic robot facial expression control method, system, device, and storage medium of this invention can effectively overcome the shortcomings of existing technologies, and can simultaneously achieve annotation following and facial expression reproduction (or mirroring), providing an excellent user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of biomimetic robot technology, and in particular to a biomimetic robot facial expression control method, system, device and storage medium. Background Technology

[0002] Everyday experience tells us that when robots interact with humans face-to-face, if they can continuously "look" at the other person and mimic or respond to their facial expressions in real time, the naturalness and approachability of the communication will be significantly enhanced. Therefore, it is necessary to implement and test these capabilities in the developed bionic robots to enable them to function better.

[0003] Based on the above objectives, when developing bionic robot heads, people focus on whether the robot head possesses the following two capabilities simultaneously: firstly, gaze following, that is, moving the eyeballs according to the detected face position so that the gaze is always roughly facing the person being communicated with; secondly, expression mirroring, that is, detecting the eyelid, eyebrow, and cheek movements of the other person's face, and driving the corresponding facial joints of the robot through redirection mapping (converting the movements of the other person's face into the robot's own joint angles according to the corresponding relationship), thereby achieving expression cloning and enabling expression control of the bionic robot, laying a foundation for the subsequent bionic robot to make relevant expressions during communication.

[0004] Several solutions related to eye movement and facial expression reproduction in bionic robots have emerged in the prior art. For example, patent document CN101618280B discloses a humanoid robot device and behavior control method with human-computer interaction function. It embeds a small CCD sensor in each eyeball to form the robot's binocular vision, and sets up eye movement units, eyelid transmission mechanisms, and facial expression and mouth shape driving mechanisms, and then controls it based on an artificial emotion model. However, the visual tracking and facial expression reproduction of this prior art are designed according to independent channels. When the eyeballs rotate, the eyelids close, or other actions obstruct the field of view of the intraocular camera, its facial expression reproduction ability is affected, and it appears very unnatural. As another example, patent document US11185990B2 discloses a method for a robot to learn and reproduce human facial expressions. It measures facial muscle signals through epidermal electrodermal sensors, obtains facial movement units through non-negative matrix decomposition, and uses iterative learning to correct the servo control coefficients to make the robot's expressions approximate human expressions. While the existing technology focuses on the accuracy of facial expression reproduction itself, its sensing devices are installed independently of the facial expression execution mechanism. Facial movements do not affect the sensing input. It does not address the coupling problem when facial expression execution and visual perception share the same intraocular camera, resulting in poor anthropomorphism of the robot face during gaze interaction and limited appearance realism.

[0005] In summary, existing technologies are ill-equipped to handle the aforementioned challenges in controlling facial expressions in bionic robots, particularly when the eye movements required for gaze tracking and the eyelid movements required for facial mirroring obstruct the field of view of the intraocular camera, which is crucial for both. This is especially true for bionic robot heads aiming for high human-likeness, where the camera is mounted inside the rotating eyeball and covered by a bionic silicone eyelid. During blinking, squinting, and opening the eyes, the lens becomes unnatural because excessive eyeball rotation can obstruct the lens from the silicone skin at the eye socket edge, and excessive eyelid closure can completely cover the lens. This results in increased frequency of eyeball tremors and a higher failure rate in face tracking. Summary of the Invention

[0006] This invention provides a method, system, device, and storage medium for controlling the facial expressions of a bionic robot. By addressing the aforementioned technical deficiencies, a novel design approach is adopted, which can simultaneously achieve annotation following and facial expression reproduction (or mirroring), enabling the bionic robot to display natural facial expressions and provide an excellent user experience during interaction.

[0007] The bionic robot facial expression control method provided in this embodiment of the invention includes the following steps: S1. Acquire image frames from the stereo camera; S2. Based on the image frame, generate a target face bounding box and extract the target face expression features; S3. Based on the target face bounding box, generate the normalized deviation e(ex, ey) of the center of the target face bounding box relative to the image center, where ex represents the horizontal degree of freedom and ey represents the vertical degree of freedom; S4. Incrementally update the target angles of the vertical and horizontal degrees of freedom of the binocular cameras according to the following proportional control law formula: Δθ=Kp·e Where: e is the normalized deviation e(ex, ey) in S3, Kp is the preset proportional gain, and Δθ is the rotation increment superimposed on the target angle of the corresponding degree of freedom in this frame; S5. Based on the angle increment and the target facial expression features, clamp the target angle within the pre-calibrated safe field of view and generate a servo control signal, which is used to generate robot expressions.

[0008] Optionally, the method for extracting the target facial expression features in step S2 is: facial landmark detection.

[0009] Optionally, the formation of a safe field of view in step S5 is predetermined through the following steps: S501. Starting from the neutral position, gradually increase the eyeball rotation angle and observe the occlusion of the bionic eyelid or silicone skin in the intraocular camera image. S502. The maximum eyeball turning angle before the occlusion occurs is used as the travel boundary to calibrate the safe travel distance of the field of vision.

[0010] Optionally, step S5, particularly after step S502, may also include: S503. Coordinate and limit the eyelid closing action of the eyeball, that is, when the gaze tracking is active, limit the closing amplitude of the upper and lower eyelids obtained by the redirection mapping to no more than a preset upper limit of visible closing. The upper limit of visible closing is determined by calibration and is the closing amplitude when the eyelid begins to block the field of view of the intraocular camera. Any part exceeding this limit is truncated.

[0011] Optionally, step S5, particularly after step S503, may also include: S504. When the duration of no face detection exceeds the preset grace period, the target angle is controlled to slowly move towards the partially centered target from the moment the face is lost. The partial centering target is the neutral position plus the difference between the angle at the moment of loss of face and the neutral position multiplied by a preset retention coefficient; the retention coefficient is a proportional value between 0 and 1, representing the proportion of the original deflection orientation retained after centering.

[0012] Optionally, step S5 may be followed by: S6. Perform a safety limit check on the servo control signal, and send the servo control signal that has passed the safety limit check to each servo.

[0013] Optionally, step S6 further includes discarding the servo control signal that fails the safety limit check.

[0014] This invention also provides a bionic robot facial expression control system, comprising: The image acquisition module is used to acquire and obtain image frames output by the binocular camera installed inside the rotatable eyeball; The face detection module is used to perform face detection on the image frame to obtain the target face bounding box; The gaze-following module is used to calculate the normalized deviation of the center of the target face frame relative to the center of its image, incrementally update the target angle of the vertical and horizontal degrees of freedom of the eye according to the proportional control law formula, and clamp it within the pre-calibrated visual field safety range. The expression mirroring module is used to extract the facial expression features of the target person from the same image frame, redirect the expression features to the target angle of the facial joints and generate servo control signals, and apply coordinated restrictions to the amplitude of eyelid closing movements that may obstruct the field of view of the binocular camera. The face-lost-centering module is used to control the target angle of the eye to slowly move towards the partially centered target when the duration of no face detection exceeds a preset grace period; The command sending module is used to send the servo control signals to the control box to drive the eyeball servo, eyelid servo, and other facial servo to perform robot expression generation.

[0015] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the facial expression control method described in any of the preceding claims.

[0016] This invention also provides a computer scale storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the facial expression control method described in any of the preceding claims.

[0017] Compared with the prior art, the beneficial effects achieved by the embodiments of the present invention are: (1) Elimination of self-occlusion: By clamping the target angle of the eyeball within the calibrated visual field safety range, the closure amplitude of the mirror eyelid is controlled in a coordinated manner. Practice has shown that the availability of the intraocular camera's visual field is close to 100% during gaze tracking and expression mirroring, structurally eliminating input interruption caused by "self-movement blocking one's own visual field".

[0018] (2) Fast convergence and no eccentric swing: Closed-loop incremental proportional control is adopted. By utilizing the objective physical characteristic that "the camera moves with the eye", the face is pushed back to the vicinity of the center of the screen within 3 to 5 frames by only turning slightly in the direction of reducing the deviation. This basically eliminates the existing technical defects such as eccentric pause and continuous swing of the absolute mapping scheme.

[0019] (3) Suppressing cyclic jitter: When the face is reproduced, only a small correction is needed to re-lock it. The cyclic jitter of "deflection → occlusion → reset → deflection again" is reduced from continuous oscillation to at most one slow motion.

[0020] (4) Stable locking in multi-person scenes: The two eyes independently detect and select the person with higher confidence as the target. Even when others pass by, the main person being communicated with is still locked, and the target jump rate is significantly reduced.

[0021] (5) Follow and mirror share the same camera and face detection front end, without the need to add any sensors. The computing power overhead is only one detection. The two functions are completed in the same control cycle.

[0022] (6) The bionic robot has natural expressions during the interaction process, and the user experience is excellent. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the hardware structure of the bionic robot facial expression control method applicable to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an application of the facial expression control method in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the hardware relationship of the bionic robot facial expression control method according to an embodiment of the present invention. Figure 4 This is a flowchart of a bionic robot facial expression control method according to an embodiment of the present invention; Figure 5 A schematic diagram of the visual field safety travel provided for an embodiment of the present invention (showing the relationship between eyeball rotation angle and bionic eyelid occlusion). Figure 6 This is a timing diagram of the missing face portion of the sequence provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0027] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a communication connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances. The technical solutions of this invention are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0028] like Figure 1 and Figure 2 As shown, the bionic robot facial expression control method provided in this embodiment of the invention includes the following steps: S1. Acquire image frames from the stereo camera; S2. Based on the image frame, generate a target face bounding box and extract the target face expression features; S3. Based on the target face bounding box, generate the normalized deviation e(ex, ey) of the center of the target face bounding box relative to the image center, where ex represents the horizontal degree of freedom and ey represents the vertical degree of freedom; S4. Incrementally update the target angles of the vertical and horizontal degrees of freedom of the binocular cameras according to the following proportional control law formula: Δθ=Kp·e Where: e is the normalized deviation e(ex, ey) in S3, Kp is the preset proportional gain, and Δθ is the rotation increment superimposed on the target angle of the corresponding degree of freedom in this frame; S5. Based on the angle increment and the target facial expression features, clamp the target angle within the pre-calibrated safe field of view and generate a servo control signal, which is used to generate robot expressions.

[0029] The target angle mentioned in step S5 includes the target angle of the eyeball and the target angle of the facial joint.

[0030] The incremental update in step S4 only applies to the target angles of the vertical and horizontal degrees of freedom of the eyeball. That is, with the calibrated neutral position as the initial value, the rotation increment Δθ obtained in step S4 is superimposed frame by frame on the current target angle of the eyeball and accumulated. Then, the accumulated target angle of the eyeball is clamped within the safe travel range of the field of vision.

[0031] It should be noted that the target angle of the facial joints is not updated incrementally, but is directly calculated joint by joint from the target facial expression features through redirection mapping.

[0032] One possible method for generating servo control signals is as follows: the target angles of the eyeballs and facial joints are sent to a control box. The control box, according to a preset mapping from joints to servo control boards and registers, converts each target angle into a pulse-width modulation (PWM) signal based on a calibration relationship and writes it to the corresponding servo control board to drive the eyeball servo, eyelid servo, and other facial servos. Preferably, the period of the PWM signal is 15-25 milliseconds, and its high-level pulse width, within the range of 0.5 to 2.5 milliseconds, corresponds to a servo rotation angle of 0 to 180 degrees.

[0033] Preferably, the method for extracting the target facial expression features in step S2 is: facial landmark detection method.

[0034] Preferably, the formation of a safe field of view in step S5 is predetermined through the following steps: S501. Starting from the neutral position, gradually increase the eyeball rotation angle and observe the occlusion of the bionic eyelid or silicone skin in the intraocular camera image. S502. The maximum eyeball turning angle before the occlusion occurs is used as the travel boundary to calibrate the safe travel distance of the field of vision.

[0035] Preferably, step S5, particularly after step S502, further includes: S503. Coordinate and limit the eyelid closing action of the eyeball, that is, when the gaze tracking is active, limit the closing amplitude of the upper and lower eyelids obtained by the redirection mapping to no more than a preset upper limit of visible closing. The upper limit of visible closing is determined by calibration and is the closing amplitude when the eyelid begins to block the field of view of the intraocular camera. Any part exceeding this limit is truncated.

[0036] Preferably, step S5, particularly after step S503, further includes: S504. When the duration of no face detection exceeds the preset grace period, the target angle is controlled to slowly move towards the partially centered target from the moment the face is lost. The partial centering target is the neutral position plus the difference between the angle at the moment of loss of face and the neutral position multiplied by a preset retention coefficient; the retention coefficient is a proportional value between 0 and 1, representing the proportion of the original deflection orientation retained after centering.

[0037] Preferably, after step S5, the method further includes: S6. Perform a safety limit check on the servo control signal, and send the servo control signal that has passed the safety limit check to each servo.

[0038] Preferably, step S6 further includes discarding the servo control signal that fails the safety limit check.

[0039] like Figure 1 As shown, this embodiment provides a bionic robot head for the specific implementation of the aforementioned bionic robot expression control method. Externally, it includes: binoculars 1 and various servo motors and specific structures 2 for generating robot expressions. (See figure) Figure 3 As shown in the figure, the embodiments of the present invention also provide another hardware relationship schematic diagram. Specifically, the intraocular camera 101 is installed inside the rotatable eyeball 107 and can rotate with the eyeball 107, outputting image frames from both eyes; the bionic eyelid 102 is a bionic silicone eyelid covering the front of the intraocular camera 101, driven to open and close by the eyelid servo motor 104; the eyeball servo motor 103 drives the eyeball 107 to rotate around the vertical and horizontal degrees of freedom; the host computer 105 runs the face detection, gaze tracking control, and expression mirroring redirection program; the control box 106 is an embedded computer (in this embodiment, a Raspberry Pi) located inside the robot head, which receives the joint angle dictionary sent by the host computer 105 via the ZMQ response terminal bound to tcp: / / *:8765, and writes it to each servo control board via the Modbus serial bus; the facial expression servo motor 108 includes servos for the eyebrow and cheek joints; the images acquired by the intraocular camera 101 are compressed into JPEG and pushed to the host computer 105 at 4 to 10 frames per second, with the data size of a single frame compressed to the 60KB level to ensure real-time transmission. The host computer 105 and the control box 106 together constitute the main body for collaborative control, with the user (interaction partner) 109 located in front of the robot. Practical verification has shown that this hardware structure can implement the bionic robot facial expression control method provided by this invention.

[0040] Below, this application will provide a detailed explanation of each step in the aforementioned bionic robot facial expression control method in a practical manner. It should be noted that all specific steps of the aforementioned bionic robot facial expression control method can be implemented in the specific steps described below, and each of the following specific descriptions further explains the aforementioned bionic robot facial expression control method.

[0041] I. Image Acquisition and Face Detection As mentioned earlier, the intraocular camera 101 is a camera installed inside the rotatable eyeball that rotates with the eyeball. It is the sole source of visual input for this method, and its output binocular image frame can be two side-by-side images, each 640 pixels wide and 480 pixels high (combined frame 1280×480). One possible face detection method is to directly use an open-source face detection model: In this embodiment, the YuNet model included with OpenCV is used, with the model file face_detection_yunet_2023mar.onnx. This model has been pre-trained by OpenCV on the publicly available WIDER FACE face dataset. After being called, it outputs several face bounding boxes and their respective confidence scores for an image, with a confidence threshold of 0.6. Face detection can be performed independently in the left and right images. The image with the highest confidence score is selected from each image, and then the image with the higher confidence score from the other two images is selected as the target face bounding box. The target face bounding box is the rectangle (represented by the coordinates of the top-left corner and width and height) output by the face detection model that encloses the area where the face is located. Its center is used as the localization reference for subsequent gaze tracking; thus, even when others are nearby, the model can still lock onto the primary interaction subject with a higher score. To reduce detection time, in a preferred embodiment, each image is scaled by 0.5 times (width reduced to approximately 320 pixels) before being sent to the detector, and the bounding box coordinates are then proportionally restored to the original resolution. Since gaze tracking only requires normalization of the bias, the impact of this downscaling on tracking accuracy is negligible, while the number of detected pixels is reduced to one-quarter, and the detection throughput is increased by approximately one time. The scaling factor can be configured via environmental variables; a value of 1.0 indicates full-resolution detection.

[0042] II. Eye Follow The normalized bias, simply put, refers to the deviation of the target face bounding box center from the image center, proportionally calculated based on the width and height of the image (ranging from -1 to 1, where 0 indicates the face is exactly centered, and the sign indicates which side it's off to). Its function is to make the control law independent of the specific resolution. The normalized bias is explained below. Assuming the pixel coordinates of the target face bounding box center within its frame are (cx, cy), and the image center is (320, 240), then the normalized bias is ex = (cx − 320) / 320, ey = (cy − 240) / 240.

[0043] like Figure 4As shown, the execution steps for each frame can be: acquiring binocular image frames (step 201) → detecting and selecting the best of two independent faces (step 202) → determining whether a face is detected (step 203); calculating the normalized deviation (ex,ey) when detected (step 204) → dead zone judgment (step 205): when |ex| < 0.03, the horizontal remains stationary, and when |ey| < 0.03, the vertical remains stationary → incremental update with proportional control (step 206): θ_h ← θ_h + kp_h·ex, θ_v ← θ_v + kp_v·ey, where kp_h = 7.0 and kp_v = 5.0 → clamping to the safe range of the field of view (step 207): θ_v clamped to [14, 46], θ_h clamped to [0, 46] → Same-frame expression feature extraction and redirection mapping (step 208) → Eyelid closure amplitude coordination limitation (step 209) → Merging eyeball and facial joint target angles and sending them to control box 106 via ZMQ (step 210). When no face is detected, proceed to face loss timing and partial centering branch (step 211, see Example 5 for details). The "incremental update" approach does not convert the face position into the absolute eye angle all at once. Instead, it adds a small increment Δθ=Kp·e proportional to the deviation to each frame based on the current target eye angle (i.e., the angle that the controller expects the servo to turn to) and accumulates it (θ_h ← θ_h +kp_h·ex, θ_v ← θ_v + kp_v·ey). The larger the deviation, the larger the increment. When the deviation enters the dead zone, the increment becomes zero. This is a closed-loop control that approximates the deviation frame by frame. The "clamping" approach (step 207) refers to forcibly restricting a value within a given range. If it is greater than the upper limit, the upper limit is taken; if it is less than the lower limit, the lower limit is taken; if it is within the range, it remains unchanged. For example, θ_v is clamped to [14,46], that is, when θ_v is less than 14, 14 is taken; when it is greater than 46, 46 is taken. This ensures that the target eye angle always falls within the safe travel range of the field of vision. The neutral position is set with vertical degrees of freedom eyes_vertical = 31 and horizontal degrees of freedom eyes_horizontal = 14, representing the calibrated position of both eyes looking straight ahead. Incremental updates use the neutral position as the initial value. Since the intraocular camera 101 rotates with the eyeball 107, each slight turn in the direction of reducing deviation brings the face closer to the center of the image. The closed loop pushes the face back into the dead zone in the center of the image within 3 to 5 frames. The proportional gain value balances convergence speed and overshoot, and no oscillation occurs at an image rate of 4 to 10 frames per second. The servo motors on different fuselages may be installed in opposite directions, so inverted flags for the horizontal and vertical directions are set. The calibration is performed according to the actual steering configuration, while the control law itself remains unchanged.

[0044] III. Safety travel distance calibration like Figure 5As shown, the eyeball 107 rotates around the eyeball rotation axis 301, and the camera optical axis 302 of the intraocular camera 101 installed inside it deflects along with the eyeball. The mechanically movable range of the eyeball 107 is the hard safety envelope 306 (vertical 0 to 80°, horizontal 0 to 60°) – also known as the "hard safety envelope," which refers to the maximum range of rotation allowed by the eyeball's mechanical structure itself. It is an insurmountable physical upper limit; exceeding this range may cause the connecting rod to seize, the transmission component to be strained, and damage the structure. When the eyeball rotation angle increases to a certain extent, the field of view cone 303 of the intraocular camera 101 begins to be cut into by the occlusion boundary 304 of the bionic eyelid 102 or the orbital silicone skin, forming an occlusion area 307. One possible calibration process is as follows: place the eyeball in a neutral position, gradually increase the rotation angle along a single degree of freedom, and observe whether silicone occlusion appears at the edge of the image of the intraocular camera 101 at each step; record the rotation angle at the first occurrence of occlusion, and backtrack one step as the travel boundary in that direction. After calibration in all four directions, the safe travel range of the field of vision 305 is obtained: vertical degrees of freedom [14, 46], horizontal degrees of freedom [0, 46]. The clamping operation in step 207 during runtime ensures that the eyeball will not enter the occlusion zone 307 in any control cycle, thus ensuring that the field of vision availability is close to 100% during fixation tracking. This calibration can be repeated after replacing the silicone face mask or adjusting the camera installation. The calibration results are saved in configuration form on the host computer 105.

[0045] IV. The process of facial expression control, also known as facial expression mirroring and coordination constraints. Expression mirroring and gaze following share the same camera and face detection front end in steps 201 to 202, without adding any sensors. Facial key points are extracted and expression features are calculated for the target face. Facial key point detection libraries such as MediaPipe Face Mesh and dlib can be used to locate feature points of the eyelids, eyebrows, cheeks, etc., from the image. Then, the opening and closing degrees and displacements of the upper and lower eyelids, eyebrows, and cheeks are obtained from the distances and displacements between these feature points. The "redirection mapping" described is not an existing general redirection method, but rather a scale-invariant cross-identity redirection method proposed in this invention for two sets of objects with different morphologies, joint numbers, and motion ranges: the "communication object's face" and the "bionic robot face." The detailed process is as follows.

[0046] Firstly, scale-invariant feature extraction: Existing methods often directly drive the image by the pixel coordinates or displacement of facial key points, which may mistakenly interpret the overall scaling caused by the interaction object moving closer or further away from the camera as "the expression getting bigger or smaller", resulting in image distortion. This invention first subtracts the centroid of each set of key points to remove the translation of the face in the image, and then divides it by the distance between the outer corners of the eyes for normalization. Since the distance between the outer corners of the eyes is basically unchanged by expressions such as blinking, opening the mouth, and raising eyebrows, and only scales with the distance of the face, the normalized result is a scale-invariant feature that only reflects the change in expression and is independent of the distance of the face.

[0047] Secondly, cross-identity joint-by-joint mapping: Since the number of joints and motion range of the communication object and the robot are not the same, this invention establishes a separate mapping from the scale-invariant feature to the joint angle for each facial joint of the robot. Taking the eyelid opening and closing degree o (value from 0 to 1, 0 for fully closed and 1 for fully open) as an example, the angle of the eyelid servo 104 is obtained by linear mapping = lower joint limit + o × (upper joint limit − lower joint limit), where the lower joint limit and the upper joint limit are the endpoints of the movable range obtained by the calibration of the eyelid servo. The eyebrow raising and lowering are written into the angle range [lower joint limit, upper joint limit] of the facial expression servo 108 respectively by the eyebrow feature and the cheek movement is written into the angle range [lower joint limit, upper joint limit] of the facial expression servo 108 respectively by the linear mapping of the eyebrow feature. For joints with left and right symmetry (such as the upper and lower eyelids of one eye), they are mapped independently and then issued in the same cycle to ensure left and right coordination.

[0048] Third, the stability brought by the constant scale: Since the influence of the distance of the face has been eliminated in the first step, the joint angles mapped by the same expression remain consistent when the communication object is close to or far from the camera, and the expression can be stably cloned without the other party maintaining a fixed distance.

[0049] The aforementioned "scale invariant + cross-identity joint-by-joint mapping" redirection method is the core difference between this invention and existing technologies, and it is also a key improvement that enables real human expressions to directly and stably drive the robot's face. It is the core technology proposed in this invention.

[0050] Specifically, the coordination limitation (step 209) involves comparing the mapped upper and lower eyelid closure amplitudes with a preset upper limit for visible closure when gaze tracking is active. Any excess is truncated to ensure the bionic eyelid 102 does not obscure the lens of the intraocular camera 101. The "preset upper limit for visible closure" is a core parameter distinguishing this solution from existing technologies. It refers to the maximum allowable eyelid closure amplitude when gaze tracking is active, and its value is determined by calibration rather than arbitrarily specified. During calibration, the closure amplitude is gradually increased from full eyelid opening, while simultaneously observing whether the bionic eyelid 102 begins to intrude into the field of vision in the intraocular camera 101 image. The closure amplitude one level before the occlusion threshold is reached is taken as the upper limit, and calibrated in the same batch as the visual field safety travel. The eyebrow and cheek joints do not obstruct the field of vision, and their mirror mapping is unrestricted and executed in full. Thus, when the other party blinks, the robot blinks "indicatively" with a limited amplitude, which preserves the semantic meaning of the mirrored expression and maintains the continuous field of vision required for following. In a modified embodiment, when the other party closes their eyes for a long time (more than a preset duration), the robot is allowed to complete a full blink and freeze the follow update during the blink, and resume after the blink ends, thus taking into account both the integrity of the expression and the continuity of tracking.

[0051] V. The section about losing one's face returns to the center. like Figure 6As shown, the face presence signal 401 changes from present to absent at the moment of face loss 402. The circuit first keeps the current angle unchanged and enters the grace period 403.

[0052] The "grace period" is a specific parameter set in this scheme, which refers to the waiting time after the face disappears without any return-to-center action. It is used to tolerate the brief loss of the face (such as the other party turning their head quickly or momentarily obscuring it). In this embodiment, it is set to 1.2 seconds. If the face reappears during the grace period, the following will resume directly without any return-to-center action. If no face is detected by the end of the grace period, the angle at the moment of face loss is recorded, and "partial repositioning" (also known as "half-repositioning") is performed—that is, the eyeball is not fully repositioned to the neutral position at once, but only partially repositioned towards the neutral position, retaining the remaining portion of the original orientation. The repositioning ratio is determined by the "retention coefficient": the "retention coefficient" is a preset ratio between 0 and 1 defined in this scheme, representing the proportion of the original deflection orientation still retained after repositioning (a retention coefficient of 1 indicates no repositioning, and 0 indicates full repositioning). The partial repositioning target line 406 is uniquely determined by target = neutral position + (angle at the moment of face loss - neutral position) × retention coefficient, showing a clear linear relationship between it and the angle at the moment of face loss and the neutral position, rather than arbitrary interpolation; in this embodiment, the retention coefficient is 0.5, i.e., target = neutral position + (angle at the moment of face loss - neutral position) × 0.5, only half is retracted, retaining half of the original orientation; then, in each control cycle, exponential easing is performed with an interpolation factor of 0.25 (centering curve 404): current angle ← current angle + (target − current angle) × 0.25, gradually approaching the partial centering target rather than the neutral line 405. When the face reappears at recapture time 407, the closed loop continues to make incremental corrections from the current angle. The technical principle of this design is: a complete reset will cause the eyeball to rotate significantly. If the face reappears exactly in the original direction, it needs to deflect significantly again, forming a cycle of "deflection → occlusion → reset → deflection again" jitter; while partial centering not only alleviates the possible edge occlusion under large angles, but also keeps the eyeball orientation biased towards the direction where the face last appeared. The correction amount after recapture is halved. In actual tests, the reciprocating movement of the eyeball under the face loss-reappearance cycle is reduced from continuous oscillation to at most one easing.

[0053] VI. Order Issuance and Safe Implementation The servo control signals in step S5 include a series of servo control signals used to generate facial expressions, primarily for controlling the rotation angle of the servo motor. One possible servo control signal may send two types of target angles, with different sources and update methods, which are then sent together within the same control cycle. The first is the target angle of the eyeball, namely the target angle θ_v of the eyeball pitch degree of freedom eyes_vertical and the target angle θ_h of the eyeball yaw degree of freedom eyes_horizontal. The two are initialized with the calibrated neutral position (θ_v=31, θ_h=14, that is, the position where the eyes are looking straight ahead). The two are incrementally updated frame by frame according to the proportional control law in step S4, that is, θ_h ← θ_h + kp_h·ex, θ_v ← θ_v + kp_v·ey - the rotation increment Δθ=Kp·e of the current frame is added to the current target angle and accumulated. In this embodiment, kp_h=7.0 and kp_v=5.0 are taken. When the absolute value of the normalized deviation component is less than the dead zone of 0.03, the corresponding degree of freedom remains unchanged. After accumulation, they are clamped to the safe travel range of the field of view θ_v∈[14, 46] and θ_h∈[0, 46] respectively. Secondly, the target angles of facial joints, namely the target angles of the upper eyelid, lower eyelid, eyebrows, cheeks, etc., do not participate in the above incremental updates. Instead, they are directly calculated joint by joint using the scale-invariant cross-identity retargeting mapping of the target facial expression features extracted from the same frame (joint angle = lower joint limit + normalized feature value × (upper joint limit - lower joint limit)), which is a frame-by-frame direct assignment. The closing amplitude of the upper and lower eyelids, which may obstruct the field of view of the intraocular camera, is further truncated by the coordination limit of the visible closing upper limit. The reason for only performing incremental updates on the eyeball is that the intraocular camera 101 rotates with the eyeball 107, and the eyeball angle is within the closed loop of "rotation → image deviation change → further rotation", which must be converged iteratively frame by frame according to the deviation. However, the facial joints do not participate in this closed loop and do not have corresponding feedback deviation values. Their angles are directly determined by the expression of the person being interacted with, so an open-loop direct mapping is used. The two types of target angles are merged into a single dictionary of joint names to angle values ​​and then distributed all at once. The facial servos and eye servos used to generate expressions are driven by the control box in the same way as described below.

[0054] The servo configuration and motion parameters of this embodiment are as follows: The intraocular camera 101 is installed inside the eyeball and rotates with the eyeball; the eyeball 107 is driven by two servos, one of which controls the pitch degree of freedom eyes_vertical, with a mechanical range of 0 to 80 degrees, of which the safe travel distance of the field of vision is 14 to 46 degrees, and the other controls the yaw degree of freedom eyes_horizontal, with a mechanical range of 0 to 60 degrees, of which the safe travel distance of the field of vision is 0 to 46 degrees; each eye is also equipped with one upper eyelid servo and one lower eyelid servo to drive the opening and closing of the bionic eyelid 102. The target angle of the eyelid servo is obtained by linear mapping of the eyelid opening and closing degree through embodiment four and falls within the calibrated range of its respective motion; in addition, there are several facial expression servos 108 such as eyebrows and cheeks (more than twenty facial joints in total), and the angle of each joint is expressed in terms of 0 to 180 degrees. The movement of each servo is driven by a pulse width modulation signal with a period of 20 milliseconds. The high-level pulse width is linearly corresponding to the servo's rotation angle from 0 to 180 degrees in the range of 0.5 to 2.5 milliseconds. For example, a pulse width of 1.5 milliseconds corresponds to approximately 90 degrees. The host computer 105 merges the two degrees of freedom of the eyeball and the target angle of the facial joints into a dictionary of joint names to angle values, and sends it to the control box 106 (bound to tcp: / / *:8765) in ZMQ request-response mode. The minimum interval between two consecutive sends is 20 milliseconds. The control box 106 splits the command and writes it to each servo control board according to the preset joint to board / register mapping: Specifically, the control box converts the target angle (0 to 180 degrees) of each joint into the PWM control quantity to drive the servo according to the calibration relationship of the board and register to which the joint belongs, that is, the duty cycle / pulse width of the pulse width modulation signal (typically, a high-level pulse width of 0.5 to 2.5 milliseconds within a 20-millisecond period corresponds to the servo's rotation angle of 0 to 180 degrees), and then writes it to the corresponding servo control board. Before writing, each angle is checked for safety limits. Angles that exceed the limits are discarded, forming the third layer of protection after the visual field safety travel clamping and coordinated limit cutoff.

[0055] VII. Facial expression control or generation methods, also known as coordination with autonomous micro-behaviors and operational management. One possible method for controlling or generating facial expressions is to schedule spontaneous blinking and eyebrow micro-movements according to the interaction state: 3 to 6 seconds for the idle state, 6 to 12 seconds for the listening state, 8 to 16 seconds for the thinking state, and 5 to 10 seconds for the speaking state. During the activation of coordinated control, the owner of the two degrees of freedom of the eyeball (the only program allowed to write commands to the same joint at the same time) is registered as the gaze-following loop. Writing to the eyeball by autonomous micro-behaviors is discarded because it is not the owner, avoiding mutual interference between spontaneous micro-movements and following. Spontaneous blinking can still be executed, but its closing amplitude is also constrained by the coordination limitation of step 209 to ensure that the visual interruption time at the moment of blinking is minimized. The operation interface provides a follow switch: checking it starts the image receiving and detection thread of the intraocular camera 101 and takes over the owner of the eyeball; unchecking it or switching back to a neutral posture stops following, releases the owner, and stops the image thread. During the follow activation, the components involving the two degrees of freedom of the eyeball in the preset posture button are shielded to prevent the posture switch from pulling the eyeball away from the target. The image source can be selected between the robot's binoculars (image stream forwarded by the control box 106) and an external USB camera. When an external camera is selected, the normalization deviation calculation method is the same, and the stroke clamping is still performed according to the calibration value of the intraocular camera to ensure safety.

[0056] VIII. Implementation of Modifications (a) Monocular Deformation: Only one intraocular camera (101) is used for detection and tracking, with no change in method steps, suitable for cost-sensitive single-camera scenarios. (b) Detector Replacement: YuNet can be replaced with other detection models that output face bounding boxes and confidence scores, with no change in normalization bias calculation and control law. (c) Multi-Camera Adaptation: Neutral position and field-of-view safety travel values ​​for both black-headed and white-headed cameras are calibrated and written into the configuration, and the control program is shared; clamping intervals and inverted flags are automatically loaded with the camera's identity. (d) Expression Mirror Range Expansion: Mouth movements are included in the mirror beyond eyelids, eyebrows, and cheeks, and mouth joints do not obstruct the field of view or have coordination limitations. (e) Control Law Expansion: A differential term is added to the proportional control to suppress overshoot of fast-moving targets, with no change in dead zone and clamping logic. (f) Target Memory Expansion: Combines face recognition to bind the identity of locked targets. After a target is briefly obscured, it prioritizes re-locking faces with the same identity, further reducing target jumps in multi-person scenarios.

[0057] In the above embodiments, gaze tracking (steps 204 to 207) and expression mirroring (steps 208 to 209) are completed serially within the same frame and sent out at once. They share the detection results, and the additional computing power overhead per frame is only one redirection mapping. Actual performance: Face detection time per frame is less than 15 milliseconds at 0.5x downscaling, and the total time for redirection mapping and clamping calculation is less than 1 millisecond. On ordinary desktop processors, the total overhead of detection and control per frame is much smaller than the frame interval corresponding to an image rate of 4 to 10 frames per second, with sufficient margin. After a face deviates from the center, it is usually pushed back into the dead zone within 3 to 5 frames. Since incremental control takes convergence within the dead zone as a static condition and does not accumulate mapping errors, there is no cumulative drift even after several hours of continuous operation. In the face loss-reappearance loop scenario, the reciprocating eye movement is reduced from continuous oscillation of the absolute mapping scheme to at most one easing.

[0058] This invention also provides a bionic robot facial expression control system, comprising: The image acquisition module is used to acquire and obtain image frames output by the binocular camera installed inside the rotatable eyeball; The face detection module is used to perform face detection on the image frame to obtain the target face bounding box; The gaze-following module is used to calculate the normalized deviation of the center of the target face frame relative to the center of its image, incrementally update the target angle of the vertical and horizontal degrees of freedom of the eye according to the proportional control law formula, and clamp it within the pre-calibrated visual field safety range. The expression mirroring module is used to extract the facial expression features of the target person from the same image frame, redirect the expression features to the target angle of the facial joints and generate servo control signals, and apply coordinated restrictions to the amplitude of eyelid closing movements that may obstruct the field of view of the binocular camera. The face-lost-centering module is used to control the target angle of the eye to slowly move towards the partially centered target when the duration of no face detection exceeds a preset grace period; The command sending module is used to send the servo control signals to the control box to drive the eyeball servo, eyelid servo, and other facial servo to perform robot expression generation.

[0059] Since the above modules have been described in the specific embodiments above, no further explanation will be given here.

[0060] Furthermore, when the processes described above in the embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0061] In this invention, unless otherwise explicitly specified and limited, the first feature being "on" or "below" the second feature can mean that the first feature and the second feature are in direct contact, or that the first feature and the second feature are in indirect contact through an intermediate medium.

[0062] Furthermore, "above," "on top of," and "above" the first feature in relation to the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. Similarly, "below," "under," and "beneath" the first feature in relation to the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0063] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling facial expressions in a bionic robot, characterized in that, Includes the following steps: S1. Acquire image frames from the stereo camera; S2. Based on the image frame, generate a target face bounding box and extract the target face expression features; S3. Based on the target face bounding box, generate the normalized deviation e(ex, ey) of the center of the target face bounding box relative to the image center, where ex represents the horizontal degree of freedom and ey represents the vertical degree of freedom; S4. Incrementally update the target angles of the vertical and horizontal degrees of freedom of the binocular cameras according to the following proportional control law formula: Δθ=Kp·e Where: e is the normalized deviation e(ex, ey) in S3, Kp is the preset proportional gain, and Δθ is the rotation increment superimposed on the target angle of the corresponding degree of freedom in this frame; S5. Based on the angle increment and the target facial expression features, clamp the target angle within the pre-calibrated safe field of view and generate a servo control signal, which is used to generate robot expressions.

2. The facial expression control method according to claim 1, characterized in that, The method for extracting the target facial expression features in step S2 is: facial landmark detection.

3. The facial expression control method according to claim 1, characterized in that, The formation of a safe field of view in step S5 is predetermined through the following steps: S501. Starting from the neutral position, gradually increase the eyeball rotation angle and observe the occlusion of the bionic eyelid or silicone skin in the intraocular camera image. S502. The maximum eyeball turning angle before the occlusion occurs is used as the travel boundary to calibrate the safe travel distance of the field of vision.

4. The facial expression control method according to claim 3, characterized in that, Step S5, especially after step S502, also includes: S503. Coordinate and limit the eyelid closing action of the eyeball, that is, when the gaze tracking is active, limit the closing amplitude of the upper and lower eyelids obtained by the redirection mapping to no more than a preset upper limit of visible closing. The upper limit of visible closing is determined by calibration and is the closing amplitude when the eyelid begins to block the field of view of the intraocular camera. Any part exceeding this limit is truncated.

5. The facial expression control method according to claim 4, characterized in that, Step S5, especially after step S503, also includes: S504. When the duration of no face detection exceeds the preset grace period, the target angle is controlled to slowly move towards the partially centered target from the moment the face is lost. The partial centering target is the neutral position plus the difference between the angle at the moment of loss of face and the neutral position multiplied by a preset retention coefficient; the retention coefficient is a proportional value between 0 and 1, representing the proportion of the original deflection orientation retained after centering.

6. The facial expression control method according to claim 5, characterized in that, Step S5 is followed by: S6. Perform a safety limit check on the servo control signal, and send the servo control signal that has passed the safety limit check to each servo.

7. The facial expression control method according to claim 6, characterized in that, Step S6 also includes discarding the servo control signal that fails the safety limit check.

8. A biomimetic robot facial expression control system, characterized in that, include: The image acquisition module is used to acquire and obtain image frames output by the binocular camera installed inside the rotatable eyeball; The face detection module is used to perform face detection on the image frame to generate a target face bounding box; The gaze-following module is used to calculate the normalized deviation of the center of the target face frame relative to the center of its image, incrementally update the target angle of the vertical and horizontal degrees of freedom of the eye according to the proportional control law formula, and clamp it within the pre-calibrated visual field safety range. The expression mirroring module is used to extract the facial expression features of the target person from the same image frame, redirect the expression features to the target angle of the facial joints and generate servo control signals, and apply coordinated restrictions to the amplitude of eyelid closing movements that may obstruct the field of view of the binocular camera. The face-lost-centering module is used to control the target angle of the eye to slowly move towards the partially centered target when the duration of no face detection exceeds a preset grace period; The command sending module is used to send the servo control signals to the control box to drive the eyeball servo, eyelid servo, and other facial servo to perform robot expression generation.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the facial expression control method according to any one of claims 1 to 7.

10. A computer-calibrated storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the facial expression control method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Humanoid-head robot device with human-computer interaction function and behavior control method thereof

    CN101618280B

  • Method for learning and embodying human facial expression by robot

    US11185990B2