Facial expression recognition and imitation algorithm for virtual character or robot

Abstract

The invention discloses a facial expression recognition and simulation algorithm for a virtual character or a robot, and belongs to the field of image recognition and simulation, and the algorithm mainly comprises the steps: inputting a cut face image into a depth model, recognizing face key points in the face image through the depth model, and carrying out the recognition and simulation of the face key points; and obtaining a head pose parameter, and predicting a value of a facial expression according to the facial key points. Calculating the head pose of the robot: calculating a rotation matrix through the head pose parameters, and converting the rotation matrix into an Euler angle of the head pose of the robot; mapping the predicted facial expression value into a rotation angle of a robot facial control steering engine; therefore, the robot is controlled to operate. According to the method, the micro-expression robot expression simulation data with relatively small dynamic amplitude can be obtained, a large amount of robot identification simulation data can be conveniently and accurately mapped into a robot entity from a large amount of video expressions, and the overall head pose is linked, so that the expression expression is more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition and imitation, and in particular to an algorithm for imitating facial expressions of virtual characters or robots. Background Art

[0002] With the development of artificial intelligence technology, intelligent human-computer interaction technology has become more and more interesting, so it is widely loved by users, but traditional intelligent human-computer interaction technology is limited to voice and image interaction, which can no longer meet the diversified needs of users; therefore, emerging expression recognition and interaction technology and eye tracking interaction technology have been born, but these technologies all need a carrier to better present. The traditional presentation carrier is generally an AI virtual character, but the current simulated anthropomorphic expression robot is undoubtedly the best carrier for its human-computer interaction. It can not only integrate and present various human-computer interaction technologies in a diversified manner, but also is the most intimate and natural interaction carrier for users. Therefore, mature expression robot products can be widely used in the fields of education assistance, medical companionship and care, customer guidance services, entertainment demonstrations, and social interaction in the future.

[0003] However, the current expression robot technology still needs a large amount of expression recognition and imitation data to make the robot's expression interaction more natural and vivid. In the prior art, a patent document with a publication number of CN116597484A discloses a robot real-time expression imitation method and device. The method can obtain the key point positions of all faces in the image by identifying the face image to be detected, and determine the face to be imitated in the face image to be detected according to these key point positions. The current frame of the face image to be detected is compared with the previous frame in real time to obtain the transformation trend and transformation ratio of the face to be imitated. The transformation trend is used to adjust the robot's head rotation angle, and the transformation ratio is used to calculate the rotation angle and running time of all the robot's servos, so that the robot's servos can control the robot's movements and imitate the expressions of the face to be imitated. Although the above scheme can provide robot expression imitation, it is difficult to provide a large amount of expression recognition imitation data by adopting the real-time expression imitation method. It obtains the transformation trend and transformation ratio of the coordinates of all key points of the face to be imitated, and adjusts the robot according to the transformation trend. It is necessary to realize the imitation data collection in the expression with a large dynamic amplitude. However, for some micro-expressions with a small dynamic amplitude, the above scheme is difficult to reflect the imitation action on the robot, and thus it is difficult to obtain the robot expression imitation data of micro-expressions with a small dynamic amplitude. Summary of the invention

[0004] In order to make up for the deficiencies in the existing image recognition and imitation technologies, the present invention proposes a robot expression imitation data that can obtain micro-expressions with a small dynamic amplitude, and is convenient for accurately mapping a large number of video expressions to robot entities to obtain a large number of robot recognition and imitation data, and linking the overall head posture to make the expression more accurate. At the same time, the present invention can also associate virtual characters to perform synchronous expression recognition and imitation, conduct joint interaction, or conduct comparative reference observation experiments to achieve a virtual character or robot facial expression recognition and imitation algorithm.

[0005] The specific technical solutions are as follows:

[0006] An algorithm for robot facial expression recognition and imitation is provided, comprising the following steps:

[0007] S1, cyclically collect face images and perform cropping preprocessing on the face images;

[0008] S2, input the cropped face image into the deep model mediapipe, identify the facial key points in the face image through the deep model mediapipe, and obtain the head pose parameters, and then predict the value of the facial expression based on the facial key points;

[0009] S3. Calculate the robot head pose: establish three basic rotation matrices through the head pose parameters, the product of the three basic rotation matrices is the rotation matrix, and then transform the rotation matrix into three Euler angles corresponding to the robot head pose;

[0010] S4, mapping the predicted facial expression value within the predicted original data set to the target data set of the robot facial control steering gear rotation, thereby obtaining the rotation angle of the robot facial control steering gear;

[0011] S5. Finally, according to the Euler angle and the rotation angle, the robot head posture and the robot face control servo rotation are controlled.

[0012] Furthermore, the robot head pose is calculated in S3 as follows:

[0013] According to the depth model mediapipe, the head pose parameters are α, β and γ, and the three basic rotation matrices are established as follows:

[0014]

[0015]

[0016]

[0017] Then, the product of the three basic rotation matrices is the rotation matrix RM= ,Finally, the rotation matrix is ​​transformed into three Euler angles corresponding to the robot’s head poses.

[0018] Furthermore, in S1, face images are cyclically acquired through an image acquisition device.

[0019] Furthermore, filtering is performed on the values ​​of the Euler angle and the rotation angle obtained in S3 and S4 respectively.

[0020] Furthermore, the filtering processing method is mean filtering processing, which calculates the average value of the angle values ​​generated in the adjacent time of the occurrence time of the currently processed angle value, and replaces the currently processed angle value with the obtained average value to adjust the robot head posture control or the robot face control steering gear rotation, specifically:

[0021]

[0022] Among them, the To obtain a stable angle value after filtering, is the angle value currently being processed, Based on The selected neighborhood window size, for The angle values ​​obtained in adjacent times.

[0023] Further, among them The value range is 5-10.

[0024] Furthermore, the deep model mediapipe recognizes 52 facial key points in the face image, and 3 head pose parameters are obtained. Then, the facial expression values ​​predicted by the facial key points are also 52.

[0025] The invention also provides an algorithm for imitating facial expression recognition of virtual characters, which selects 52 shape keys and 3 head shape poses related to the human face in the blender software, and binds them to the corresponding positions of the head of the virtual character respectively, then forms a mapping relationship between the 3 head pose parameters obtained by recognizing the human face image through the deep model mediapipe and the 3 head shape poses in the blender software, and forms a mapping relationship between the values ​​of 52 facial expressions predicted by the facial key points and the 52 shape keys selected in the blender software, thereby realizing the binding control of the 52 shape keys of the facial expression of the virtual character and the 3 head shape poses.

[0026] The beneficial effects of the present invention are: it is possible to obtain robot expression imitation data of micro-expressions with a small dynamic amplitude, and it is convenient to accurately map a large number of video expressions to robot entities to obtain a large number of robot recognition and imitation data, and link the overall head posture to make the expression of expression more accurate. At the same time, it is also possible to associate virtual characters for synchronous expression recognition and imitation, conduct joint interaction, or conduct comparative reference observation experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0028] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0029] Example:

[0030] like Figure 1 As shown: An algorithm for robot facial expression recognition and imitation includes the following steps:

[0031] S1. The face images are collected cyclically by an image acquisition device, or directly extracted by a video screen, and the face images are cropped and preprocessed.

[0032] S2. Input the cropped face image into the deep model mediapipe (an open source cross-platform framework developed by Google, mainly used to build multimedia processing and machine learning applications), identify the facial key points in the face image through the deep model mediapipe, and derive the head pose parameters, and then predict the value of the facial expression based on the facial key points.

[0033] Among them, there are 52 facial key points in the face image identified by the deep model mediapipe, and 3 head pose parameters are obtained. Then, the facial expression values ​​predicted by the facial key points are also 52.

[0034] S3. Calculate the robot's head pose: Establish three basic rotation matrices through the head pose parameters. The product of the three basic rotation matrices is the rotation matrix. Then the rotation matrix is ​​converted into three Euler angles corresponding to the robot's head pose, linking the overall head pose to make the expression more accurate and facilitate the precise mapping of the subsequent facial expression values.

[0035] The robot head pose is calculated in S3 as follows:

[0036] According to the depth model mediapipe, the head pose parameters are α, β and γ, and the three basic rotation matrices are established as follows:

[0037]

[0038]

[0039]

[0040] Then, the product of the three basic rotation matrices is the rotation matrix RM= ,Finally, the rotation matrix is ​​transformed into three Euler angles corresponding to the robot’s head poses.

[0041] The rotation matrix is ​​converted into three Euler angles corresponding to the robot head posture as follows:

[0042] If the rotation matrix is ​​set to:

[0043]

[0044] Then calculate the shaking angle for:

[0045] ,like When the rotation is specific, it is necessary to determine The value of .

[0046] Calculate the pitch angle for:

[0047] , whose value range is .

[0048] Calculate the tilt head angle for:

[0049] ,like When the rotation is specific, it is necessary to determine The value of .

[0050] If the rotation order is different, the formula for calculating the Euler angle will also be different. The calculation order above is Z - Y - X. If it is X - Y - Z, the calculation process and formula will change; and in practical applications, due to the universal lock problem of Euler angles, that is, when the pitch angle is This will result in a loss of one degree of freedom, and special attention should be paid to this issue during conversion and use.

[0051] S4. According to the position of the predicted facial expression value within the range of the original data set, it is mapped to the target data set of the robot facial control servo rotation, so as to obtain the rotation angle of the robot facial control servo. For example, the minimum and maximum values ​​of the facial expression predicted by the deep model mediapipe for recognizing the face image are 0 and 1 respectively, that is, the limit value range is , and assuming that the rotation limit range of the servo is 0°~90°, when the actual predicted value of the facial expression is 0.1, the rotation angle of the robot facial control servo is 9°; when the actual predicted value of the facial expression is 0.5, the rotation angle of the robot facial control servo is 45°, and so on. This mapping method can obtain robot expression imitation data of micro-expressions with small dynamic amplitude, and has low requirements for image quality. It is easy to extract face images from a large number of existing videos or film and television dramas with rich expressions, accurately map them to robot entities to obtain a large amount of robot recognition imitation data, and based on the first determination of the head posture, it can more accurately extract and express facial expressions.

[0052] For the values ​​of Euler angle and rotation angle obtained in S3 and S4, filtering is performed to avoid jitter caused by excessive movement of the robot. The filtering method is mean filtering. The angle values ​​generated in the adjacent time of the occurrence time of the currently processed angle value are averaged, and the average value is used to replace the currently processed angle value to adjust the robot head posture or the robot face to control the rotation of the servo. Specifically:

[0053]

[0054] Among them, the To obtain a stable angle value after filtering, is the angle value currently being processed, Based on The selected neighborhood window size is The value ranges from 5 to 10. Taking 28 frames per second, the most common video processing, as an example, The value is 5. Because it involves taking values ​​from subsequent time, there is a certain delay, but the delay should not be too large, otherwise it will easily cause obvious sluggishness in human-computer interaction. for The angle values ​​obtained in adjacent times.

[0055] S5. Finally, according to the Euler angle and the rotation angle, the robot head posture and the robot face control servo rotation are controlled.

[0056] The present invention also provides an algorithm for imitating facial expression recognition of virtual characters, which selects 52 morphological keys and 3 head morphological postures of human faces in blender software (3D computer graphics software), and binds them to corresponding positions of the head of the virtual character respectively, and then forms a mapping relationship between the 3 head morphological posture parameters obtained by recognizing the human face image through the deep model mediapipe and the 3 head morphological postures in the blender software. The mapping relationship is relatively simple and is generally directly input. The values ​​of 52 facial expressions predicted by facial key points are mapped to the 52 morphological keys selected in the blender software, thereby realizing the binding control of the 52 morphological keys of the facial expressions of the virtual character and the 3 head morphological postures. The virtual character and the robot entity participate in the expression recognition imitation in parallel at the same time, and then the virtual character can be associated to perform synchronous expression recognition imitation, conduct joint interaction or conduct comparative reference observation experiments.

[0057] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims.

Claims

1. A robot facial expression recognition and imitation algorithm, characterized in that: The following steps are involved: S1, cyclically collect face images and perform cropping preprocessing on the face images; S2, input the cropped face image into the deep model mediapipe, identify the facial key points in the face image through the deep model mediapipe, and obtain the head pose parameters, and then predict the value of the facial expression based on the facial key points; S3. Calculate the robot head pose: establish three basic rotation matrices through the head pose parameters, the product of the three basic rotation matrices is the rotation matrix, and then transform the rotation matrix into three Euler angles corresponding to the robot head pose; S4, mapping the predicted facial expression value within the predicted original data set to the target data set of the robot facial control steering gear rotation, thereby obtaining the rotation angle of the robot facial control steering gear; S5. Finally, according to the Euler angle and the rotation angle, the robot head posture and the robot face control servo rotation are controlled.

2. The robot facial expression recognition and imitation algorithm according to claim 1, characterized in that: The robot head pose is calculated in S3 as follows: According to the depth model mediapipe, the head pose parameters are α, β and γ, and the three basic rotation matrices are established as follows: Then, the product of the three basic rotation matrices is the rotation matrix RM= ,Finally, the rotation matrix is ​​transformed into three Euler angles corresponding to the robot’s head poses.

3. The robot facial expression recognition and imitation algorithm according to claim 1 or 2, characterized in that: In S1, face images are cyclically acquired through an image acquisition device.

4. The robot facial expression recognition and imitation algorithm according to claim 1 or 2, characterized in that: The values ​​of the Euler angle and the rotation angle obtained in S3 and S4 are filtered.

5. The robot facial expression recognition and imitation algorithm according to claim 4, characterized in that: The filtering method is mean filtering, which calculates the average value of the angle values ​​generated in the adjacent time of the current angle value, and replaces the current angle value with the average value to adjust the robot head posture or the robot face to control the steering gear rotation. Specifically: Among them, the To obtain a stable angle value after filtering, is the angle value currently being processed, Based on The selected neighborhood window size, for The angle values ​​obtained in adjacent times.

6. The robot facial expression recognition and imitation algorithm according to claim 5, characterized in that: in The value range is 5-10.

7. The robot facial expression recognition and imitation algorithm according to any one of claims 1, 2, 5 or 6, characterized in that: The deep model mediapipe recognizes 52 facial key points in the face image, and 3 head pose parameters are obtained. Then, the facial expression values ​​predicted by the facial key points are also 52.

8. An algorithm for imitating facial expression recognition of virtual characters, characterized in that: By selecting 52 shape keys and 3 head shape poses related to the human face in the blender software, and binding them to the corresponding positions of the virtual character's head respectively, then mapping the 3 head pose parameters obtained by recognizing the face image through the deep model mediapipe with the 3 head shape poses in the blender software, and mapping the values ​​of the 52 facial expressions predicted by the facial key points with the 52 shape keys selected in the blender software.

Images