Expression driving method of three-dimensional digital person and electronic equipment

Abstract

The application provides an expression driving method of a three-dimensional digital person and an electronic device. The application is used to improve the accuracy of expression driving of a three-dimensional digital person. The method comprises the following steps: performing face detection on a current frame image to obtain bounding boxes of multiple faces, wherein the current frame image contains a target object and the target object in a mirror; performing bounding box clipping on the current frame image by using the bounding boxes of the multiple faces to obtain multiple face images, wherein the multiple face images include real face images and mirror face images; if, based on feature point extraction on the multiple face images, the position coordinates of multiple feature points in the multiple face images and the confidence of the multiple feature points are obtained, it is determined that no valid face region can be detected in the current frame image; and then, the expression parameters of the target object in the current frame image are obtained according to the position coordinates of the multiple feature points in the mirror face images, and the expression driving of a three-dimensional digital person that is set in advance is performed by using the expression parameters.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a method and electronic device for driving the facial expressions of a three-dimensional digital human. Background Technology

[0002] In remote 3D communication, 3D digital humans are the foundation for building such communication systems. One of the core elements is how to efficiently and accurately drive the facial expressions of the 3D digital human's head based on modeling. Furthermore, the presentation effect of the 3D digital human's facial expression driving and rendering on the terminal device directly affects the user's perception and experience.

[0003] Existing methods for driving facial expressions in 3D digital humans typically involve pre-constructing a parametric head model of the 3D digital human, using blendshape to calculate facial expression parameters from the current frame's facial image, and then using the calculated blended deformation facial parameters to globally update the facial expression on the parametric head model of the 3D digital human. Examples include expressions like opening the mouth and closing the eyes. However, blendshape's facial parameter calculation generally relies on the extraction of facial feature points, which requires high-quality facial data. If a valid facial region cannot be obtained (e.g., a user's profile or head turning), the determined facial parameters will be inaccurate, leading to low accuracy in driving facial expressions in 3D digital humans and resulting in driving anomalies. Summary of the Invention

[0004] This application provides a method and electronic device for driving the facial expressions of a 3D digital human, which improves the accuracy of facial expression driving in 3D digital humans and avoids driving abnormalities.

[0005] In a first aspect, embodiments of this application provide a method for driving the facial expressions of a three-dimensional digital human, the method comprising:

[0006] Face detection is performed on the current frame image to obtain bounding boxes of multiple faces in the current frame image, wherein the current frame image contains the target object and the target object in the mirror, and the mirror is positioned to the left and right of the target object;

[0007] The current frame image is cropped using the bounding boxes of the multiple faces to obtain multiple face images, wherein the multiple face images include real face images and mirrored face images;

[0008] Feature points are extracted from the multiple face images to obtain multiple feature points in the multiple face images and the confidence level of the multiple feature points;

[0009] Based on multiple feature points in the real face image and the confidence level of the multiple feature points, determine whether a valid face region can be detected in the current frame image;

[0010] If not, then based on multiple feature points in the mirrored face image, the expression parameters of the target object in the current frame image are obtained, and the expression parameters are used to drive the expression of the pre-set 3D digital human.

[0011] A second aspect of this application provides an electronic device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0012] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0013] Face detection is performed on the current frame image to obtain bounding boxes of multiple faces in the current frame image, wherein the current frame image contains the target object and the target object in the mirror, and the mirror is positioned to the left and right of the target object;

[0014] The current frame image is cropped using the bounding boxes of the multiple faces to obtain multiple face images, wherein the multiple face images include real face images and mirrored face images;

[0015] Feature points are extracted from the multiple face images to obtain multiple feature points in the multiple face images and the confidence level of the multiple feature points;

[0016] Based on multiple feature points in the real face image and the confidence level of the multiple feature points, determine whether a valid face region can be detected in the current frame image;

[0017] If not, then based on multiple feature points in the mirrored face image, the expression parameters of the target object in the current frame image are obtained, and the expression parameters are used to drive the expression of the pre-set 3D digital human.

[0018] According to a third aspect of the present invention, a computer storage medium is provided, the computer storage medium storing a computer program for performing the method as described in the first aspect.

[0019] In the above embodiments of this application, face detection is performed on the current frame image to obtain bounding boxes for the real face and the mirrored face in the current frame image. Then, based on each bounding box, the real face image and the mirrored face image are obtained. If, based on the position coordinates and confidence levels of multiple feature points in the real face image, it is determined that a valid face region cannot be detected in the current frame image, then the expression parameters of the target object in the current frame image are obtained based on the positions of multiple feature points in the mirrored face image. The expression parameters are then used to drive the expression of a pre-set 3D digital human. Therefore, in the embodiments of this application, when it is determined that a valid face region cannot be detected in the real face image, the expression parameters of the target object in the current frame image are obtained based on the position coordinates of multiple feature points in the mirrored face image. This allows for the determination of a valid face region, making the determined expression parameters more accurate, improving the accuracy of the expression driving of the 3D digital human, and avoiding driving anomalies. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 An exemplary illustration shows an application scenario diagram of the facial expression driving method for three-dimensional digital humans provided in an embodiment of this application;

[0022] Figure 2 An exemplary schematic diagram of one of the expression-driven processes for a three-dimensional digital human provided in this application embodiment is shown;

[0023] Figure 3 An exemplary diagram illustrates the placement relationship between the electronic device and the mirror provided in an embodiment of this application;

[0024] Figure 4 An exemplary illustration shows a flowchart of a process for determining whether a valid face region can be detected in the current frame image, provided in an embodiment of this application.

[0025] Figure 5 A schematic diagram illustrating the yaw angle and pitch angle provided in an embodiment of this application is shown as an example.

[0026] Figure 6 An exemplary illustration shows one of the flowcharts for determining facial expression parameters provided in an embodiment of this application;

[0027] Figure 7This document exemplarily illustrates a second schematic diagram of the specific process for determining facial expression parameters provided in an embodiment of this application.

[0028] Figure 8 This document exemplarily illustrates a second schematic diagram of the expression-driven process for a three-dimensional digital human provided in an embodiment of this application.

[0029] Figure 9 An exemplary diagram of the facial expression driving device for a three-dimensional digital human provided in an embodiment of this application is shown;

[0030] Figure 10 An exemplary hardware structure diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0031] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0032] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0033] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0034] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to be omnipresent but not exclusive; for example, a product or device comprising a series of components is not necessarily limited to those explicitly listed, but may include other components not explicitly listed or inherent to such product or device.

[0035] As used in this application, the term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0036] The following is an overview of the ideas behind the embodiments of this application.

[0037] In existing technologies, a parametric head model of a 3D digital human is typically pre-constructed. Blendshape is then used to calculate expression parameters on the current frame's facial image. The calculated hybrid deformation expression parameters are then used to globally update the expression on the parametric head model of the 3D digital human, such as expressions like opening the mouth or closing the eyes. However, Blendshape expression parameter calculation generally relies on the extraction of facial feature points, which requires high-quality facial data. If a valid facial region cannot be obtained (e.g., a user's profile or head turning), the determined expression parameters will be inaccurate. This leads to low accuracy in expression driving for the 3D digital human and can result in driving anomalies.

[0038] Due to the low accuracy of expression-driven techniques for 3D digital humans in existing technologies, this application provides a method for driving expressions in 3D digital humans. This method involves performing face detection on a current frame image to obtain bounding boxes for the real face and its mirror image. Based on these bounding boxes, real and mirror face images are obtained. If, based on the coordinates and confidence levels of multiple feature points in the real face image, a region of face cannot be detected in the current frame image, then the expression parameters of the target object in the current frame image are obtained based on the positions of multiple feature points in the mirror face image. These expression parameters are then used to drive the pre-defined expressions of the 3D digital human. Therefore, in this application, when a region of face cannot be detected in the real face image, the expression parameters of the target object in the current frame image are obtained based on the coordinates of multiple feature points in the mirror face image. This allows for the determination of effective face regions, making the determined expression parameters more accurate and improving the accuracy of expression-driven techniques for 3D digital humans, thus avoiding driving anomalies.

[0039] like Figure 1 The diagram shown illustrates an application scenario of the facial expression driving method for a 3D digital human according to an embodiment of this application. This application scenario uses an electronic device as the server for illustration. The application scenario includes a VR headset 110 and a server 120.

[0040] In one possible application scenario, the VR headset 110 acquires the current frame image of the target object and sends it to the server 120. Upon receiving the current frame image, the server 120 performs face detection on it, obtaining bounding boxes for multiple faces within the current frame image. The current frame image includes the target object and the target object reflected in a mirror, with the mirror positioned to the left and right of the target object. Then, the server 120 uses the bounding boxes of the multiple faces to crop the current frame image, obtaining multiple face images, including real faces. The server 120 extracts feature points from the multiple face images and mirrored face images respectively, obtaining multiple feature points and confidence levels of the multiple feature points in the multiple face images; then, based on the multiple feature points in the real face image and the confidence levels of the multiple feature points, the server 120 determines whether a valid face region can be detected in the current frame image; if not, the server 120 obtains the expression parameters of the target object in the current frame image based on the multiple feature points in the mirrored face image, and uses the expression parameters to drive the expression of the pre-set 3D digital human.

[0041] in, Figure 1 The server 120 and the VR headset 110 can interact via a communication network. The communication network can be either wireless or wired.

[0042] For example, server 120 can access the network via cellular mobile communication technology to communicate with VR headset 110, wherein the cellular mobile communication technology includes, for example, 5th generation mobile networks (5G) technology.

[0043] Optionally, the server 120 can access the network and communicate with the VR headset 110 via short-range wireless communication, such as Wireless Fidelity (Wi-Fi) technology.

[0044] The description in this application focuses on a single VR headset 110 and a single server 120. However, those skilled in the art should understand that the illustrated VR headset 110 and server 120 are intended to illustrate the operation of the VR headset 110 and server 120 involved in the technical solutions of this application, and do not imply any limitation on the number, type, or location of the VR headset 110 and server 120. It should be noted that adding additional modules to or removing individual modules from the illustrated environment will not change the underlying concept of the exemplary embodiments of this application.

[0045] It should be noted that the facial expression driving method for 3D digital humans proposed in this application is not only applicable to... Figure 1 The application scenarios shown also apply to any facial expression-driven device with a 3D digital human.

[0046] The following describes an exemplary embodiment of the three-dimensional digital human expression-driven method of this application, in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the methods and principles of this application, and the implementation of this application is not limited in any way in this respect.

[0047] like Figure 2 The diagram shown illustrates a flowchart of a three-dimensional digital human's expression-driven method, which may include the following steps:

[0048] Step 201: Perform face detection on the current frame image to obtain bounding boxes of multiple faces in the current frame image, wherein the current frame image contains the target object and the target object in the mirror, and the mirror is positioned to the left and right of the target object;

[0049] like Figure 3 As shown, the electronic device and the mirror are positioned in the embodiment of this application. The electronic device and the mirror are positioned in the embodiment of this application so that the camera of the electronic device can detect the real face and the face in the two mirrors. The face displayed in the mirror cannot be directly detected by the camera. For example, when the user turns his head to the right, the camera can only capture the left side of the user's face, but the camera cannot directly capture the right side of the user's face. However, the right side of the user's face can be displayed in the mirror on the right side of the user.

[0050] It should be noted that the placement relationship between the electronic device and the mirror described above in this application embodiment can be set according to the actual situation, and this application embodiment does not limit it. Furthermore, the number of mirrors can also be set according to the actual situation, and this application embodiment does not limit the specific number of mirrors.

[0051] In this embodiment, a face detection algorithm is used to detect faces in the current frame image. The face detection algorithm can be RCNN (Region Convolutional Neural Networks) or MTCNN (Multi-task convolutional neural network), etc. The specific face detection algorithm can be set according to the actual situation, and this embodiment does not limit the face detection algorithm.

[0052] Step 202: Use the bounding boxes of the multiple faces to crop the current frame image to obtain multiple face images, wherein the multiple face images include real face images and mirrored face images;

[0053] To improve accuracy, in one embodiment, before performing step 202, the size of the bounding boxes of the plurality of faces is transformed to a specified size. In this embodiment, the specified size is 256*256. However, this embodiment does not limit the specified size; the specified size can be set according to actual circumstances.

[0054] Step 203: Extract feature points from the multiple face images to obtain the position coordinates of multiple feature points in the multiple face images and the confidence level of the multiple feature points;

[0055] In this embodiment, a feature point extraction algorithm is used to extract feature points from a face image. This algorithm can be either SURF (Speeded-Up Robust Features) or SIFT (Scale Invariant Feature Transform). However, the specific feature extraction algorithm can be set according to the actual situation, and this embodiment does not limit the feature extraction algorithm.

[0056] Step 204: Based on the position coordinates of multiple feature points in the real face image and the confidence level of the multiple feature points, determine whether a valid face region can be detected in the current frame image;

[0057] like Figure 4 The diagram illustrates the process of determining whether a valid face region can be detected in the current frame image, including the following steps:

[0058] Step 401: Based on the position coordinates of multiple feature points in the real face image and the position coordinates of multiple standard feature points corresponding to the pre-set 3D digital human, obtain the rotation and translation matrix of the face in the real face image, wherein the number of multiple feature points is the same as the number of multiple standard feature points.

[0059] In one embodiment, step 401 can be specifically implemented as follows: inputting the position coordinates of multiple feature points of the real face image and the position coordinates of multiple standard feature points corresponding to the three-dimensional digital human into the singular value decomposition algorithm to obtain the rotation and translation matrix of the face.

[0060] In this embodiment, the rotation and translation matrix is ​​determined using a singular value decomposition algorithm, which is a method used in the prior art. This embodiment will not be described in detail here.

[0061] Step 402: Based on the rotation and translation matrix of the face, obtain the rotation angle of the face, wherein the rotation angle of the face includes the yaw angle and the pitch angle;

[0062] like Figure 5 The diagram illustrates the yaw and pitch angles in an embodiment of this application. In this embodiment, the center point of the face is used to establish a spatial rectangular coordinate system with the origin as the coordinate origin. The yaw angle in this embodiment is the angle by which the face rotates around the z-axis, i.e., the angle when the face turns left or right. The pitch angle is the angle by which the face rotates around the Y-axis, i.e., the angle when the face looks up or down.

[0063] In this embodiment, the rotation and translation matrix is ​​a 3x3 matrix, and the form of the rotation and translation matrix can be as follows: The yaw angle can be obtained using formula (1):

[0064]

[0065] Where ω is the yaw angle, r 21 r is the value in the second row and first column of the rotation and translation matrix. 11 This represents the value in the first row and first column of the rotation and translation matrix.

[0066] The pitch angle can be obtained using formula (2):

[0067] v = -sin -1 (r 31 )......(2);

[0068] Where v is the pitch angle, r 31 The value is in the third row and first column of the rotation and translation matrix.

[0069] Step 403: Determine whether the rotation angle of the face and the confidence level of the multiple feature points meet the specified conditions. If yes, proceed to step 405; otherwise, proceed to step 404.

[0070] In one embodiment, the rotation angle of the face and the confidence levels of the plurality of feature points are determined to meet specified conditions in the following manner:

[0071] If the yaw angle is within a first preset angle range, the pitch angle is within a second preset angle range, and the number of target feature points is greater than a specified number, then the rotation angle of the face and the confidence level of the multiple feature points are determined to meet the specified conditions, wherein the target feature points are feature points with a confidence level greater than a specified threshold; otherwise, the rotation angle of the face and the confidence level of the multiple feature points are determined not to meet the specified conditions.

[0072] In this embodiment, the first preset angle range is (-45°, 45°), and the second preset angle range is (-30°, 30°). However, this embodiment does not limit the first and second preset angle ranges; they can be set according to actual conditions. Furthermore, the specified quantity and threshold in this embodiment can also be set according to actual conditions; this embodiment does not limit the specified quantity and threshold.

[0073] Step 404: Determine that no valid face region can be detected in the current frame image;

[0074] Step 405: Determine the effective face regions that can be detected in the current frame image.

[0075] In one embodiment, if a valid face region can be detected in the current frame image, the expression parameters of the target object in the current frame image are obtained based on the position coordinates of multiple feature points in the real face image and the position coordinates of multiple standard feature points corresponding to the pre-set 3D digital human, and the expression parameters are used to drive the expression of the pre-set 3D digital human.

[0076] In this embodiment, the facial expression parameters are determined using the blendshape algorithm. The expression for the blendshape algorithm is shown in formula (3):

[0077]

[0078] Where F is the matrix corresponding to the position coordinates of multiple feature points in the real face image, B is the matrix corresponding to the position coordinates of multiple standard feature points corresponding to the pre-set 3D digital human, and β i For the pre-set i-th expression basis matrix, ω i Let be the expression parameters corresponding to the i-th expression basis matrix, and n be the total number of expression basis matrices.

[0079] In this embodiment, the size of the matrix corresponding to the position coordinates of multiple feature points in the real face image is 478*3, the size of the matrix corresponding to the position coordinates of multiple standard feature points is 478*3, and the size of any expression basis matrix is ​​478*3. The value of n is 52. The corresponding expression parameter ω can be solved using the above formula (3). i The specific solution method is the matrix solution method in the prior art, and will not be described in detail in the embodiments of this application.

[0080] Therefore, in the embodiments of this application, when a valid face region can be detected in the image, the expression parameters can be determined directly based on the real face image. At this time, a target mirror face image is not required. Thus, the embodiments of this application can reduce the amount of computation and improve the efficiency of expression-driven processing while ensuring accuracy.

[0081] Step 205: If not, then based on the position coordinates of multiple feature points in the mirrored face image, obtain the expression parameters of the target object in the current frame image, and use the expression parameters to drive the expression of the pre-set 3D digital human.

[0082] In this embodiment of the application, the number of mirrored face images is the same as the number of mirrors on both sides of the user, so as to... Figure 3 For example, if the number of mirrors on both sides of the user is 2, then the number of mirrored face images is determined to be 2. However, this embodiment of the application does not limit the number of mirrors, and can be set according to the actual situation.

[0083] like Figure 6 The diagram shown illustrates the process for determining facial expression parameters, including the following steps:

[0084] Step 601: Using the yaw angle of the face, determine the target mirror face image in the mirror face image;

[0085] The established rectangular coordinate system Figure 5 Taking the example in the example, if the yaw angle is positive, the target mirrored face image is determined to be the mirrored face image corresponding to the user's left side mirror; if the yaw angle is negative, the target mirrored face image is determined to be the mirrored face image corresponding to the user's right side mirror. The specific Cartesian coordinate system can be set according to the actual situation, and this embodiment does not limit it. If the direction of the Cartesian coordinate system changes, the corresponding target mirrored face image will change, which will not be elaborated further in this embodiment.

[0086] Step 602: If the yaw angle of the face is within the third preset angle range, then the expression parameters are determined based on the position coordinates of multiple feature points in the target mirror face image;

[0087] In one embodiment, step 602 can be specifically implemented as follows: for any one of the plurality of feature points, using a pre-set correspondence between mirror position coordinates and real position coordinates, the position coordinates of the feature point in the target mirror face image are transformed to obtain the real position coordinates of the feature point in the real face image; based on the real position coordinates of the plurality of feature points in the target mirror face image and the position coordinates of the plurality of standard feature points corresponding to the pre-set three-dimensional digital human, the expression parameters are obtained.

[0088] In this embodiment, the method of obtaining the expression parameters based on the real position coordinates of multiple feature points in the target mirror face image and the position coordinates of multiple standard feature points corresponding to the pre-set 3D digital human also uses the blendshape algorithm. The only difference is that F in formula (3) is replaced by the matrix corresponding to the position coordinates of multiple feature points in the real face image with the matrix corresponding to the real position coordinates of multiple feature points in the target mirror face image, and then the expression parameters are calculated. This embodiment will not be elaborated further here.

[0089] In this embodiment, the third preset angle range is greater than 60° or less than -60°. However, the specific range can be set according to the actual situation. This embodiment does not limit the third preset angle range.

[0090] Step 603: If the yaw angle of the face is within the fourth preset angle range, the expression parameters are obtained based on the position coordinates of multiple feature points in the target mirror face image and the position coordinates of multiple feature points in the real face image.

[0091] The fourth angle range in this embodiment is (-60°, -45°) and (45°, 60°). However, the fourth angle range in this embodiment can be set according to the actual situation, and this embodiment does not limit the fourth angle range.

[0092] Therefore, in this embodiment, the corresponding expression parameters can be determined by the angular range of the face's yaw angle. Specifically, when the face's yaw angle is within a third preset angle range, it indicates that a frontal face cannot be detected in the real face image. Therefore, expression parameters need to be determined based on the target mirror face image to ensure the accuracy of the determined expression parameters. When the face's yaw angle is within a fourth preset angle range, it indicates that a portion of the face can be detected in both the real face image and the target mirror face image. Therefore, the two need to be combined to determine the expression parameters to improve their accuracy.

[0093] like Figure 7The diagram shown illustrates the specific process for determining facial expression parameters in step 603, which may include the following steps:

[0094] Step 701: For any feature point, using the pre-set correspondence between the mirror position coordinates and the real position coordinates, transform the position coordinates of the feature point in the target mirror face image to obtain the real position coordinates of the feature point in the real face image.

[0095] Step 702: Based on the real position coordinates of multiple feature points in the target mirror face image and the position coordinates of multiple standard feature points corresponding to the pre-set 3D digital human, obtain the first intermediate expression parameters;

[0096] The method for determining the first intermediate expression parameter is the same as the method for determining the expression parameter in step 602 described above, and will not be repeated here in this embodiment.

[0097] Step 703: Obtain the second intermediate expression parameters based on the position coordinates of multiple feature points in the real face image and the position coordinates of the multiple standard feature points;

[0098] The method for determining the second intermediate expression parameter in this embodiment is the same as the method for determining the expression parameter in step 405 described above, and will not be repeated here.

[0099] Step 704: Perform a weighted summation of the first intermediate expression parameter and the second intermediate expression parameter to obtain the expression parameter.

[0100] In one embodiment, the weights of the first intermediate expression parameter and the second intermediate expression parameter can be determined in the following manner:

[0101] By utilizing the pre-set correspondence between the rotation angle of the face and the weight parameters, the weight parameters corresponding to the rotation angle of the face in the current frame image are determined. The weight parameters include the weight of the first intermediate expression parameter and the weight of the second intermediate expression parameter.

[0102] The weighting rules for the first and second intermediate expression parameters in this embodiment are as follows: If the face region in the real face image is larger than the face region in the target mirror face image (e.g., the yaw angle in the face rotation angle is within a first specified range), then the weight of the second intermediate expression parameter is set to be greater than the weight of the first intermediate expression parameter (e.g., the yaw angle is within a second specified range). If the face region in the real face image is smaller than the face region in the target mirror face image, then the weight of the second intermediate expression parameter is set to be less than the weight of the first intermediate expression parameter. Specific weight values ​​can be set according to specific circumstances, and this embodiment does not limit them further.

[0103] In this embodiment, both the first and second intermediate expression parameters are 52*1 matrices. For any two intermediate expression parameters that occupy the same position in the two matrices, the first and second intermediate expression parameters are weighted and summed to obtain the expression parameter corresponding to that position. For example, if the matrix corresponding to the first intermediate expression parameter is [a, b, c, ...] and the matrix corresponding to the second intermediate expression parameter is [m, n, e, ...], then the obtained expression parameter is [a*A + m*M, b*B + n*N, c*C + e*E, ...]. Here, A, M, B, N, C, and E are pre-set weights.

[0104] In one embodiment, the facial expression parameters in step 603 can also be obtained by formula (4):

[0105] ω i =β i (R*(στ+σ′τ′)+TB)....(4);

[0106] Where R is the rotation matrix of the face, T is the translation matrix of the face, σ is the matrix corresponding to the position coordinates of multiple feature points in the real face image, τ is the confidence matrix of the position coordinates of multiple feature points in the real face image, σ′ is the matrix corresponding to the position coordinates of multiple feature points in the target mirror face image, τ′ is the confidence matrix of the position coordinates of multiple feature points in the target mirror face image, B is the matrix corresponding to the position coordinates of multiple standard feature points corresponding to the pre-set 3D digital human, and β i For the pre-set i-th expression basis matrix, ω i Let be the expression parameters corresponding to the i-th expression basis matrix. The rotation and translation matrices in this embodiment are obtained based on the face rotation credential matrix, which is a method used in the prior art and will not be described further in this embodiment.

[0107] To further understand the methods in this application, such as Figure 8 The diagram shown is a flowchart of the facial expression driving method for a 3D digital human in this application, which may include the following steps:

[0108] Step 801: Perform face detection on the current frame image to obtain bounding boxes of multiple faces in the current frame image, wherein the current frame image contains the target object and the target object in the mirror, and the mirror is positioned to the left and right of the target object;

[0109] Step 802: Use the bounding boxes of the multiple faces to crop the current frame image to obtain multiple face images, wherein the multiple face images include real face images and mirrored face images;

[0110] Step 803: Extract feature points from the multiple face images to obtain the position coordinates of multiple feature points in the multiple face images and the confidence level of the multiple feature points;

[0111] Step 804: Based on the position coordinates of multiple feature points in the real face image and the position coordinates of multiple standard feature points corresponding to the pre-set 3D digital human, obtain the rotation and translation matrix of the face in the real face image, wherein the number of multiple feature points is the same as the number of multiple standard feature points.

[0112] Step 805: Based on the rotation and translation matrix of the face, obtain the rotation angle of the face, wherein the rotation angle of the face includes the yaw angle and the pitch angle;

[0113] Step 806: Determine whether the rotation angle of the face and the confidence level of the multiple feature points meet the specified conditions. If yes, proceed to step 807; otherwise, proceed to step 808.

[0114] Step 807: Based on the position coordinates of multiple feature points in the real face image and the position coordinates of multiple standard feature points corresponding to the pre-set 3D digital human, obtain the expression parameters of the target object in the current frame image;

[0115] Step 808: Using the yaw angle of the face, determine the target mirror face image in the mirror face image;

[0116] Step 809: Determine whether the yaw angle of the face is within the third preset angle range. If yes, proceed to step 810; otherwise, proceed to step 811.

[0117] Step 810: Determine the expression parameters based on the position coordinates of multiple feature points in the target mirror face image;

[0118] Step 811: For any feature point, using the pre-set correspondence between the mirror position coordinates and the real position coordinates, transform the position coordinates of the feature point in the target mirror face image to obtain the real position coordinates of the feature point in the real face image.

[0119] Step 812: Based on the real position coordinates of multiple feature points in the target mirror face image and the position coordinates of multiple standard feature points corresponding to the pre-set 3D digital human, obtain the first intermediate expression parameters;

[0120] Step 813: Obtain the second intermediate expression parameters based on the position coordinates of multiple feature points in the real face image and the position coordinates of the multiple standard feature points;

[0121] Step 814: Perform a weighted summation of the first intermediate expression parameter and the second intermediate expression parameter to obtain the expression parameter;

[0122] Step 815: Use the expression parameters to drive the pre-set three-dimensional digital human to express emotions.

[0123] Based on the same inventive concept, the facial expression driving method for a three-dimensional digital human as described above can also be implemented by a facial expression driving device for a three-dimensional digital human. The effect of this facial expression driving device for a three-dimensional digital human is similar to that of the aforementioned method, and will not be described again here.

[0124] Figure 9 This is a schematic diagram of the structure of a facial expression driving device for a three-dimensional digital human according to an embodiment of the present disclosure.

[0125] like Figure 9 As shown, the facial expression driving device 900 for a three-dimensional digital human disclosed herein may include a face detection module 910, a cropping module 920, a feature point extraction module 930, a judgment module 940, and an expression parameter determination module 950.

[0126] The face detection module 910 is used to perform face detection on the current frame image and obtain bounding boxes of multiple faces in the current frame image, wherein the current frame image includes a target object and the target object in a mirror, and the mirror is positioned to the left and right of the target object;

[0127] The cropping module 920 is used to crop the current frame image using the bounding boxes of the multiple faces to obtain multiple face images, wherein the multiple face images include real face images and mirrored face images;

[0128] The feature point extraction module 930 is used to extract feature points from the multiple face images respectively, and obtain the position coordinates of multiple feature points in the multiple face images and the confidence level of the multiple feature points;

[0129] The judgment module 940 is used to determine whether a valid face region can be detected in the current frame image based on the position coordinates of multiple feature points in the real face image and the confidence level of the multiple feature points.

[0130] The expression parameter determination module 950 is used to, if not, obtain the expression parameters of the target object in the current frame image based on the position coordinates of multiple feature points in the mirrored face image, and use the expression parameters to drive the expression of the pre-set three-dimensional digital human.

[0131] In one embodiment, the facial expression parameter determination module 950 is further configured to:

[0132] If a valid face region can be detected in the current frame image, then based on the position coordinates of multiple feature points in the real face image and the position coordinates of multiple standard feature points corresponding to the pre-set 3D digital human, the expression parameters of the target object in the current frame image are obtained, and the expression parameters are used to drive the expression of the pre-set 3D digital human.

[0133] In one embodiment, the determining module 940 is specifically used for:

[0134] Based on the position coordinates of multiple feature points in the real face image and the position coordinates of multiple standard feature points corresponding to a pre-set 3D digital human, the rotation and translation matrix of the face in the real face image is obtained, wherein the number of multiple feature points is the same as the number of multiple standard feature points.

[0135] Based on the rotation and translation matrix of the face, the rotation angle of the face is obtained, wherein the rotation angle of the face includes yaw angle and pitch angle;

[0136] If the rotation angle of the face and the confidence level of the multiple feature points meet the specified conditions, then it is determined that a valid face region can be detected in the current frame image.

[0137] Otherwise, it is determined that no valid face region can be detected in the current frame image.

[0138] In one embodiment, the determining module 940 is further configured to:

[0139] The rotation angle of the face and the confidence levels of the multiple feature points are determined to meet specified conditions using the following methods:

[0140] If the yaw angle is within a first preset angle range, the pitch angle is within a second preset angle range, and the number of target feature points is greater than a specified number, then the rotation angle of the face and the confidence of the multiple feature points are determined to meet the specified conditions, wherein the target feature points are feature points with a confidence greater than a specified threshold.

[0141] Otherwise, it is determined that the rotation angle of the face and the confidence level of the multiple feature points do not meet the specified conditions.

[0142] In one embodiment, the number of mirrored face images is the same as the number of mirrors;

[0143] The facial expression parameter determination module 950 is specifically used for:

[0144] Using the yaw angle of the face, the target mirror face image is determined in the mirror face image;

[0145] If the yaw angle of the face is within the third preset angle range, the expression parameters are determined based on the position coordinates of multiple feature points in the target mirror face image.

[0146] If the yaw angle of the face is within the fourth preset angle range, the expression parameters are obtained based on the position coordinates of multiple feature points in the target mirror face image and the position coordinates of multiple feature points in the real face image.

[0147] In one embodiment, the facial expression parameter determination module 950 is further configured to:

[0148] For any feature point, the position coordinates of the feature point in the target mirror face image are transformed by using the pre-set correspondence between the mirror position coordinates and the real position coordinates to obtain the real position coordinates of the feature point in the real face image.

[0149] Based on the real position coordinates of multiple feature points in the target mirror face image and the position coordinates of multiple standard feature points corresponding to a pre-set 3D digital human, the first intermediate expression parameters are obtained; and,

[0150] The second intermediate expression parameters are obtained based on the position coordinates of multiple feature points in the real face image and the position coordinates of the multiple standard feature points.

[0151] The first intermediate expression parameter and the second intermediate expression parameter are weighted and summed to obtain the expression parameter.

[0152] In one embodiment, the facial expression parameter determination module 950 is further configured to:

[0153] The facial expression parameters are obtained using the following formula:

[0154] ω i =β i (R*(στ+σ′τ′)+TB);

[0155] Where R is the rotation matrix of the face, T is the translation matrix of the face, σ is the matrix corresponding to the position coordinates of multiple feature points in the real face image, τ is the confidence matrix of the position coordinates of multiple feature points in the real face image, σ′ is the matrix corresponding to the position coordinates of multiple feature points in the target mirror face image, τ′ is the confidence matrix of the position coordinates of multiple feature points in the target mirror face image, B is the matrix corresponding to the position coordinates of multiple standard feature points corresponding to the pre-set 3D digital human, and β i For the pre-set i-th expression basis matrix, ω i Let be the expression parameters corresponding to the i-th expression basis matrix.

[0156] After introducing a three-dimensional digital human expression driving method and apparatus according to an exemplary embodiment of the present invention, the electronic device according to another exemplary embodiment of the present invention will be introduced next.

[0157] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as "circuit", "module", or "system".

[0158] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code that, when executed by the processor, causes the processor to perform the steps in the facial expression driving method for a three-dimensional digital human according to various exemplary embodiments of the present invention described above. For example, the processor may perform actions such as... Figure 2 Steps 201-205 are shown in the diagram.

[0159] The following reference Figure 10 To describe an electronic device 1000 according to this embodiment of the present invention. Figure 10 The electronic device 1000 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0160] like Figure 10As shown, the electronic device 1000 is manifested in the form of a general electronic device. The components of the electronic device 1000 may include, but are not limited to: at least one processor 1001, at least one computer storage medium 1002, and a bus 1003 connecting different system components (including the computer storage medium 1002 and the processor 1001).

[0161] Bus 1003 represents one or more of several bus structures, including computer storage media bus or computer storage media controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0162] Computer storage medium 1002 may include readable media in the form of volatile computer storage media, such as random access computer storage medium (RAM) 1021 and / or cache storage medium 1022, and may further include read-only computer storage medium (ROM) 1023.

[0163] The computer storage medium 1002 may also include a program / utility 1025 having a set (at least one) of program modules 1024, such program modules 1024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0164] Electronic device 1000 can also communicate with one or more external devices 1004 (e.g., keyboard, pointing device, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1005. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1006. As shown, network adapter 1006 communicates with other modules used in electronic device 1000 via bus 1003. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0165] In some possible implementations, various aspects of the facial expression driving method for a three-dimensional digital human provided by the present invention can also be implemented in the form of a program product, which includes program code that, when the program product is run on a computer device, causes the computer device to perform the steps in the facial expression driving method for a three-dimensional digital human according to various exemplary embodiments of the present invention as described above.

[0166] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for driving facial expressions in a three-dimensional digital human, characterized in that, The method includes: Face detection is performed on the current frame image to obtain bounding boxes of multiple faces in the current frame image, wherein the current frame image contains the target object and the target object in the mirror, and the mirror is positioned to the left and right of the target object; The current frame image is cropped using the bounding boxes of the multiple faces to obtain multiple face images, wherein the multiple face images include real face images and mirrored face images; Feature points are extracted from the multiple face images to obtain the position coordinates of multiple feature points in the multiple face images and the confidence level of the multiple feature points; Based on the position coordinates of multiple feature points in the real face image and the position coordinates of multiple standard feature points corresponding to a pre-set 3D digital human, the rotation and translation matrix of the face in the real face image is obtained, wherein the number of multiple feature points is the same as the number of multiple standard feature points. Based on the rotation and translation matrix of the face, the rotation angle of the face is obtained, wherein the rotation angle of the face includes yaw angle and pitch angle; If the rotation angle of the face and the confidence level of the multiple feature points meet the specified conditions, then it is determined that a valid face region can be detected in the current frame image; otherwise, it is determined that a valid face region cannot be detected in the current frame image. If not, then when the yaw angle of the face is within the third preset angle range, the expression parameters of the target object in the current frame image are obtained based on the position coordinates of multiple feature points in the mirror face image; or, when the yaw angle of the face is within the fourth preset angle range, the expression parameters are obtained based on the position coordinates of multiple feature points in the target mirror face image and the position coordinates of multiple feature points in the real face image, and the expression parameters are used to drive the expression of the pre-set 3D digital human.

2. The method according to claim 1, characterized in that, The method further includes: If a valid face region can be detected in the current frame image, then based on the position coordinates of multiple feature points in the real face image and the position coordinates of multiple standard feature points corresponding to the pre-set 3D digital human, the expression parameters of the target object in the current frame image are obtained, and the expression parameters are used to drive the expression of the pre-set 3D digital human.

3. The method according to claim 1, characterized in that, The rotation angle of the face and the confidence levels of the multiple feature points are determined to meet specified conditions using the following methods: If the yaw angle is within a first preset angle range, the pitch angle is within a second preset angle range, and the number of target feature points is greater than a specified number, then the rotation angle of the face and the confidence of the multiple feature points are determined to meet the specified conditions, wherein the target feature points are feature points with a confidence greater than a specified threshold. Otherwise, it is determined that the rotation angle of the face and the confidence level of the multiple feature points do not meet the specified conditions.

4. The method according to claim 1, characterized in that, The number of mirrored face images is the same as the number of mirrors.

5. The method according to claim 4, characterized in that, The step of obtaining the expression parameters based on the position coordinates of multiple feature points in the target mirror face image and the position coordinates of multiple feature points in the real face image includes: For any feature point, the position coordinates of the feature point in the target mirror face image are transformed by using the pre-set correspondence between the mirror position coordinates and the real position coordinates to obtain the real position coordinates of the feature point in the real face image. Based on the real position coordinates of multiple feature points in the target mirror face image and the position coordinates of multiple standard feature points corresponding to a pre-set 3D digital human, the first intermediate expression parameters are obtained; and, The second intermediate expression parameters are obtained based on the position coordinates of multiple feature points in the real face image and the position coordinates of the multiple standard feature points. The first intermediate expression parameter and the second intermediate expression parameter are weighted and summed to obtain the expression parameter.

6. The method according to claim 4, characterized in that, The step of obtaining the expression parameters based on the position coordinates of multiple feature points in the target mirror face image and the position coordinates of multiple feature points in the real face image includes: The facial expression parameters are obtained using the following formula: ； in, Let be the rotation matrix of the face. Let be the translation matrix of the face. This is a matrix corresponding to the position coordinates of multiple feature points in the real face image. This is a confidence matrix representing the position coordinates of multiple feature points in the real face image. This is a matrix corresponding to the position coordinates of multiple feature points in the target mirror face image. This is a confidence matrix representing the position coordinates of multiple feature points in the target mirrored face image. A matrix corresponding to the position coordinates of multiple standard feature points of a pre-set 3D digital human. For the pre-set i-th expression basis matrix, Let be the expression parameters corresponding to the i-th expression basis matrix.

7. An electronic device, characterized in that, It includes a processor and a memory, which are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: Face detection is performed on the current frame image to obtain bounding boxes of multiple faces in the current frame image, wherein the current frame image contains the target object and the target object in the mirror, and the mirror is positioned to the left and right of the target object; The current frame image is cropped using the bounding boxes of the multiple faces to obtain multiple face images, wherein the multiple face images include real face images and mirrored face images; Feature points are extracted from the multiple face images to obtain the position coordinates of multiple feature points in the multiple face images and the confidence level of the multiple feature points; Based on the position coordinates of multiple feature points in the real face image and the position coordinates of multiple standard feature points corresponding to a pre-set 3D digital human, the rotation and translation matrix of the face in the real face image is obtained, wherein the number of multiple feature points is the same as the number of multiple standard feature points. Based on the rotation and translation matrix of the face, the rotation angle of the face is obtained, wherein the rotation angle of the face includes yaw angle and pitch angle; If the rotation angle of the face and the confidence level of the multiple feature points meet the specified conditions, then it is determined that a valid face region can be detected in the current frame image; otherwise, it is determined that a valid face region cannot be detected in the current frame image. If not, then when the yaw angle of the face is within the third preset angle range, the expression parameters of the target object in the current frame image are obtained based on the position coordinates of multiple feature points in the mirror face image; or, when the yaw angle of the face is within the fourth preset angle range, the expression parameters are obtained based on the position coordinates of multiple feature points in the target mirror face image and the position coordinates of multiple feature points in the real face image, and the expression parameters are used to drive the expression of the pre-set 3D digital human.

8. The electronic device according to claim 7, characterized in that, The processor is also configured to: If a valid face region can be detected in the current frame image, then based on the position coordinates of multiple feature points in the real face image, the expression parameters of the target object in the current frame image are obtained, and the expression parameters are used to drive the expression of the pre-set 3D digital human.

Images