Facial expression tracking method, apparatus, device, and storage medium

Abstract

The application provides an expression tracking method and device, equipment and storage medium, relates to the computer vision field of artificial intelligence, and the method comprises the following steps: obtaining a neutral face model of a target person and M expression bases BS corresponding to the neutral face model, M>0; collecting the face of the target person in multiple view directions to obtain multiple current images corresponding to the multiple views respectively; using an expression tracking algorithm, calculating M BS coefficients corresponding to the M BSs respectively for tracking the facial expressions in the multiple current images; based on the neutral face model, the M BSs and the M coefficients, constructing tracking images of the multiple current images. The expression tracking method provided by the application can improve the expression tracking effect.

Description

[0001] This application claims priority to Chinese Patent Application No. 202111478236.2, filed on December 6, 2021, entitled “Method, Apparatus, Device and Storage Medium for Facial Expression Tracking”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer vision in artificial intelligence, and more specifically, to facial expression tracking methods, apparatus, devices, and storage media. Background Technology

[0003] Three-dimensional facial expression tracking refers to tracking the facial expressions of characters in a video and using this tracking to drive different 3D face models, enabling the 3D face models to display the corresponding expressions. Typically, a 3D face model can be implemented as a 3D Morphable Face Model (3DMM). A 3DMM is a general 3D parametric face model that uses a fixed number of points to represent a face. The core idea of ​​3DMM is to match faces one-to-one in 3D space and obtain a face model by performing orthogonal basis weighted linear summation on multiple face images in a database, thus achieving expression tracking.

[0004] Specifically, based on the concept of 3DMM, multiple faces can be constructed using an average face model, i.e., the expression basis (BS) corresponding to the average face model. Based on this, when expression tracking is required, a face model with expression can be constructed using 3DMM based on the initial coefficients of the expression basis (BS) corresponding to the average face model. Then, the constructed face model is projected onto the image to be tracked to obtain the projected image. Next, the initial coefficients of the BS corresponding to the average face model can be adjusted according to the difference between the facial key points in the projected image and the facial key points in the image to be tracked until the result converges, so that the face model constructed by 3DMM can better match the facial expression in the image to be tracked.

[0005] However, due to the differences between various facial images, such as the variations in facial shapes among people of different ages, using an average face model may result in the average face model not resembling the contours of the person in the video. This can affect the fit of the projection results and reduce the effectiveness of expression tracking. Furthermore, the expressive power of the Baseline Graph (BS) also affects expression tracking performance. Using the BS corresponding to the average face model for tracking will lead to poor BS expressive power, further reducing the effectiveness of expression tracking. Summary of the Invention

[0006] This application provides an expression tracking method, apparatus, device, and storage medium that can improve expression tracking performance.

[0007] On the one hand, this application provides an expression tracking method, including:

[0008] Obtain the neutral face model of the target character and the M tables corresponding to the neutral face model, where M > 0;

[0009] The face of the target person is captured from multiple viewpoints, resulting in multiple current images corresponding to each viewpoint.

[0010] Using an expression tracking algorithm, calculate the M BS coefficients that are used to track facial expressions in the multiple current images and are respectively associated with the M BSs;

[0011] Based on the neutral face model, the M BSs, and the M coefficients, multiple tracking images of the current image are constructed.

[0012] On the other hand, this application provides an expression tracking device, including:

[0013] The acquisition unit is used to acquire the neutral face model of the target person and the M tables BS corresponding to the neutral face model, where M>0;

[0014] The acquisition unit is used to acquire the face of the target person from multiple perspectives, and obtain multiple current images corresponding to the multiple perspectives respectively;

[0015] The computing unit is used to calculate the M BS coefficients corresponding to the M BSs respectively for tracking facial expressions in the multiple current images using an expression tracking algorithm;

[0016] The construction unit is used to construct multiple tracking images of the current image based on the neutral face model, the M BSs, and the M coefficients.

[0017] On the other hand, this application provides an electronic device, including:

[0018] Processor, adapted to implement computer instructions; and,

[0019] A computer-readable storage medium storing computer instructions adapted to be loaded by a processor and to execute the method of the first aspect described above.

[0020] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when read and executed by a processor of a computer device, cause the computer device to perform the method described in the first aspect.

[0021] On the other hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the first aspect.

[0022] Based on the above technical solutions, this application introduces a neutral face model of the target person and M expression bases (BS) corresponding to the neutral face model. Compared with the scheme of expression tracking based on the average face model, the tracking image constructed based on the neutral face model of the target person and the M expression bases (BS) corresponding to the neutral face model can not only improve the expressive power of the BS, but also make the facial expressions in the tracking image more consistent with the contour of the target person, thereby improving the expression tracking effect.

[0023] Furthermore, based on the concept of multiple perspectives, this application designs multiple current images to capture the face of the target person from multiple perspective directions to obtain images corresponding to each of the multiple perspectives, so as to perform expression tracking on the multiple current images. This can enrich the reference information of the expression tracking algorithm, and correspondingly improve the accuracy of the M BS coefficients, thereby further improving the expression tracking effect.

[0024] In addition to improving the facial expression tracking effect, it also facilitates the feasibility of implementing facial expression tracking algorithms in actual products. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of facial reference points involved in an embodiment of this application.

[0027] Figure 2 This is a schematic diagram of an application scenario involved in an embodiment of this application.

[0028] Figure 3 This is a schematic flowchart of the facial expression tracking method provided in the embodiments of this application.

[0029] Figure 4 This is a schematic diagram of the BS provided in the embodiments of this application.

[0030] Figure 5 This is a schematic diagram of the construction of a tracking image provided in an embodiment of this application.

[0031] Figure 6 This is a schematic diagram of multiple current images provided in the embodiments of this application.

[0032] Figure 7 This is a schematic diagram illustrating the construction of a neutral face model based on multiple videos collected from multiple viewpoints, as provided in an embodiment of this application.

[0033] Figure 8 This is a schematic diagram of an image-based neutral face model provided in an embodiment of this application.

[0034] Figure 9 This is a schematic block diagram of the facial expression tracking device provided in the embodiments of this application.

[0035] Figure 10 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0037] The solutions provided in this application may involve artificial intelligence technology.

[0038] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0039] It should be understood that artificial intelligence (AI) technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0040] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0041] This application's embodiments may relate to Computer Vision (CV) technology within artificial intelligence. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision that uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and further performs image processing to transform the computer-processed images into those more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0042] This application's embodiments may also relate to Machine Learning (ML) in artificial intelligence technology. ML is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0043] The solutions provided in this application also relate to video processing technology in the field of network media. Network media differs from traditional audio and video equipment in its operating methods. Network media relies on technologies and equipment provided by information technology (IT) equipment developers to transmit, store, and process audio and video signals. Traditional Serial Digital Induction (SDI) transmission lacks true network switching characteristics. Significant work is required to utilize SDI to create some of the network functions provided by Ethernet and Internet Protocol (IP). Therefore, network media technology in the video industry has emerged. Furthermore, video processing technology for network media can include the transmission, storage, and processing of audio and video signals, as well as the audio and video...

[0044] Furthermore, the solutions provided in this application may also involve facial expression tracking technology.

[0045] To facilitate understanding of the technical solution provided in this application, the following explanation is provided regarding the relevant content of facial expression tracking.

[0046] 2D facial landmark detection: Automatically locates a set of predefined facial landmarks (such as the corners of the eyes and the corners of the mouth).

[0047] Figure 1 This is a schematic diagram of facial reference points involved in an embodiment of this application.

[0048] like Figure 1 As shown, facial reference points can be marked around the outline of the face, the corners of the eyes, and the corners of the mouth to achieve the detection of the face or facial expressions.

[0049] 3D facial expression tracking: This refers to tracking the facial expressions of characters in a video and using this tracking to drive different 3D facial models, enabling the 3D facial models to display expressions corresponding to the characters in the video. It should be understood that all intermediate models, neutral face models, and expression bases (BS) involved in this application are 3D facial models.

[0050] 3D Morphable Models (3DMMs): 3DMMs are a general-purpose parametric model of 3D faces, representing faces using a fixed number of points. The core idea of ​​3DMMs is to match faces one-to-one in 3D space and obtain a face model by performing orthogonal basis weighted linear summation on multiple faces in a database, thereby achieving expression tracking.

[0051] Each 3D face can be represented by a basis vector space consisting of all faces in a database. Solving for the model of any 3D face is actually equivalent to solving for the coefficients of each basis vector.

[0052] The basic attributes of a human face include shape and texture. Each human face can be represented as a linear superposition of shape vectors and texture vectors.

[0053] Shape Vector: S = (X1, Y1, Z1, X2, Y2, Z2, ..., Yn, Zn),

[0054] Texture Vector: T = (R1, G1, B1, R2, G2, B2, ..., Rn, Bn),

[0055] Where n is the number of face samples in the dataset, Xi, Yi, Zi are the coordinates of the shape vector of the i-th face sample in the dataset, and Ri, Gi, Bi are the coordinates of the texture vector of the i-th face sample in the dataset.

[0056] Any face model can be obtained by weighted combination of m face models in the dataset as follows:

[0057]

[0058]

[0059] Among them, S model For a 3D human face shape model, a i Let S be the target value of the face shape parameter, i = 1…m, where m is the number of face samples in the dataset, and S is the target value of the face shape parameter. i Let i be the shape vector of the i-th face sample in the dataset. T is the mean of the shape vectors of all face samples in the dataset. model For a 3D human face texture model, b i The target value for the face texture parameters, T i Let be the texture vector of the i-th face sample in the dataset, and T be the mean of the texture vectors of all face samples in the dataset.

[0060] Constraints refer to finding an element within a given function that minimizes or maximizes a certain metric. Constraints can also be termed mathematical programming (e.g., linear programming). The function can be called the objective function or cost function. A feasible solution to an objective function that minimizes or maximizes a certain metric is called the optimal solution. In the context of this application, the facial expression tracking algorithm can be used to: solve for the optimal solution under multiple constructed constraints, and use the solved optimal solution as the M BS coefficients corresponding to M BS values ​​for tracking facial expressions in multiple current images.

[0061] Average face: This can be a composite face of a group of people obtained through computer technology. For example, facial features can be extracted from a certain number of ordinary faces, averaged based on measurement data, and then computer technology can be used to create a composite face, resulting in an attractive face. In other words, a group of people with an average face have a high degree of similarity in appearance, greatly increasing the chances of them looking alike.

[0062] Blend shape: A technique for deforming a single mesh to achieve a number of predefined shapes and any number of combinations, referred to as a deformable target in Maya / 3ds Max. For example, a single mesh can be the basic shape of a default shape, such as an expressionless human face, i.e., the neutral face model involved in this application. Other shapes of the basic shape are used for blending / deformation, representing different expressions (smiling, frowning, closed eyelids). These other shapes are collectively referred to as blend shapes or deformable targets, i.e., the M BS corresponding to the neutral face model involved in this application.

[0063] Figure 2 This is a schematic diagram of an application scenario involved in an embodiment of this application.

[0064] like Figure 2 As shown, the system includes an acquisition device 101, a computing device 102, and a display device 103. The acquisition device 101 acquires N facial images of the user, which can be facial images acquired from N viewing directions. The computing device 102 reconstructs a three-dimensional facial model based on the N facial images acquired by the acquisition device 101, using the expression tracking method provided in this embodiment. The display device 103 displays the three-dimensional facial model reconstructed by the computing device 102.

[0065] For example, computing device 101 may be a user device, such as a mobile phone, tablet computer, laptop computer, handheld computer, mobile internet device (MID) or other terminal device with browser installation function.

[0066] For example, computing device 102 can be a server. There can be one or more servers. When there are multiple servers, at least two servers provide different services, and / or at least two servers provide the same service, such as providing the same service through load balancing; this embodiment does not limit this. The server can be equipped with a reconstruction model, which supports the training and application of the reconstruction model. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can also be a node in a blockchain.

[0067] For example, when the computing device 102 has a display function, the display device 103 can be a display in the computing device 102.

[0068] For example, the display device 103 and the computing device 102 are different devices, and the display device 103 is connected to the computing device 102 via a network. The network can be an intranet, the Internet, the Global System for Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, voice communication network, or other wireless or wired networks.

[0069] It should be noted that the application scenarios of this application include, but are not limited to, facial expression tracking of characters in 3D games / 3D films or online facial expression tracking. For example, the solution provided in this application can be applied to the creation of 3D virtual humans, which can use captured facial expressions to drive 3D virtual characters, which can be game characters or animated characters.

[0070] For example, facial expression tracking can be achieved through the following steps:

[0071] Step 1: Detect facial landmarks in the image to be tracked using existing landmark detection methods.

[0072] Step 2: Annotate the key facial features in the image to be tracked based on the feature points of the face in the image to be tracked.

[0073] Step 3: The tracking result of a specific image to be tracked within the video is obtained through the following steps:

[0074] Step 3-1: Reconstruct a face model with expressions using the initial coefficients of the BS corresponding to the average face model.

[0075] Step 3-2: Based on the pose information of the average face model, project the reconstructed face model in Step 3-1 onto the image to be tracked to obtain the projected image; based on this, the error between the facial key points of the image to be tracked and the facial key points of the projected image can be calculated, and the error can be used as supervision information to update the BS coefficients of the BS corresponding to the average face model.

[0076] Step 3-3: Repeat steps 3-1 and 3-2 until convergence (i.e., the error between the facial key points of the image to be tracked and the facial key points of the projected head is minimized), obtain the BS coefficients of the average face model corresponding to the image to be tracked, and finally construct the tracking image based on the BS coefficients of the average face model.

[0077] In other words, based on the concept of 3DMM, multiple faces can be constructed using the average face model, i.e., the expression basis (BS) corresponding to the average face model. Based on this, when expression tracking is required, a face model with expression can be constructed using 3DMM based on the initial coefficients of the expression basis (BS) corresponding to the average face model. Then, the constructed face model is projected onto the image to be tracked to obtain the projected image. Next, the initial coefficients of the BS corresponding to the average face model can be adjusted according to the difference between the facial key points in the projected image and the facial key points in the image to be tracked until the result converges, so that the face model constructed by 3DMM can better match the facial expression of the image to be tracked.

[0078] However, due to the differences between various facial images, such as the variations in facial shapes among people of different ages, using an average face model may result in the average face model not resembling the contours of the person in the video. This can affect the fit of the projection results and reduce the effectiveness of expression tracking. Furthermore, the expressive power of the Baseline Graph (BS) also affects expression tracking performance. Using the BS corresponding to the average face model for tracking will lead to poor BS expressive power, further reducing the effectiveness of expression tracking.

[0079] In view of this, embodiments of this application provide an expression tracking method, apparatus, device, and storage medium that can improve the expression tracking effect.

[0080] Specifically, on the one hand, this application introduces a neutral face model of the target person and M expression bases (BSs) corresponding to this neutral face model. Compared with the scheme of expression tracking based on the average face model, the tracking image constructed based on the neutral face model of the target person and the M expression bases (BSs) corresponding to this neutral face model not only makes the facial expressions in the tracking image more consistent with the contours of the target person, but also improves the fitting degree of the projection results and the expressive power of the BSs, thereby improving the expression tracking effect. On the other hand, this application is based on a multi-view approach. Multiple current images are designed to acquire the face of the target person from multiple viewpoints to obtain images corresponding to each of the multiple viewpoints, so as to perform expression tracking on these multiple current images. This can enrich the reference information of the expression tracking algorithm, and correspondingly improve the accuracy of the M BS coefficients, further improving the expression tracking effect.

[0081] Figure 4A schematic flowchart of an expression tracking method 200 according to an embodiment of this application is shown. The expression tracking method 200 can be executed by any electronic device with data processing capabilities. For example, the electronic device can be implemented as a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms. The server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations. For ease of description, the expression tracking method provided in this application will be described below using an expression tracking device as an example.

[0082] like Figure 4 As shown, the facial expression tracking method 200 may include:

[0083] S210, Obtain the neutral face model of the target person and the M tables BS corresponding to the neutral face model, M>0;

[0084] S220, capture the face of the target person from multiple perspectives to obtain multiple current images corresponding to the multiple perspectives respectively;

[0085] S230, using an expression tracking algorithm, calculate the M BS coefficients used to track facial expressions in the multiple current images and corresponding to the M BS respectively;

[0086] S240, based on the neutral face model, the M BSs and the M coefficients, construct multiple tracking images of the current image.

[0087] In this embodiment, a neutral face model of the target person and M expression bases (BS) corresponding to the neutral face model are introduced. Compared with the scheme of expression tracking based on the average face model, the tracking image constructed based on the neutral face model of the target person and the M expression bases (BS) corresponding to the neutral face model can not only improve the expressive power of the BS, but also make the facial expressions in the tracking image more consistent with the contours of the target person, thereby improving the expression tracking effect.

[0088] Furthermore, based on the concept of multiple perspectives, this application designs multiple current images to capture the face of the target person from multiple perspective directions to obtain images corresponding to each of the multiple perspectives, so as to perform expression tracking on the multiple current images. This can enrich the reference information of the expression tracking algorithm, and correspondingly improve the accuracy of the M BS coefficients, thereby further improving the expression tracking effect.

[0089] In addition to improving the facial expression tracking effect, it also facilitates the feasibility of implementing facial expression tracking algorithms in actual products.

[0090] Figure 4 This is a schematic diagram of the BS provided in the embodiments of this application.

[0091] like Figure 4 As shown, the nine face samples (BS) on the left are those corresponding to the average face model, and the nine face samples on the right are those corresponding to the neutral face model. It can be seen that the BS corresponding to the neutral face model is more consistent with the contour of the target person. Accordingly, when constructing tracking images based on the neutral face model and its corresponding BS, it can not only improve the expressive power of the BS, but also make the facial expressions in the tracking images more consistent with the contour of the target person, thereby improving the expression tracking effect.

[0092] It should be noted that this application does not specifically limit the value of M. For example, M = 185. Of course, in other alternative embodiments, the value of M can also be other values, for example, M can be an integer.

[0093] Figure 5 This is a schematic diagram of the construction of a tracking image provided in an embodiment of this application.

[0094] like Figure 5 As shown, after obtaining the neutral face model and the M face blocks (BSs), multiple tracking images of the current image can be constructed based on the M coefficients corresponding to each of the M BSs. Specifically, the tracking images can be constructed according to the following formula:

[0095]

[0096] Where, α i B represents the i-th coefficient among M coefficients. i Let Bi be the i-th face among M face types, and B0 be the neutral face model.

[0097] It should be noted that this application does not specifically limit the implementation method of acquiring the multiple current images. For example, in one implementation, an image can be acquired in one viewpoint direction and then transformed in multiple viewpoint directions to obtain multiple current images; alternatively, images can be acquired simultaneously in multiple viewpoint directions to obtain the multiple current images. Optionally, the multiple current images can be images in Red Green Blue (RGB) format or other formats; this application does not specifically limit this.

[0098] The solution provided in this application can track the facial expressions of a target person based on multiple current images corresponding to multiple viewpoints. Correspondingly, this application also provides a supervised mode for the multi-viewpoint facial expression tracking algorithm. On the one hand, it can constrain the geometric differences between facial key points in the projected image of the constructed facial expression model and facial key feature points in the current image across multiple viewpoints, thereby improving the matching area of ​​facial key points and thus improving the accuracy of the constraints. Furthermore, it provides methods for applying error constraints and stacking constraints in the temporal domain to improve the accuracy of the facial expression tracking algorithm in solving the BS coefficients under different facial expressions, thereby improving the tracking effect. The implementation methods of various constraints are illustrated below.

[0099] In some embodiments, S230 may include:

[0100] Using the M BSs and the M initial coefficients corresponding to the M BSs, an intermediate image with facial expression is constructed; based on the pose information of the neutral face model, the intermediate image is projected onto the multiple current images to obtain multiple projected images; at least one loss of the multiple projected images is calculated; based on the at least one loss, the M initial coefficients are adjusted to obtain the M BS coefficients.

[0101] For example, after obtaining the at least one loss, the at least one loss can be used to supervise the expression tracking effect to adjust M initial coefficients and obtain the final BS coefficients used to construct the tracking image of the multiple current images; or, by adjusting the M initial coefficients so that the at least one loss satisfies the constraints for constructing the expression tracking, the adjusted BS coefficients that satisfy the constraints are determined as the final BS coefficients used to construct the tracking image of the multiple current images; accordingly, after determining the M BS coefficients, the tracking image of the current image can be constructed based on the M BS coefficients.

[0102] For example, the intermediate image can be projected onto the multiple current images using the camera's intrinsic parameters to obtain multiple projected images. Of course, projection can also be performed in other ways, and this application does not specifically limit this method.

[0103] In this embodiment, the multiple projected images are designed to be obtained by projecting the intermediate image onto the multiple current images based on the pose information of the neutral face model. This makes the facial expressions of the multiple projected images more consistent with the contours of the target person. Accordingly, it can improve the accuracy of at least one loss, and thus improve the accuracy of the M BS and the expression tracking effect.

[0104] Figure 6 This is a schematic diagram of multiple current images provided in the embodiments of this application.

[0105] like Figure 6 As shown, three current images can be acquired in three viewing directions: the first current image can be acquired in the middle viewing direction, the second current image can be acquired in the left viewing direction, and the third current image can be acquired in the right viewing direction.

[0106] In accordance with the scheme of this application, after constructing an intermediate image with facial expressions, the intermediate image can be projected onto the first current image to obtain a first projected image, the intermediate image can be projected onto the second current image to obtain a second projected image, and the intermediate image can be projected onto the third current image to obtain a third projected image, based on the pose information of the neutral face model. Then, at least one loss can be calculated based on at least one of the following: the difference between the first current image and the first projected image, the difference between the second current image and the second projected image, and the difference between the third current image and the third projected image. The at least one loss is used to supervise the facial expression tracking effect. That is, the facial expression tracking effect can be supervised based on at least one of the following: the difference between the first current image and the first projected image, the difference between the second current image and the second projected image, and the difference between the third current image and the third projected image.

[0107] In some embodiments, the at least one loss includes a first loss; the first loss is characterized as the difference between facial key points in a first current image and facial key points in a first projected image; wherein the first current image is an image of the target person's face captured in the intermediate viewing direction among the multiple viewing directions, and the first projected image is a projected image obtained by projecting the intermediate image onto the first current image.

[0108] In this embodiment, the constraint of the expression tracking algorithm is designed as a geometric constraint in the middle view direction; that is, the middle image (i.e., the 3D model with expression) is projected onto the first current image to obtain the first projected image. Based on this, the objective function or loss function is designed as the difference between the facial key points of the first current image and the facial key points of the first projected image. Finally, the effect of expression tracking is supervised based on the calculated difference.

[0109] For example, the first loss can be calculated using the following formula:

[0110]

[0111] Where, ε lm This indicates the first loss; 86 represents the number of facial landmarks; w i V represents the weight of the i-th facial keypoint, P(·) represents the reprojection function used to project the intermediate image onto the first current image; i0 Let α represent the i-th facial keypoint of the neutral face model. j Let ΔV represent the j-th initial coefficient among the M initial coefficients. i j L represents the face keypoint of the i-th person in the j-th BS among the M BSs. i This represents the i-th facial keypoint in the first current image.

[0112] In some embodiments, the at least one loss includes a second loss; the second loss is characterized as at least one of the following: the error between facial key points in the left region of the second current image and facial key points in the left region of the second projected image, and the difference between facial key points in the right region of the third current image and facial key points in the right region of the third projected image; wherein the second current image is an image of the target person's face captured in the left viewing direction among the plurality of viewing directions, the second projected image is a projection image obtained by projecting the intermediate image onto the second current image; the third current image is an image of the target person's face captured in the right viewing direction among the plurality of viewing directions, and the third projected image is a projection image obtained by projecting the intermediate image onto the third current image.

[0113] In this embodiment, the constraints of the expression tracking algorithm are designed as geometric constraints in the left view direction and geometric constraints in the right view direction. That is, the intermediate image (i.e., the 3D model with the expression) is projected onto the second current image to obtain the second projected image, and the intermediate image is projected onto the third current image to obtain the third projected image. Based on this, the objective function or loss function is designed as: the difference between the facial key points in the left region of the second current image and the facial key points in the left region of the second projected image, and the difference between the facial key points in the right region of the third current image and the facial key points in the right region of the third projected image. Finally, the effect of expression tracking is supervised based on the calculated differences.

[0114] For example, the second current image is obtained by transforming the first current image based on a transformation matrix from the center view direction to the left view direction. Similarly, the third current image is obtained by transforming the first current image based on a transformation matrix from the center view direction to the right view direction.

[0115] For example, the second loss can be determined by the following formula:

[0116]

[0117] Where, ε geo This indicates the second loss, w jT represents the weight of the j-th facial keypoint, P(·) represents the reprojection function used to project the intermediate image onto the second and third current images; mid This represents the first current image, ΔT. mid→left This represents the transformation matrix from the center viewpoint to the left viewpoint. Let ΔT represent the j-th facial keypoint in the second current image. mid→right This represents the transformation matrix from the center viewpoint to the right viewpoint. The j-th facial keypoint in the third current image is represented; J represents the number of facial keypoints in the left region of the second current image and the number of facial keypoints in the right region of the third current image.

[0118] In some embodiments, the at least one loss includes a third loss, which is characterized as the optical flow error between the plurality of projected images and the plurality of reference images respectively; wherein the plurality of reference images are projected images obtained by projecting the intermediate image onto the plurality of adjacent images respectively, and the plurality of adjacent images are images acquired in the plurality of viewing directions before acquiring the plurality of current images, used for tracking facial expressions, and adjacent to the plurality of current images.

[0119] In this embodiment, the constraint of the expression tracking algorithm is designed as a temporal constraint between two consecutive images, namely, an optical flow constraint between two consecutive images. Optical flow refers to the displacement between a pixel in frame i-1 (for example, a point on a moving car) and the corresponding pixel in frame i. Specifically, the intermediate image (i.e., a 3D model with facial expressions) is projected onto the multiple current images to obtain multiple projected images, and the intermediate image is projected onto multiple neighboring images acquired in multiple viewpoint directions before acquiring the multiple current images, which are used to track facial expressions and are adjacent to the multiple current images to obtain multiple reference images. Based on this, the objective function or loss function is designed as the optical flow difference between the multiple projected images and the multiple reference images, and finally, the effect of expression tracking is supervised based on the calculated optical flow difference.

[0120] For example, the third loss is characterized as the optical flow error between the face regions of the multiple projected images and the face regions of the multiple reference images. For instance, the face regions of the multiple projected images and the face regions of the multiple reference images can be obtained first using a face segmentation algorithm, and then the optical flow error between the face regions of the multiple projected images and the face regions of the multiple reference images can be calculated to finally obtain the third loss.

[0121] For example, this third loss can be calculated using the following formula:

[0122]

[0123] Where, ε flow This represents the third loss, G represents the number of effective optical flow points, and w g T represents the weight of the g-th effective optical flow point among the effective optical flow points, and P(·) represents the reprojection function used to project the intermediate image onto the multiple current images and multiple neighboring images; i Let α represent the head pose of the i-th image. i T represents the BS coefficients used in the i-th image, which can be one of multiple current images. i-1 Let α represent the head pose of the (i-1)th image. i-1 This represents the BS coefficients used in the (i-1)th image, which can be one of the multiple reference images. This represents the optical flow between the (i-1)th image and the ith image.

[0124] In some embodiments, the at least one loss includes a fourth loss, which is characterized as at least one of the following: the sum of the M initial coefficients, the difference between the M initial coefficients and the M reference coefficients, and the difference between the pose information of the plurality of current images and the pose information of the plurality of neighboring images, respectively; wherein the plurality of neighboring images are images acquired in the plurality of view directions before acquiring the plurality of current images, used for tracking facial expressions, and adjacent to the plurality of current images, and the M reference coefficients are the BS coefficients of the tracking images used to construct the plurality of neighboring images.

[0125] In this embodiment, the constraints of the expression tracking algorithm are designed as the sum of the M initial coefficients, the smoothness of the pose information in the time domain, and the smoothness of the BS coefficients in the time domain. That is, the objective function or loss function is designed as at least one of the following: the sum of the M initial coefficients, the difference between the M initial coefficients and the M reference coefficients, and the difference between the pose information of the multiple current images and the pose information of the multiple neighboring images. The effect of expression tracking is supervised based on the calculated differences.

[0126] For example, this fourth loss can be calculated using the following formula:

[0127]

[0128] Among them, w lm ε represents the weight of the first loss. lm Indicates the first loss, w geo ε represents the weight of the second loss. geo Indicates the second loss, w flow ε represents the weight of the third loss. flow Indicates the third loss, w reg α represents the weight of the sum of the M initial coefficients.j w represents the j-th initial coefficient among the M initial coefficients. pos The weights represent the differences between the pose information of the current images and the pose information of the neighboring images, respectively; T′ represents the image among the neighboring images; T represents the image among the current images; w exp α′ represents the weight of the difference between the M initial coefficients and the M reference coefficients. j This represents the j-th reference coefficient among the M reference coefficients, α j This represents the j-th initial coefficient among the M initial coefficients.

[0129] In some embodiments, S240 may include:

[0130] If the M BSs correspond to M regions of the face respectively, and M is an even number, then when the first coefficient corresponding to the first region and the second coefficient corresponding to the second region are different among the M coefficients, and the first region and the second region are symmetrical, the average value of the first coefficient and the second coefficient is determined; the average value is determined as the coefficient corresponding to the first region and the coefficient corresponding to the second region; based on the neutral face model, the M BSs, the coefficients corresponding to the regions other than the first region and the second region in the M regions, the coefficient corresponding to the first region, and the coefficient corresponding to the second region, the tracking images of the multiple current images are constructed.

[0131] In this embodiment, the M BSs are designed as BSs corresponding to symmetrical regions, and the coefficients of the left and right symmetrical BSs are averaged to ensure that the expressions on the left and right faces are symmetrical, making the expressions in the tracked image more in line with human facial expression habits, thereby improving the expression tracking effect.

[0132] In some embodiments, S210 may include:

[0133] Acquire multiple videos captured from these multiple viewpoints;

[0134] The face of the target person was reconstructed in 3D using these multiple videos to obtain the neutral face model;

[0135] Obtain the base pattern (BS) corresponding to the average face model;

[0136] The base face (BS) corresponding to the average face model is transferred to the neutral face model to obtain the M base faces.

[0137] For example, the face shape (BS) corresponding to the average face model can be a manually created BS. Expression transfer technology can be used to transfer the BS corresponding to the average face model to the neutral face model to obtain the M BS. For instance, EBR technology can be used to transfer the BS corresponding to the average face model to the neutral face model to obtain the M BS.

[0138] In some embodiments, when constructing a 3D model of the target person's face using the multiple videos, frames can be extracted from the multiple videos to obtain N images; then, K images can be selected from the N images, where N ≥ K > 0; finally, the 3D model of the target person's face can be constructed based on the K images to obtain the neutral face model.

[0139] For example, frames can be extracted from the multiple videos at preset time intervals to obtain N images; then, K images of better quality can be selected from the N images; finally, the face of the target person can be 3D constructed based on the K images to obtain the neutral face model. For example, K images suitable for building the neutral face model can be selected to 3D construct the face of the target person to obtain the neutral face model.

[0140] In some embodiments, when selecting K images from the N images, at least one of the following images can be deleted from the N images to obtain the K images: an image in which the difference between the facial key points and the facial key points in the neutral face image exceeds a preset range, an image in which the variance is less than or equal to a first preset threshold, and the second image in two adjacent images; wherein the variance of the difference between the second image and the first image in the two images is less than or equal to a second preset threshold.

[0141] For example, when deleting images from N images where the difference between facial key points and facial key points in a neutral face image exceeds a preset range, the neutral face image in the neutral face state can be identified first from the N images; then, H distances between facial key points in the neutral face image are calculated, where the H distances characterize the mouth shape and eye shape of the target person in the neutral face image; H > 0; the distance between facial key points in each of the N images is calculated; and images where the difference between the distance between facial key points in the N images and the K distances exceeds a preset range are deleted. Optionally, the first image extracted from the video at the intermediate viewpoint among the multiple viewpoints can be identified as the neutral face image.

[0142] For example, when deleting images from N images whose variance is less than or equal to a first preset threshold, the variance of each of the N images can be calculated first; then, images whose variance is less than or equal to the first preset threshold can be deleted. For example, the Laplacian operator can be used to convolve the images, and then the variance of the images can be calculated. For example, the Laplacian operator can use a 3x3 matrix or other forms of matrix.

[0143] For example, when deleting the second image from two adjacent images in N images, the variance of the difference between the pixel values ​​of the second image and the first image in the two adjacent images in the N images can be calculated first; if the difference is less than or equal to a second preset threshold, the second image is deleted; otherwise, the second image is retained.

[0144] In this embodiment, when selecting K images from the N images, images in which the difference between the facial key points in the N images and the facial key points in the neutral face image exceeds a preset range are deleted. This is equivalent to deleting images in the N images whose expressions are significantly inconsistent with those in the neutral face image. Images in the N images whose variance is less than or equal to a first preset threshold are also deleted. This is equivalent to deleting blurry images in the N images. The second image in two adjacent images in the N images is also deleted. This is equivalent to deleting distorted images in the N images, such as images of camera movement speed blocks. Based on this, the quality of the constructed neutral face model can be improved.

[0145] In some embodiments, a 3D model of the target person's face is constructed based on the K images to obtain a pre-constructed model; based on the pose information of the pre-constructed model, the pre-constructed model is projected onto the K images to obtain K projected images; the number of matching facial key points between the K images and the K projected images is calculated respectively; images with a number of matching facial key points greater than or equal to a third preset threshold are selected from the K images to obtain T images; K≥T>1; a 3D model of the target person's face is constructed based on the T images to obtain the neutral face model.

[0146] For example, by using the results of a Structure From Motion (SFM) algorithm, such as reprojection error and the number of matching points, T images that are important in constructing a neutral face model can be selected from K images. For example, T images with the number of matching facial key points greater than or equal to a third preset threshold can be selected. Then, the face of the target person can be reconstructed in 3D based on the selected T images to obtain the final neutral face model.

[0147] In this embodiment, the pre-built model can effectively filter out T relatively important images. Then, based on the T relatively important images, the 3D face of the target person can be reconstructed, which can further eliminate images that do not contribute much to the establishment of the neutral face model, effectively improving the model quality of the neutral face model.

[0148] Figure 7 This is a schematic diagram illustrating the construction of a neutral face model based on multiple videos collected from multiple viewpoints, as provided in an embodiment of this application.

[0149] like Figure 7 As shown, after acquiring multiple videos from multiple viewpoints, frames can be extracted from these videos to obtain N images. After acquiring these N images, images that are clearly inconsistent with the neutral facial expression, blurry images, and distorted images can be deleted to obtain K images. Next, the K images can be used to construct a 3D model of the target person's face to obtain a pre-constructed model. After obtaining the pre-constructed model, T images with a number of matching facial key points greater than or equal to a third preset threshold can be selected from the K images based on the pre-constructed model. Finally, the T images are used to construct a 3D model of the target person's face to obtain the neutral face model.

[0150] In some embodiments, the face of the target person is constructed in 3D based on the T images to obtain a first intermediate model; the vertex coordinates of the intermediate model are smoothed to obtain a second intermediate model; the deformed areas in the second intermediate model are repaired to obtain a third intermediate model; the third intermediate model is scanned using the Non-rigid Iterative Closest Point (NICP) algorithm to obtain the neutral face model.

[0151] Figure 8 This is a schematic diagram of an image-based neutral face model provided in an embodiment of this application.

[0152] like Figure 8 As shown, the smoothness of the first intermediate model is low. By smoothing and repairing the first intermediate model, a third intermediate model with better quality can be obtained. Furthermore, for this third intermediate model, based on the NICP algorithm, the template model can be used to scan the third intermediate model. In other words, the NICP algorithm can be used to obtain a clean 3D model with the same topological relationship as the template model and the same shape as the third intermediate model, i.e., a neutral face model, which further improves the model quality of the neutral face model.

[0153] It should be noted that since the 3D model of the target person's face constructed based on these T images is calculated using video frames, its quality is limited and contains a lot of noise; it can be considered the original high-resolution model. In this embodiment, smoothing the first intermediate model, repairing the deformed areas of the second intermediate model, and scanning the third intermediate model using the NICP algorithm are equivalent to performing three denoising processes on the first intermediate model, which can ensure the quality of the final neutral face model.

[0154] It should be understood that for rigid bodies, deformation can typically include rotational and translational deformation. In this case, the Iterative Closest Point (ICP) algorithm can be used to solve the problem of rigid body registration. For objects such as faces, hands, and bodies, deformation includes both rigid body deformation (local rotation and translation caused by different poses) and non-rigid body deformation (such as differences in shape leading to variations in size or height). When seeking a matching relationship between two point sets (source and target point sets), the ICP algorithm allows non-rigid deformation within the source point set.

[0155] The NICP algorithm can use a template model to scan a template and then obtain an output model. The output model can be a clean 3D model with the same topological relationship as the template model and the same shape as the scanned model.

[0156] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.

[0157] It should also be understood that, in the various method embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0158] The methods provided in the embodiments of this application have been described above. The apparatus provided in the embodiments of this application will be described below.

[0159] Figure 9 This is a schematic block diagram of the facial expression tracking device 300 provided in the embodiments of this application.

[0160] like Figure 9 As shown, the facial expression tracking device 300 may include:

[0161] The acquisition unit 310 is used to acquire the neutral face model of the target person and the M tables BS corresponding to the neutral face model, where M>0;

[0162] Acquisition unit 320 is used to acquire the face of the target person from multiple perspectives and obtain multiple current images corresponding to the multiple perspectives respectively;

[0163] The calculation unit 330 is used to calculate the M BS coefficients corresponding to the M BSs respectively for tracking facial expressions in the multiple current images using an expression tracking algorithm;

[0164] The construction unit 340 is used to construct multiple tracking images of the current image based on the neutral face model, the M BS and the M coefficients.

[0165] In some embodiments, the computing unit 330 is specifically used for:

[0166] Using the M BSs and the M initial coefficients corresponding to the M BSs respectively, an intermediate image with facial expressions is constructed;

[0167] Based on the pose information of the neutral face model, the intermediate image is projected onto the multiple current images to obtain multiple projected images;

[0168] Calculate at least one loss of the plurality of projected images;

[0169] Based on the at least one loss, the M initial coefficients are adjusted to obtain the M BS coefficients.

[0170] In some embodiments, the at least one loss includes a first loss; the first loss is characterized as the difference between facial key points in a first current image and facial key points in a first projected image;

[0171] Wherein, the first current image is an image of the target person's face captured in the middle view direction of the multiple view directions, and the first projected image is a projected image obtained by projecting the middle image onto the first current image.

[0172] In some embodiments, the at least one loss includes a second loss; the second loss is characterized as at least one of the following: the error between facial key points in the left region of the second current image and facial key points in the left region of the second projected image, and the difference between facial key points in the right region of the third current image and facial key points in the right region of the third projected image;

[0173] Wherein, the second current image is an image of the target person's face captured in the left view direction among the multiple view directions, and the second projected image is a projected image obtained by projecting the intermediate image onto the second current image;

[0174] The third current image is an image of the target person's face captured in the right-side view direction among the multiple view directions, and the third projected image is a projected image obtained by projecting the intermediate image onto the third current image.

[0175] In some embodiments, the at least one loss includes a third loss, which is characterized as optical flow error between the plurality of projected images and the plurality of reference images, respectively;

[0176] The plurality of reference images are projection images obtained by projecting the intermediate image onto the plurality of adjacent images, and the plurality of adjacent images are images acquired in the plurality of viewing directions before the plurality of current images are acquired, used for tracking facial expressions, and adjacent to the plurality of current images.

[0177] In some embodiments, the at least one loss includes a fourth loss, which is characterized as at least one of the following:

[0178] The sum of the M initial coefficients, the difference between the M initial coefficients and the M reference coefficients, and the difference between the pose information of the multiple current images and the pose information of the multiple neighboring images;

[0179] The multiple adjacent images are images acquired in the multiple view directions before acquiring the multiple current images, used for tracking facial expressions, and adjacent to the multiple current images. The M reference coefficients are the BS coefficients of the tracking images used to construct the multiple adjacent images.

[0180] In some embodiments, the building unit 340 is specifically used for:

[0181] If the M BSs correspond to M regions of a face respectively, and M is an even number, then when the first coefficient corresponding to the first region and the second coefficient corresponding to the second region are different among the M coefficients, and the first region and the second region are symmetrical, the average value of the first coefficient and the second coefficient is determined.

[0182] The average value is determined as the coefficient corresponding to the first region and the coefficient corresponding to the second region;

[0183] Based on the neutral face model, the M face blocks (BS), the coefficients corresponding to the regions other than the first and second regions in the M regions, the coefficients corresponding to the first region, and the coefficients corresponding to the second region, a tracking image of the plurality of current images is constructed.

[0184] In some embodiments, the acquisition unit 310 is specifically used for:

[0185] Acquire multiple videos captured from the multiple viewpoint directions;

[0186] The neutral face model is obtained by constructing a 3D model of the target person's face using the multiple videos.

[0187] Obtain the base pattern (BS) corresponding to the average face model;

[0188] The base face (BS) corresponding to the average face model is transferred to the neutral face model to obtain the M base face (BS).

[0189] In some embodiments, the acquisition unit 310 is specifically used for:

[0190] Frames are extracted from each of the multiple videos to obtain N images;

[0191] Select K images from the N images; N ≥ K > 0;

[0192] The neutral face model is obtained by constructing a 3D model of the target person's face based on the K images.

[0193] In some embodiments, the acquisition unit 310 is specifically used for:

[0194] To obtain the K images, delete at least one of the following images from the N images:

[0195] Images in which the difference between facial landmarks and facial landmarks in a neutral face image exceeds a preset range, images in which the variance is less than or equal to a first preset threshold, and the second image among two adjacent images;

[0196] Wherein, the variance of the difference between the second image and the first of the two images is less than or equal to a second preset threshold.

[0197] In some embodiments, the acquisition unit 310 is specifically used for:

[0198] Based on the K images, a 3D model of the target person's face is constructed to obtain a pre-constructed model;

[0199] Based on the pose information of the pre-built model, the pre-built model is projected onto the K images respectively to obtain K projected images;

[0200] Calculate the number of facial key points matched between the K images and the K projected images respectively;

[0201] From the K images, select images in which the number of matched facial key points is greater than or equal to a third preset threshold to obtain T images; K≥T>1;

[0202] The neutral face model is obtained by constructing a 3D model of the target person's face based on the T images.

[0203] In some embodiments, the acquisition unit 310 is specifically used for:

[0204] Based on the T images, a three-dimensional model of the target person's face is constructed to obtain a first intermediate model;

[0205] The vertex coordinates of the intermediate model are smoothed to obtain the second intermediate model;

[0206] Repair the deformed areas in the second intermediate model to obtain the third intermediate model;

[0207] The neutral face model is obtained by scanning the third intermediate model using the non-rigid iterative nearest point (NICP) algorithm.

[0208] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details are omitted here. Specifically, the expression tracking device 300 can correspond to the corresponding subject in the method 200 of the embodiments of this application, and each unit in the expression tracking device 300 is for implementing the corresponding process in method 200. For the sake of brevity, further details are omitted here.

[0209] It should also be understood that the various units in the expression tracking device 300 involved in the embodiments of this application can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This can achieve the same operation without affecting the technical effect of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the expression tracking device 300 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by multiple units working together. According to another embodiment of this application, the expression tracking device 300 involved in the embodiments of this application and the expression tracking method of the embodiments of this application can be constructed by running a computer program (including program code) capable of performing the steps involved in the corresponding method on a general-purpose computing device including processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM). Computer programs can be recorded on, for example, a computer-readable storage medium, loaded into an electronic device via the computer-readable storage medium, and run therein to implement the corresponding methods provided in the embodiments of this application.

[0210] In other words, the units mentioned above can be implemented in hardware, in software instructions, or in a combination of hardware and software. Specifically, the steps of the method embodiments in this application can be completed by the integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software in the decoding processor. Optionally, the software can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads the information in the memory and completes the steps in the above method embodiments in conjunction with its hardware.

[0211] Figure 10 This is a schematic structural diagram of the electronic device 400 provided in the embodiments of this application.

[0212] like Figure 10 As shown, the electronic device 400 includes at least a processor 410 and a computer-readable storage medium 420. The processor 410 and the computer-readable storage medium 420 can be connected via a bus or other means. The computer-readable storage medium 420 stores a computer program 421, which includes computer instructions. The processor 410 executes the computer instructions stored in the computer-readable storage medium 420. The processor 410 is the computing and control core of the electronic device 400, and is adapted to implement one or more computer instructions, specifically to load and execute one or more computer instructions to achieve a corresponding method flow or function.

[0213] As an example, processor 410 may also be referred to as a central processing unit (CPU). Processor 410 may include, but is not limited to: general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0214] As an example, the computer-readable storage medium 420 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device; optionally, it may also be at least one computer-readable storage medium located remotely from the aforementioned processor 410. Specifically, the computer-readable storage medium 420 includes, but is not limited to, volatile memory and / or non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0215] like Figure 10 As shown, the electronic device 400 may also include a transceiver 430.

[0216] The processor 410 can control the transceiver 430 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 430 may include a transmitter and a receiver. The transceiver 430 may further include antennas, and the number of antennas may be one or more.

[0217] It should be understood that the various components in the communication device 400 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0218] In one implementation, the electronic device 400 can be any electronic device with data processing capabilities; the computer-readable storage medium 420 stores first computer instructions; the processor 410 loads and executes the first computer instructions stored in the computer-readable storage medium 420 to achieve... Figure 1 The corresponding steps in the method embodiment shown; in specific implementation, the first computer instruction in the computer-readable storage medium 420 is loaded by the processor 410 and the corresponding steps are executed. To avoid repetition, they will not be described again here.

[0219] According to another aspect of this application, embodiments of this application also provide a computer-readable storage medium (Memory), which is a memory device in electronic device 400 for storing programs and data. For example, computer-readable storage medium 420. It is understood that computer-readable storage medium 420 here may include both built-in storage media in electronic device 400 and extended storage media supported by electronic device 400. The computer-readable storage medium provides storage space that stores the operating system of electronic device 400. Furthermore, the storage space also stores one or more computer instructions suitable for loading and execution by processor 410, which may be one or more computer programs 421 (including program code).

[0220] According to another aspect of this application, embodiments of this application also provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. For example, computer program 421. In this case, the data processing device 400 may be a computer, and the processor 410 reads the computer instructions from the computer-readable storage medium 420. The processor 410 executes the computer instructions, causing the computer to perform the facial expression tracking method provided in the various alternative methods described above.

[0221] In other words, when implemented using software, it can be implemented entirely or partially in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes of the embodiments of this application are run or the functions of the embodiments of this application are implemented. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0222] Those skilled in the art will recognize that the units and process steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0223] Finally, it should be noted that the above content is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A facial expression tracking method, characterized in that, include: Obtain the neutral face model of the target person and the M expression bases (BS) corresponding to the neutral face model, where M > 0; The face of the target person is captured from multiple viewpoints to obtain multiple current images corresponding to the multiple viewpoints respectively; Using an expression tracking algorithm, calculate M BS coefficients that are used to track facial expressions in the multiple current images and correspond to the M expression bases BS respectively; Based on the neutral face model, the M expression bases (BS), and the M BS coefficients, multiple tracking images of the current image are constructed.

2. The method according to claim 1, characterized in that, The process of using an expression tracking algorithm to calculate M BS coefficients corresponding to the M expression bases (BSs) in the multiple current images for tracking facial expressions includes: Using the M expression bases (BS) and the M initial coefficients corresponding to the M expression bases (BS), an intermediate image with expressions is constructed. Based on the pose information of the neutral face model, the intermediate image is projected onto the multiple current images to obtain multiple projected images; Calculate at least one loss of the plurality of projected images; Based on the at least one loss, the M initial coefficients are adjusted to obtain the M BS coefficients.

3. The method according to claim 2, characterized in that, The at least one loss includes a first loss; the first loss is characterized as the difference between facial key points in the first current image and facial key points in the first projected image; Wherein, the first current image is an image of the target person's face captured in the middle view direction of the multiple view directions, and the first projected image is a projected image obtained by projecting the middle image onto the first current image.

4. The method according to claim 2, characterized in that, The at least one loss includes a second loss; the second loss is characterized as at least one of the following: the error between facial key points in the left region of the second current image and facial key points in the left region of the second projected image, and the difference between facial key points in the right region of the third current image and facial key points in the right region of the third projected image; Wherein, the second current image is an image of the target person's face captured in the left view direction among the multiple view directions, and the second projected image is a projected image obtained by projecting the intermediate image onto the second current image; The third current image is an image of the target person's face captured in the right-side view direction among the multiple view directions, and the third projected image is a projected image obtained by projecting the intermediate image onto the third current image.

5. The method according to claim 2, characterized in that, The at least one loss includes a third loss, which is characterized as the optical flow error between the plurality of projected images and the plurality of reference images, respectively. The plurality of reference images are projection images obtained by projecting the intermediate image onto the plurality of adjacent images, and the plurality of adjacent images are images acquired in the plurality of viewing directions before the plurality of current images are acquired, used for tracking facial expressions, and adjacent to the plurality of current images.

6. The method according to claim 2, characterized in that, The at least one loss includes a fourth loss, which is characterized as at least one of the following: The sum of the M initial coefficients, the difference between the sum of the M initial coefficients and the M reference coefficients, and the difference between the pose information of the multiple current images and the pose information of the multiple adjacent images; Wherein, the plurality of adjacent images are images acquired in the plurality of viewing directions before the plurality of current images are acquired, used for tracking facial expressions, and adjacent to the plurality of current images, and the M reference coefficients are the BS coefficients of the tracking images used to construct the plurality of adjacent images.

7. The method according to any one of claims 1 to 6, characterized in that, The process of constructing multiple tracking images for the current image based on the neutral face model, the M expression bases (BS), and the M BS coefficients includes: If the M expression bases BS correspond to M regions of the face respectively, and M is an even number, then when the first coefficient corresponding to the first region and the second coefficient corresponding to the second region are different among the M BS coefficients, and the first region and the second region are symmetrical, the average value of the first coefficient and the second coefficient is determined. The average value is determined as the coefficient corresponding to the first region and the coefficient corresponding to the second region; Based on the neutral face model, the M expression bases (BS), the coefficients corresponding to the regions other than the first and second regions in the M regions, the coefficients corresponding to the first region, and the coefficients corresponding to the second region, a tracking image of the multiple current images is constructed.

8. The method according to any one of claims 1 to 6, characterized in that, The process of obtaining the neutral face model of the target person and the M expression bases (BS) corresponding to the neutral face model includes: Acquire multiple videos captured from the multiple viewpoint directions; The neutral face model is obtained by constructing a 3D model of the target person's face using the multiple videos. Obtain the base pattern (BS) corresponding to the average face model; The base expression (BS) corresponding to the average face model is transferred to the neutral face model to obtain the M expression base expressions (BS).

9. The method according to claim 8, characterized in that, The process of constructing a 3D model of the target person's face using the multiple videos to obtain the neutral face model includes: Frames are extracted from each of the multiple videos to obtain N images; Select K images from the N images; N ≥ K > 0; The neutral face model is obtained by constructing a 3D model of the target person's face based on the K images.

10. The method according to claim 9, characterized in that, The step of selecting K images from the N images includes: To obtain the K images, delete at least one of the following images from the N images: Images in which the difference between facial landmarks and facial landmarks in a neutral face image exceeds a preset range, images in which the variance is less than or equal to a first preset threshold, and the second image among two adjacent images; Wherein, the variance of the difference between the second image and the first of the two images is less than or equal to a second preset threshold.

11. The method according to claim 9, characterized in that, The step of constructing a 3D model of the target person's face using the multiple videos to obtain the neutral face model includes: Based on the K images, a 3D model of the target person's face is constructed to obtain a pre-constructed model; Based on the pose information of the pre-built model, the pre-built model is projected onto the K images respectively to obtain K projected images; Calculate the number of facial key points matched between the K images and the K projected images respectively; From the K images, select images in which the number of matched facial key points is greater than or equal to a third preset threshold to obtain T images; K≥T>1; The neutral face model is obtained by constructing a 3D model of the target person's face based on the T images.

12. The method according to claim 11, characterized in that, The process of constructing a 3D model of the target person's face based on the T images to obtain the neutral face model includes: Based on the T images, a three-dimensional model of the target person's face is constructed to obtain a first intermediate model; The vertex coordinates of the intermediate model are smoothed to obtain the second intermediate model; Repair the deformed areas in the second intermediate model to obtain the third intermediate model; The neutral face model is obtained by scanning the third intermediate model using the non-rigid iterative nearest point (NICP) algorithm.

13. An expression tracking device, characterized in that, include: The acquisition unit is used to acquire the neutral face model of the target person and the M expression bases BS corresponding to the neutral face model, where M>0; The acquisition unit is used to acquire the face of the target person from multiple perspectives, and obtain multiple current images corresponding to the multiple perspectives respectively; The calculation unit is used to calculate M BS coefficients for tracking facial expressions in the multiple current images and corresponding to the M expression bases BS respectively using an expression tracking algorithm; The construction unit is used to construct multiple tracking images of the current image based on the neutral face model, the M expression bases (BS), and the M BS coefficients.

14. An electronic device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 12.

16. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the method as described in any one of claims 1 to 12.

Images