A method, device, storage medium, and terminal for capturing facial expressions.

Abstract

This invention discloses a method, apparatus, storage medium, and terminal for capturing facial expressions. The method includes: acquiring a face image to be recognized and extracting a target implicit code from the face image; inputting the face image to be recognized and the target implicit code into a pre-trained facial expression capture model, and outputting facial expression information corresponding to the face image to be recognized; wherein the facial expression information is generated based on facial expression features, and the facial expression features are generated by fusing multiple extracted features. Because this application extracts multiple features and fuses them into new features for model training, it can effectively improve the model's accuracy in capturing facial expression information.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a method, apparatus, storage medium, and terminal for capturing facial expressions. Background Technology

[0002] Facial expressions are the outward manifestations of emotions and feelings. According to the basic emotion model, facial expressions can be categorized into six types: anger, disgust, fear, happiness, sadness, and surprise. Facial expression recognition has always been of great research significance and has enormous market value in various fields such as human-computer interaction, public safety, and intelligent film and television. With the continuous development of machine learning, researchers are eager to improve the accuracy of facial expression information capture.

[0003] Existing technical solutions include those that utilize 2D / 3D facial keypoints as the driving force, but because the face occupies a small portion of the overall image and issues such as motion blur and occlusion are common, the accuracy and stability of keypoint detection are limited. Other methods involve modeling the face and then regressing expression parameters through deep learning, but insufficient modeling accuracy and limited coverage of the expression feature space lead to inaccurate expressions and insufficient expression levels. Furthermore, the stability across various natural scenes reduces the accuracy of capturing facial expression information. Summary of the Invention

[0004] This application provides a method, apparatus, storage medium, and terminal for capturing facial expressions. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general description, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0005] In a first aspect, embodiments of this application provide a method for capturing facial expressions, the method comprising:

[0006] Acquire the face image to be identified and extract the target implicit code from the face image to be identified;

[0007] The image of the face to be identified and the implicit code of the target are input into a pre-trained facial expression capture model, which outputs the facial expression information corresponding to the image of the face to be identified; among which,

[0008] Facial expression information is generated based on facial expression features, which are generated by fusing multiple extracted features.

[0009] Optionally, acquiring the face image to be identified and extracting the target implicit code of the face image to be identified includes:

[0010] Acquire the image of the face to be identified;

[0011] A meta-learner is used to extract the target implicit code of the face image to be identified from a pre-generated random implicit coding matrix.

[0012] Optionally, the pre-generated random implicit coding matrix is ​​obtained by following these steps:

[0013] Collect facial expression data;

[0014] Calculate the prior distribution of facial expression data;

[0015] Construct a random implicit coding matrix based on the prior distribution.

[0016] Optionally, the pre-trained facial expression capture model includes a general feature extraction module, an identity ID extraction module, an expression extraction module, an expression optimization module, a head pose extraction module, and a fusion module;

[0017] The image of the face to be identified and the target implicit code are input into a pre-trained facial expression capture model, which outputs the facial expression information corresponding to the image of the face to be identified, including:

[0018] The image of the face to be identified is input into the general feature extraction module, which outputs the feature space of the image of the face to be identified.

[0019] The target implicit encoding and identity ID extraction module are used to extract the identity ID features corresponding to the face image to be identified from the feature space;

[0020] The expression extraction module and the expression optimization module are used to extract the final expression features corresponding to the face image to be recognized from the feature space;

[0021] A head pose extraction module is used to extract the head pose features corresponding to the face image to be identified from the feature space.

[0022] The identity ID features, final expression features, and head pose features are input into the fusion module for feature fusion, and the facial expression features corresponding to the face image to be identified are output.

[0023] Recover the facial expression information corresponding to the face image to be identified based on facial expression features.

[0024] Optionally, an expression extraction module and an expression optimization module are used to extract the final expression features corresponding to the face image to be recognized from the feature space, including:

[0025] An expression extraction module is used to extract regional features of different facial regions corresponding to the face image to be identified from the feature space;

[0026] Acquire multiple natural expression images under different lighting conditions associated with the face image to be identified;

[0027] The expression optimization module was used to extract natural expression features from multiple natural expression images under different lighting conditions;

[0028] The final expression features corresponding to the face image to be identified are determined based on the regional features of different face regions and the extracted natural expression features.

[0029] Optionally, the final expression features corresponding to the face image to be identified are determined based on the regional features of different face regions and the extracted natural expression features, including:

[0030] The regional features of different facial regions are fused with the extracted natural expression features to obtain the first feature;

[0031] The second feature is obtained by performing facial expression encoding on multiple natural facial expression images under different lighting conditions;

[0032] The difference between the first feature and the second feature is calculated to obtain the final expression feature corresponding to the face image to be identified.

[0033] Optionally, a pre-trained facial expression capture model is generated by following these steps:

[0034] Collect and label multiple facial expression data to obtain labeled facial expression data;

[0035] Data augmentation is performed on the labeled facial expression data to obtain augmented data;

[0036] Create a facial expression capture model;

[0037] Obtain the implicit encoding of each data point in the augmented data;

[0038] Each data point in the augmented data and its corresponding implicit code are sequentially input into the facial expression capture model for model training, and the model loss value is output.

[0039] When the model loss value reaches its minimum, a pre-trained facial expression capture model is generated.

[0040] Secondly, embodiments of this application provide a facial expression capture device, the device comprising:

[0041] The implicit encoding acquisition module is used to acquire the face image to be recognized and extract the target implicit encoding of the face image to be recognized;

[0042] The facial expression information output module is used to input the image of the face to be recognized and the target implicit code into a pre-trained facial expression capture model, and output the facial expression information corresponding to the image of the face to be recognized; wherein,

[0043] Facial expression information is generated based on facial expression features, which are generated by fusing multiple extracted features.

[0044] Thirdly, embodiments of this application provide a computer storage medium storing multiple instructions adapted for loading and execution of the above-described method steps by a processor.

[0045] Fourthly, embodiments of this application provide a terminal that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed by the above-described method steps.

[0046] The technical solutions provided in this application embodiment may include the following beneficial effects:

[0047] In this embodiment, the facial expression capture device first acquires a face image to be recognized and extracts the target implicit code from the face image. Then, it inputs the face image to be recognized and the target implicit code into a pre-trained facial expression capture model and outputs the facial expression information corresponding to the face image to be recognized. The facial expression information is generated based on facial expression features, which are generated by fusing multiple extracted features. Because this application extracts multiple features and fuses them into new features for model training, it can effectively improve the model's accuracy in capturing facial expression information.

[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0050] Figure 1 This is a flowchart illustrating a facial expression capture method provided in an embodiment of this application;

[0051] Figure 2 This is a schematic block diagram illustrating a facial expression capture process provided in an embodiment of this application;

[0052] Figure 3 This is a flowchart illustrating a method for training a facial expression capture model according to an embodiment of this application.

[0053] Figure 4 This is a flowchart illustrating the overall process of a facial expression capture method provided in an embodiment of this application.

[0054] Figure 5 This is a schematic diagram of the structure of a facial expression capture device provided in an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Detailed Implementation

[0056] The following description and accompanying drawings fully illustrate specific embodiments of the invention to enable those skilled in the art to practice them.

[0057] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0058] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0059] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances. Furthermore, in the description of this invention, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0060] This application provides a method, apparatus, storage medium, and terminal for capturing facial expressions to address the problems existing in the aforementioned related technologies. In the technical solution provided by this application, because multiple features are extracted and fused into new features for model training, the accuracy of the model in capturing facial expression information can be effectively improved. The following detailed description uses exemplary embodiments.

[0061] The following will be combined with the appendix Figure 1 - Appendix Figure 4 This application provides a detailed description of the facial expression capture method provided in its embodiments. This method can be implemented using a computer program and can run on a von Neumann-based facial expression capture device. The computer program can be integrated into an application or run as a standalone utility application.

[0062] Please see Figure 1This is a flowchart illustrating a facial expression capture method provided in this application embodiment.

[0063] like Figure 1 As shown, the method in this application embodiment may include the following steps:

[0064] S101, acquire the face image to be identified and extract the target implicit code of the face image to be identified;

[0065] The face image to be identified is acquired through an image acquisition device. The image acquisition device in this application can be a single ordinary camera. Using a single ordinary camera in combination with the face expression capture device in this application can simplify the face expression capture process.

[0066] In one possible implementation, when extracting the target implicit code of the face image to be identified, the face image to be identified is first acquired through a single ordinary camera, and then a meta-learner is used to extract the target implicit code of the face image to be identified from a pre-generated random implicit coding matrix.

[0067] Further, the pre-generated random implicit coding matrix is ​​obtained by following these steps: first, facial expression data is collected; then, the prior distribution of the facial expression data is calculated; and finally, the random implicit coding matrix is ​​constructed based on the prior distribution.

[0068] S102, input the face image to be identified and the target implicit code into the pre-trained face expression capture model, and output the face expression information corresponding to the face image to be identified;

[0069] Among them, facial expression information is generated based on facial expression features, which are generated by fusing multiple extracted features.

[0070] Typically, a pre-trained facial expression capture model includes a general feature extraction module, an identity ID extraction module, an expression extraction module, an expression optimization module, a head pose extraction module, and a fusion module.

[0071] In this embodiment, when outputting the facial expression information corresponding to the face image to be recognized, the face image to be recognized is first input into the general feature extraction module, which outputs the feature space of the face image to be recognized. Then, the target implicit coding and identity ID extraction module extracts the identity ID feature corresponding to the face image to be recognized from the feature space. Then, the expression extraction module and expression optimization module extract the final expression feature corresponding to the face image to be recognized from the feature space. Next, the head pose extraction module extracts the head pose feature corresponding to the face image to be recognized from the feature space. Then, the identity ID feature, the final expression feature, and the head pose feature are input into the fusion module for feature fusion, and the facial expression feature corresponding to the face image to be recognized is output. Finally, the facial expression information corresponding to the face image to be recognized is recovered based on the facial expression feature.

[0072] Specifically, when using the expression extraction module and the expression optimization module to extract the final expression features corresponding to the face image to be recognized from the feature space, the expression extraction module first extracts the regional features of different face regions corresponding to the face image to be recognized from the feature space. Then, it obtains multiple natural expression images under different lighting conditions associated with the face image to be recognized. Next, the expression optimization module extracts the natural expression features of the multiple natural expression images under different lighting conditions. Finally, the final expression features corresponding to the face image to be recognized are determined based on the regional features of different face regions and the extracted natural expression features.

[0073] Specifically, when determining the final expression features corresponding to the face image to be recognized based on the regional features of different face regions and the extracted natural expression features, the regional features of different face regions and the extracted natural expression features are first fused to obtain the first feature. Then, expression encoding is performed on multiple natural expression images under different lighting conditions to obtain the second feature. Finally, the difference between the first feature and the second feature is calculated to obtain the final expression features corresponding to the face image to be recognized.

[0074] In one possible implementation, facial expressions pose a significant technical challenge for face capture due to their richness, complexity, and dynamism. Most techniques compromise on these three aspects: either they offer sufficient richness but lack complexity, or they offer sufficient complexity but poor dynamic stability. This application addresses this by using an expression extraction module to encode the location of expressions in blocks, dividing the face into different regions for feature extraction. This ensures local accuracy and guarantees the correct extraction of up to 600 micro-expressions.

[0075] Furthermore, since different regions can combine, this application fully utilizes these characteristics of the face, introducing spatial association concepts and temporal mechanisms in the subsequent network design to further process the features extracted by the expression extraction module. The spatial association concept enables the network to learn to adaptively select combinations, thereby achieving global richness and complexity; finally, a temporal mechanism is introduced into the network, allowing it to utilize preceding and following frames to ultimately achieve dynamic stability of expressions.

[0076] Building upon the above, facial expressions not only share similarities but also possess strong individual characteristics. For example, a hearty laugh can be wide-open on one person but less pronounced on another. To address this issue of semantically consistent yet varying degrees of expression, this application introduces an expression optimization model as a fine-tuning module to participate in the training of the entire network. Specifically, 5-10 natural facial expression images under different lighting conditions are pre-collected for each individual. The expression optimization module extracts natural expression features from these images, initially fusing them with the features extracted by the expression extraction module to obtain the first feature. Simultaneously, expression encoding is performed on the 5-10 natural expression images under different lighting conditions to obtain the second feature. After obtaining these two features, the difference between the feature pairs is calculated, and the resulting feature is used as the final output expression feature of the expression optimization module. The final output expression feature effectively extracts personalized representations, which, combined with other networks, achieve personalized expression control.

[0077] Furthermore, after obtaining the facial expression features output by the expression optimization module, considering that humans move their heads along with their facial expressions, only by combining facial expressions and head movements can the expressions be more natural and expressive. Therefore, this solution designs a head pose extraction module. The task of this module is to accurately capture the movements of the human head, including left and right rotation, up and down tilting, and combined postures, and generate pose features based on the captured combined posture information.

[0078] Furthermore, when recovering facial expression information corresponding to the face image to be recognized based on facial expression features, some extreme cases exist during the expression capture process, such as excessively blurred facial movement or an angle that makes the face incomplete, which affect the final capture effect. To solve these problems, this application incorporates spatiotemporal information while maintaining the generalization ability of the original network. The design idea of ​​this network is to recover the expression information of the current frame by finding similar frames or normal frames before and after the ill-formed frames, thus making the network more stable and robust.

[0079] For example Figure 2 As shown, Figure 2This is a schematic flowchart illustrating the applicant's facial expression capture process. First, the image of the face to be recognized is acquired and the target implicit code of the face image is extracted. Then, the face image to be recognized is input into a general feature extraction module to extract the feature space. The target implicit code and the identity ID extraction module are used to extract the identity ID feature in the feature space. Then, the expression extraction module and the expression optimization module are used to extract the final expression feature in the feature space. Next, the head pose extraction module is used to extract the head pose feature in the feature space. Finally, the identity ID feature, the final expression feature, and the head pose feature are input into a fusion module for feature fusion, outputting the facial expression feature corresponding to the face image to be recognized. The facial expression information corresponding to the face image to be recognized can be recovered based on the facial expression feature.

[0080] Furthermore, after recovering the facial expression information, the facial expression information can be transferred to virtual characters for practical application.

[0081] In this embodiment, the facial expression capture device first acquires a face image to be recognized and extracts the target implicit code from the face image. Then, it inputs the face image to be recognized and the target implicit code into a pre-trained facial expression capture model and outputs the facial expression information corresponding to the face image to be recognized. The facial expression information is generated based on facial expression features, which are generated by fusing multiple extracted features. Because this application extracts multiple features and fuses them into new features for model training, it can effectively improve the model's accuracy in capturing facial expression information.

[0082] Please see Figure 3 This is a flowchart illustrating a method for training a facial expression capture model, as provided in an embodiment of this application. Figure 3 As shown, the method in this application embodiment may include the following steps:

[0083] S201, Collect and label multiple facial expression data to obtain labeled facial expression data;

[0084] S202, perform data augmentation on the labeled facial expression data to obtain augmented data;

[0085] In one possible implementation, multiple facial expression data sets are first collected. Then, these data sets are manually labeled. Finally, data augmentation operations are performed on the labeled data to enrich it. Data augmentation includes at least data translation and rotation operations.

[0086] S203, Create a facial expression capture model;

[0087] S204, Obtain the implicit code for each data point in the augmented data;

[0088] S205, input each data point in the augmented data and its corresponding implicit code into the facial expression capture model in sequence for model training, and output the model loss value;

[0089] In one possible implementation, during model training, this application introduces a random implicit code for each training data point into the input, and applies a code consistency loss function to the identity ID feature of each training data point in the output. That is, it calculates the similarity between the identity ID feature and its corresponding implicit code, and uses this similarity as the model's loss value for output. When the loss value reaches the minimum value, it can be determined that the model has converged.

[0090] S206, When the model loss value reaches its minimum, a pre-trained facial expression capture model is generated.

[0091] In one possible implementation, a pre-trained facial expression capture model is generated when the model loss value reaches its minimum, or if the model loss value has not reached its minimum, the loss value is backpropagated to update the model's parameters, and the model continues to be trained until the loss value reaches its minimum.

[0092] For example Figure 4 As shown, Figure 4 This is a flowchart illustrating the overall process of a facial expression capture method provided in this application. Figure 4 The base model is a facial expression capture model. First, n facial images are collected, and the implicit encoding of each facial image is input into the base model for training. After the base model is trained, a pre-trained facial expression capture model is obtained. In practical application scenarios, after obtaining a new facial image, the implicit encoding of the new facial image is extracted through a meta-learner. The new facial image and its implicit encoding are then input into the trained base model to obtain the identity ID feature of the new face and other features, which can then be fused.

[0093] Based on the prior characteristic of implicit encoding, this application introduces a meta-learning mechanism and uses an ID encoding network. At the same time, it introduces an encoding consistency loss function, which enables the network to extract a unique ID encoding for any person.

[0094] In this embodiment, the facial expression capture device first acquires a face image to be recognized and extracts the target implicit code from the face image. Then, it inputs the face image to be recognized and the target implicit code into a pre-trained facial expression capture model and outputs the facial expression information corresponding to the face image to be recognized. The facial expression information is generated based on facial expression features, which are generated by fusing multiple extracted features. Because this application extracts multiple features and fuses them into new features for model training, it can effectively improve the model's accuracy in capturing facial expression information.

[0095] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the embodiments of the apparatus of the present invention, please refer to the embodiments of the method of the present invention.

[0096] Please see Figure 5 This illustration shows a schematic diagram of a facial expression capture device provided in an exemplary embodiment of the present invention. This facial expression capture device can be implemented as all or part of a terminal through software, hardware, or a combination of both. The device 1 includes an implicit encoding acquisition module 10 and a facial expression information output module 20.

[0097] The implicit encoding acquisition module 10 is used to acquire the face image to be identified and extract the target implicit encoding of the face image to be identified;

[0098] The facial expression information output module 20 is used to input the image of the face to be recognized and the target implicit code into a pre-trained facial expression capture model, and output the facial expression information corresponding to the image of the face to be recognized; wherein,

[0099] Facial expression information is generated based on facial expression features, which are generated by fusing multiple extracted features.

[0100] It should be noted that the facial expression capture device provided in the above embodiments is only illustrated by the division of the above functional modules when performing the facial expression capture method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the facial expression capture device and the facial expression capture method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0101] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0102] In this embodiment, the facial expression capture device first acquires a face image to be recognized and extracts the target implicit code from the face image. Then, it inputs the face image to be recognized and the target implicit code into a pre-trained facial expression capture model and outputs the facial expression information corresponding to the face image to be recognized. The facial expression information is generated based on facial expression features, which are generated by fusing multiple extracted features. Because this application extracts multiple features and fuses them into new features for model training, it can effectively improve the model's accuracy in capturing facial expression information.

[0103] The present invention also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the facial expression capture method provided in the above-described method embodiments.

[0104] The present invention also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the facial expression capture methods of the above-described method embodiments.

[0105] Please see Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Figure 6 As shown, terminal 1000 may include: at least one processor 1001, at least one network interface 1004, user interface 1003, memory 1005, and at least one communication bus 1002.

[0106] The communication bus 1002 is used to realize the connection and communication between these components.

[0107] The user interface 1003 may include a display screen and a camera. Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0108] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0109] The processor 1001 may include one or more processing cores. The processor 1001 connects to various parts within the electronic device 1000 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005. Optionally, the processor 1001 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 1001 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip, without being integrated into the processor 1001.

[0110] The memory 1005 may include random access memory (RAM) or read-only memory. Optionally, the memory 1005 may include a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 6 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a facial expression capture application.

[0111] exist Figure 6In the terminal 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and to acquire user input data; while the processor 1001 can be used to call the facial expression capture application stored in the memory 1005 and specifically perform the following operations:

[0112] Acquire the face image to be identified and extract the target implicit code from the face image to be identified;

[0113] The image of the face to be identified and the implicit code of the target are input into a pre-trained facial expression capture model, which outputs the facial expression information corresponding to the image of the face to be identified; among which,

[0114] Facial expression information is generated based on facial expression features, which are generated by fusing multiple extracted features.

[0115] In one embodiment, when the processor 1001 performs the operations of acquiring the face image to be identified and extracting the target implicit encoding of the face image to be identified, it specifically performs the following operations:

[0116] Acquire the image of the face to be identified;

[0117] A meta-learner is used to extract the target implicit code of the face image to be identified from a pre-generated random implicit coding matrix.

[0118] In one embodiment, when generating the random implicit encoding matrix, the processor 1001 also performs the following operations:

[0119] Collect facial expression data;

[0120] Calculate the prior distribution of facial expression data;

[0121] Construct a random implicit coding matrix based on the prior distribution.

[0122] In one embodiment, when the processor 1001 executes a pre-trained facial expression capture model that inputs the image of the face to be recognized and the target implicit code into the model, and outputs the facial expression information corresponding to the image of the face to be recognized, it specifically performs the following operations:

[0123] The image of the face to be identified is input into the general feature extraction module, which outputs the feature space of the image of the face to be identified.

[0124] The target implicit encoding and identity ID extraction module are used to extract the identity ID features corresponding to the face image to be identified from the feature space;

[0125] The expression extraction module and the expression optimization module are used to extract the final expression features corresponding to the face image to be recognized from the feature space;

[0126] A head pose extraction module is used to extract the head pose features corresponding to the face image to be identified from the feature space.

[0127] The identity ID features, final expression features, and head pose features are input into the fusion module for feature fusion, and the facial expression features corresponding to the face image to be identified are output.

[0128] Recover the facial expression information corresponding to the face image to be identified based on facial expression features.

[0129] In one embodiment, when the processor 1001 extracts the final expression features corresponding to the face image to be recognized from the feature space using the expression extraction module and the expression optimization module, it specifically performs the following operations:

[0130] An expression extraction module is used to extract regional features of different facial regions corresponding to the face image to be identified from the feature space;

[0131] Acquire multiple natural expression images under different lighting conditions associated with the face image to be identified;

[0132] The expression optimization module was used to extract natural expression features from multiple natural expression images under different lighting conditions;

[0133] The final expression features corresponding to the face image to be identified are determined based on the regional features of different face regions and the extracted natural expression features.

[0134] In one embodiment, when the processor 1001 determines the final expression features corresponding to the face image to be identified based on the regional features of different face regions and the extracted natural expression features, it specifically performs the following operations:

[0135] The regional features of different facial regions are fused with the extracted natural expression features to obtain the first feature;

[0136] The second feature is obtained by performing facial expression encoding on multiple natural facial expression images under different lighting conditions;

[0137] The difference between the first feature and the second feature is calculated to obtain the final expression feature corresponding to the face image to be identified.

[0138] In one embodiment, when generating a pre-trained facial expression capture model, the processor 1001 performs the following operations:

[0139] Collect and label multiple facial expression data to obtain labeled facial expression data;

[0140] Data augmentation is performed on the labeled facial expression data to obtain augmented data;

[0141] Create a facial expression capture model;

[0142] Obtain the implicit encoding of each data point in the augmented data;

[0143] Each data point in the augmented data and its corresponding implicit code are sequentially input into the facial expression capture model for model training, and the model loss value is output.

[0144] When the model loss value reaches its minimum, a pre-trained facial expression capture model is generated.

[0145] In this embodiment, the facial expression capture device first acquires a face image to be recognized and extracts the target implicit code from the face image. Then, it inputs the face image to be recognized and the target implicit code into a pre-trained facial expression capture model and outputs the facial expression information corresponding to the face image to be recognized. The facial expression information is generated based on facial expression features, which are generated by fusing multiple extracted features. Because this application extracts multiple features and fuses them into new features for model training, it can effectively improve the model's accuracy in capturing facial expression information.

[0146] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The facial expression capture program can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0147] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for capturing facial expressions, characterized in that, The method includes: Acquire the face image to be identified and extract the target implicit code from the face image to be identified; The process of acquiring the face image to be identified and extracting the target implicit code of the face image to be identified includes: Acquire the image of the face to be identified; A meta-learner is used to extract the target implicit code of the face image to be identified from a pre-generated random implicit coding matrix. Based on the prior property of implicit coding, a meta-learning mechanism is introduced. Through the ID coding network, a coding consistency loss function is also introduced, so that the network can extract a unique ID code for any person. The pre-generated random implicit coding matrix is ​​obtained by following these steps: Collect facial expression data; Calculate the prior distribution of the facial expression data; Construct a random implicit coding matrix based on the prior distribution; The image of the face to be identified and the target implicit code are input into a pre-trained facial expression capture model, which outputs the facial expression information corresponding to the image of the face to be identified; wherein, The facial expression information is generated based on facial expression features, which are generated by fusing multiple extracted features. The pre-trained facial expression capture model includes a general feature extraction module, an identity ID extraction module, an expression extraction module, an expression optimization module, a head pose extraction module, and a fusion module. The step of inputting the image of the face to be identified and the target implicit code into a pre-trained facial expression capture model, and outputting the facial expression information corresponding to the image of the face to be identified, includes: The face image to be identified is input into the general feature extraction module, which outputs the feature space of the face image to be identified. The target implicit encoding and the identity ID extraction module are used to extract the identity ID feature corresponding to the face image to be identified from the feature space. The expression extraction module and the expression optimization module are used to extract the final expression feature corresponding to the face image to be identified from the feature space. The head pose extraction module is used to extract the head pose feature corresponding to the face image to be identified from the feature space. The identity ID feature, the final expression feature, and the head pose feature are input into the fusion module for feature fusion, and the face expression feature corresponding to the face image to be identified is output. The face expression information corresponding to the face image to be identified is recovered based on the face expression feature.

2. The method according to claim 1, characterized in that, The step of extracting the final expression features corresponding to the face image to be identified from the feature space using the expression extraction module and the expression optimization module includes: The expression extraction module is used to extract regional features of different facial regions corresponding to the face image to be identified from the feature space; Acquire multiple natural expression images under different lighting conditions associated with the face image to be identified; The expression optimization module is used to extract natural expression features from multiple natural expression images under different lighting conditions; The final expression features corresponding to the face image to be identified are determined based on the regional features of the different face regions and the extracted natural expression features.

3. The method according to claim 2, characterized in that, The final expression features corresponding to the face image to be identified are determined based on the regional features of the different face regions and the extracted natural expression features, including: The regional features of the different facial regions are fused with the extracted natural expression features to obtain the first feature; The expression encoding operation is performed on the multiple natural expression images under different lighting conditions to obtain the second feature; The difference between the first feature and the second feature is calculated to obtain the final expression feature corresponding to the face image to be identified.

4. The method according to claim 1, characterized in that, Generate a pre-trained facial expression capture model by following these steps: Collect and label multiple facial expression data to obtain labeled facial expression data; Data augmentation is performed on the labeled facial expression data to obtain augmented data; Create a facial expression capture model; Obtain the implicit encoding of each data point in the enhanced data; Each data point in the augmented data and its corresponding implicit code are sequentially input into the facial expression capture model for model training, and the model loss value is output. When the model loss value reaches its minimum, a pre-trained facial expression capture model is generated.

5. A facial expression capture device implemented using the method described in any one of claims 1-4, characterized in that, The device includes: The implicit encoding acquisition module is used to acquire the face image to be recognized and extract the target implicit encoding of the face image to be recognized; A facial expression information output module is used to input the image of the face to be recognized and the target implicit code into a pre-trained facial expression capture model, and output the facial expression information corresponding to the image of the face to be recognized; wherein, The facial expression information is generated based on facial expression features, which are generated by fusing multiple extracted features.

6. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method as claimed in any one of claims 1-4.

7. A terminal, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed as claimed in any one of claims 1-4.

Images