Facial expression recognition model training and facial expression recognition method, device, equipment and medium

Abstract

The application provides an expression recognition model training and expression recognition method and device, electronic equipment and storage medium. The expression recognition model training method comprises obtaining a first expression representation model and a second expression representation model through self-supervised pre-training; fine-tuning the first expression representation model to obtain a first expression recognition model, and performing knowledge distillation on the first expression representation model based on the first expression recognition model to obtain a second expression recognition model; performing knowledge distillation on the second expression representation model based on the second expression recognition model to obtain a third expression recognition model. Through self-supervised pre-training, a large amount of unlabeled sample data is used to reduce the cost of manual labeling. Through self-distillation on the same expression representation model, the accuracy of the second expression recognition model is improved. And through knowledge distillation of the second expression representation model based on the second expression recognition model, the accuracy of the third expression recognition model is further ensured.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an expression recognition model training and expression recognition method, apparatus, electronic device and storage medium. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this application as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] Facial expressions are a form of nonverbal communication. Humans use facial expressions to express their inner feelings and to perceive the emotions of others. Human facial expressions can be categorized in many ways, most commonly into seven basic emotions: neutral, happy, sad, surprised, afraid, angry, and disgusted. Facial expression recognition, as an important research direction in machine vision, provides crucial information for machines to perceive humans; therefore, it is a vital component of human-computer interaction. However, current facial expression recognition methods face challenges in practical applications, including insufficient accuracy, difficulty adapting to complex environmental changes, susceptibility to occlusion, and insufficient model throughput. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an expression recognition model training and expression recognition method, device, electronic device and storage medium.

[0005] To achieve the above objectives, this application provides a method for training an expression recognition model, comprising:

[0006] A first facial expression representation model and a second facial expression representation model are obtained through self-supervised pre-training; wherein the number of parameters of the baseline model of the first facial expression representation model is greater than the number of parameters of the baseline model of the second facial expression representation model.

[0007] The first expression representation model is fine-tuned to obtain the first expression recognition model, and the first expression recognition model is knowledge distilled based on the first expression recognition model to obtain the second expression recognition model.

[0008] Based on the second expression recognition model, knowledge distillation is performed on the second expression representation model to obtain the third expression recognition model.

[0009] Based on the same inventive concept, an exemplary embodiment of this application also provides an expression recognition method, including:

[0010] The expression image to be recognized is input into the third expression recognition model obtained according to the expression recognition model training method described above, and the expression recognition result of the expression image to be recognized is obtained.

[0011] Based on the same inventive concept, an exemplary embodiment of this application also provides an expression recognition model training device, including:

[0012] The pre-training module obtains a first facial expression representation model and a second facial expression representation model through self-supervised pre-training; wherein the number of parameters of the baseline model of the first facial expression representation model is greater than the number of parameters of the baseline model of the second facial expression representation model.

[0013] The first fine-tuning module fine-tunes the first expression representation model to obtain a first expression recognition model, and performs knowledge distillation on the first expression representation model based on the first expression recognition model to obtain a second expression recognition model.

[0014] The second fine-tuning module performs knowledge distillation on the second expression representation model based on the second expression recognition model to obtain the third expression recognition model.

[0015] Based on the same inventive concept, an exemplary embodiment of this application also provides an expression recognition device, including:

[0016] The recognition module inputs the expression image to be recognized into the third expression recognition model obtained according to the expression recognition model training method described above, and obtains the expression recognition result of the expression image to be recognized.

[0017] Based on the same inventive concept, an exemplary embodiment of this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor executes the program to implement the expression recognition model training or expression recognition method as described above.

[0018] Based on the same inventive concept, an exemplary embodiment of this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the facial expression recognition model training or facial expression recognition method as described above.

[0019] As can be seen from the above, the facial expression recognition model training and facial expression recognition method, device, electronic device, and storage medium provided in this application obtain a first facial expression representation model and a second facial expression representation model through self-supervised pre-training. The first facial expression representation model has a larger number of parameters than the second facial expression representation model. The first facial expression representation model is fine-tuned to obtain a first facial expression recognition model. Knowledge distillation is then performed on the first facial expression representation model based on the first facial expression recognition model to obtain a second facial expression recognition model. Furthermore, knowledge distillation is performed on the second facial expression representation model based on the second facial expression recognition model to obtain a third facial expression recognition model. Through self-supervised pre-training, a large amount of unlabeled sample data is utilized, reducing the cost of manual annotation. Self-distillation on the same facial expression representation model improves the accuracy of the second facial expression recognition model. Knowledge distillation on the second facial expression representation model further ensures the accuracy of the third facial expression recognition model. Simultaneously, because the number of parameters in the second facial expression representation model's base model is smaller, the third facial expression recognition model runs faster and has lower equipment requirements compared to the second facial expression recognition model. It can not only meet the requirements of a larger recognition throughput but also has a wider range of applications. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a method for training an expression recognition model according to an embodiment of this application.

[0022] Figure 2 This is a flowchart illustrating another expression recognition model training method according to an embodiment of this application;

[0023] Figure 3 This is a flowchart illustrating an expression recognition method according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of an expression recognition model training device according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of an expression recognition device according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the structure of a specific electronic device according to an embodiment of this application. Detailed Implementation

[0027] The principles and spirit of this application will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement this application, and are not intended to limit the scope of this application in any way. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0028] According to an embodiment of this application, a method, system, electronic device, and storage medium for training an expression recognition model are proposed.

[0029] In this article, it is important to understand that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0030] The principles and spirit of this application will be explained in detail below with reference to several representative embodiments. Invention Overview

[0032] Currently, most related technologies rely on large amounts of annotated facial image data to improve the robustness of models in recognizing various real-world scenarios. However, acquiring large amounts of annotated data is extremely costly; even the largest publicly available facial expression recognition dataset only contains around 300,000 images, which ultimately limits its coverage of real-world scenarios. Furthermore, both handcrafted feature extraction and deep learning technologies rely on manual modeling of the facial expression recognition problem, such as attention mechanisms weighting facial positions and generative models decoupling identity and facial expression information. While manual modeling, by introducing inductive biases, can help the model learn better feature representations to some extent, it may also introduce unnecessary biases. This is because the assumptions and premises underlying these methods may not perfectly model the problem, limiting the upper limit of accuracy achievable by these methods. In addition, in deep learning solutions, a common method to improve accuracy is to increase the model size; however, the increased computational cost of larger models often cannot meet the latency and throughput requirements of model inference in practical applications. Therefore, many large models are frequently subject to various limitations in practical applications.

[0033] To address the aforementioned problems, this application provides a method for training an expression recognition model, specifically including:

[0034] A first expression representation model and a second expression representation model are obtained through self-supervised pre-training. The baseline model of the first expression representation model has a larger number of parameters than the baseline model of the second expression representation model. The first expression representation model is fine-tuned to obtain a first expression recognition model. Knowledge distillation is then performed on the first expression representation model to obtain a second expression recognition model. Finally, knowledge distillation is performed on the second expression representation model to obtain a third expression recognition model. This self-supervised pre-training utilizes a large amount of unlabeled sample data, reducing the cost of manual annotation. Self-distillation on the same expression representation model improves the accuracy of the second expression recognition model. Furthermore, knowledge distillation on the second expression representation model further ensures the accuracy of the third expression recognition model. Since the baseline model of the second expression representation model has a smaller number of parameters, the third expression recognition model runs faster and has lower equipment requirements compared to the second expression recognition model. It can not only meet the requirements of a larger recognition throughput but also has a wider range of applications.

[0035] After introducing the basic principles of this application, the various non-limiting embodiments of this application will be described in detail below.

[0036] Application Scenarios Overview

[0037] In specific application scenarios, the facial expression recognition model training and facial expression recognition method of this application can be applied to various platforms or systems. As an example, this application scenario includes at least one server and at least one terminal. The server and the terminal can communicate via a network to achieve data transmission. The network can be a wired network or a wireless network; this application does not specifically limit its use.

[0038] The server can be a server that provides various services. Specifically, the server can be used to provide background services for applications running on the terminal. Optionally, in some implementations, the facial expression recognition model training and facial expression recognition method provided in this application embodiment can be executed by the server. The server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services), or as a single software program or software module. This application embodiment does not specifically limit this.

[0039] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can be any network, including but not limited to local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0040] The following describes a facial expression recognition model training method according to an exemplary embodiment of this application, using specific application scenarios. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way. Rather, the embodiments of this application can be applied to any applicable scenario.

[0041] Exemplary methods

[0042] refer to Figure 1 This application provides a method for training an expression recognition model. The subject executing this method may be, but is not limited to, a server or a terminal device. The expression recognition model training method includes the following steps:

[0043] S101, a first expression representation model and a second expression representation model are obtained through self-supervised pre-training; wherein, the number of parameters of the baseline model of the first expression representation model is greater than the number of parameters of the baseline model of the second expression representation model.

[0044] In practice, an unlabeled facial expression sample dataset can be used to perform self-supervised pre-training on the baseline model of the first facial expression representation model to obtain the first facial expression representation model, and the baseline model of the second facial expression representation model can be performed self-supervised pre-training on the baseline model of the second facial expression representation model to obtain the second facial expression representation model. Optionally, there is no limitation on the specific method of self-supervised pre-training; any self-supervised pre-training method currently available in related technologies can be selected.

[0045] It should be noted that the baseline model of the first expression representation model has a greater number of parameters than the baseline model of the second expression representation model. This means the baseline model of the first expression representation model has a more complex structure, belonging to a large-scale network model, and its recognition ability is relatively more accurate. The baseline model of the second expression representation model has a relatively simple structure, belonging to a small-scale network model, and its hardware requirements are lower, making it more adaptable compared to large-scale network models. Optionally, the baseline model of the first expression representation model can be ViT-Base, and the baseline model of the second expression representation model can be ViT-Tiny.

[0046] To obtain a more accurate facial expression representation model, in some embodiments, a first facial expression representation model and a second facial expression representation model are obtained through self-supervised pre-training, specifically including:

[0047] Using an unlabeled facial expression sample dataset and based on mask autoencoder technology, a first facial expression representation model and a second facial expression representation model were obtained, respectively.

[0048] In practice, the Masked Autoencoder (MAE) is an autoencoding method that can be used as a scalable self-supervised learner in computer vision. Its principle mainly involves first masking random parts of the input image and then reconstructing the lost pixels, thereby improving the accuracy of self-supervised learning.

[0049] S102, the first expression representation model is fine-tuned to obtain a first expression recognition model, and knowledge distillation is performed on the first expression representation model based on the first expression recognition model to obtain a second expression recognition model.

[0050] In practice, after obtaining the first facial expression representation model, a fine-tuning facial expression sample dataset can be used to fine-tune the first facial expression recognition model to obtain the second facial expression recognition model. Optionally, this fine-tuning facial expression sample dataset is a small number of labeled datasets. After obtaining the first facial expression recognition model, it can be used as a teacher model to perform knowledge distillation on the first facial expression representation model to obtain the second facial expression recognition model.

[0051] In some embodiments, a first expression recognition model is obtained by fine-tuning the first expression representation model, specifically including:

[0052] A classification layer is added after the encoder of the first expression representation model, and the first expression representation model with the added classification layer is fine-tuned based on the fine-tuned expression sample dataset to obtain the first expression recognition model.

[0053] In specific implementation, a classification layer is added after the encoder of the first expression representation model. That is, the output of the encoder of the first expression representation model is used as the input to the classification layer, and then the classification layer outputs the expression classification result. Optionally, the number of channels in the classification layer can be determined based on the number of classifications in the fine-tuned expression sample dataset. For example, on the ViT-Base model (first expression representation model) and the AffectNet 7-class dataset (fine-tuned expression sample dataset), since the ViT-Base encoder output is a 512-dimensional vector, a 512×7 linear mapping layer can be used as the classification layer. Since the fine-tuned expression sample dataset is a labeled dataset, inputting the labeled expression sample data into the first expression representation model with the added classification layer can output its corresponding expression prediction result. Calculating the label loss between this expression prediction result and the expression sample data allows for fine-tuning of the first expression representation model.

[0054] To further improve the accuracy of fine-tuning, in some embodiments, the first expression representation model after adding a classification layer is fine-tuned based on a fine-tuned expression sample dataset, specifically including:

[0055] Obtain the target number of each expression sample data in the fine-tuned expression sample dataset;

[0056] The compensation weight for each type of facial expression sample data is determined based on the target number.

[0057] The loss of the output of the first facial expression representation model after adding a classification layer is calculated using a first preset loss function and based on the compensation weights.

[0058] In practice, it's difficult to guarantee that the number of each expression sample data in the fine-tuned expression dataset is equal. Those categories with a larger number of samples will inevitably be more easily predicted by the model due to their greater proportion. Normally, a trained model should have an equal probability of recognizing each expression. Therefore, in this embodiment, the compensation weight for each expression sample data is determined by the target number of each expression sample data. Then, weights are added to each category in a preset loss function as compensation. Generally, categories with fewer samples are given higher weights, and categories with more samples are given lower weights, thereby significantly improving the model's accuracy on a few categories. Optionally, the first preset loss function can be the cross-entropy loss function.

[0059] S103, based on the second expression recognition model, perform knowledge distillation on the second expression representation model to obtain the third expression recognition model.

[0060] In practice, after obtaining the second expression recognition model, since the second expression recognition model is obtained through self-distillation and has a higher recognition accuracy than the first expression recognition model, the third expression recognition model is obtained by using the second expression recognition model as a teacher model to perform knowledge distillation on the second expression representation model. This not only has a simpler structure but also ensures a certain level of accuracy.

[0061] In some embodiments, knowledge distillation is performed on the first expression representation model based on the first expression recognition model, specifically including:

[0062] Determine the target sample data from the fine-tuned facial expression sample dataset;

[0063] The target sample data is input into the first expression recognition model and the first expression representation model respectively, and the first output result of the encoder of the first expression recognition model and the second output result of the encoder of the first expression representation model are extracted.

[0064] The loss between the first output and the second output is calculated using a second preset loss function, and the parameters of the first facial expression representation model are adjusted based on the loss result.

[0065] In practice, the target sample data can be any sample data from the fine-tuned facial expression sample dataset. Optionally, the second preset loss function can be the Logits regression loss function. Since the first output result of the encoder of the first facial expression recognition model is more accurate than the manual label, the accuracy of the second facial expression recognition model can be further improved.

[0066] In some embodiments, knowledge distillation is performed on the second expression representation model based on the second expression recognition model, specifically including:

[0067] Determine the target sample data from the fine-tuned facial expression sample dataset;

[0068] The target sample data is input into the second expression recognition model and the second expression representation model respectively, and the third output result of the encoder of the second expression recognition model and the fourth output result of the encoder of the second expression representation model are extracted.

[0069] The loss between the third and fourth output results is calculated using a second preset loss function, and the parameters of the second facial expression representation model are adjusted based on the loss result.

[0070] In practice, since the second expression recognition model is obtained through self-distillation, it can retain the experience learned during the knowledge distillation process. When the second expression recognition model is used as a teacher model to perform knowledge distillation on the second expression representation model again, the accuracy of the third expression recognition model can be further improved.

[0071] In some embodiments, the fine-tuning sample dataset used for fine-tuning the second expression representation model is the same as the fine-tuning sample dataset used for knowledge distillation of the first expression representation model.

[0072] In some embodiments, the unlabeled facial expression sample dataset for self-supervised pre-training of the first facial expression representation model is the same as the unlabeled facial expression sample dataset for self-supervised pre-training of the second facial expression representation model.

[0073] refer to Figure 2 This is a flowchart illustrating another expression recognition model training method according to an embodiment of this application. In this method, a first expression representation model and a second expression representation model are obtained through MAE pre-training on an unlabeled expression sample dataset. Then, the first expression representation model is fine-tuned directly using a fine-tuning sample dataset to obtain a first expression recognition model. Next, the first expression representation model is fine-tuned on a fine-tuning sample dataset with classification labels, and simultaneously, the first expression recognition model is introduced as a Teacher model. A second expression recognition model for expression classification is trained using knowledge distillation. Finally, the second expression representation model is fine-tuned on a fine-tuning sample dataset with classification labels, and simultaneously, the second expression recognition model is introduced as a Teacher model. A third expression recognition model for expression classification is trained using knowledge distillation. This third expression recognition model can serve as an expression recognition model supporting practical applications, achieving high classification accuracy while also meeting the throughput requirements for real-time processing under high concurrency.

[0074] The facial expression recognition model training method provided in this application obtains a first facial expression representation model and a second facial expression representation model through self-supervised pre-training. The first facial expression representation model has a larger number of parameters than the second facial expression representation model. The first facial expression representation model is fine-tuned to obtain a first facial expression recognition model. Knowledge distillation is then performed on the first facial expression representation model to obtain a second facial expression recognition model. Finally, knowledge distillation is performed on the second facial expression representation model to obtain a third facial expression recognition model. This method utilizes a large amount of unlabeled sample data through self-supervised pre-training, reducing the cost of manual annotation. Self-distillation on the same facial expression representation model improves the accuracy of the second facial expression recognition model. Furthermore, knowledge distillation on the second facial expression representation model further ensures the accuracy of the third facial expression recognition model. Since the second facial expression representation model has a smaller number of parameters, the third facial expression recognition model runs faster and has lower equipment requirements compared to the second facial expression recognition model. It can not only meet the requirements of a larger recognition throughput but also has a wider range of applications. Furthermore, the facial expression recognition model training method provided in this application, by combining MAE pre-training with self-distillation, can obtain a second facial expression recognition model with an accuracy of 69.80% on the ViT-Base backbone network, with 85.8M parameters and a throughput of 290 images / second. This accuracy is significantly higher than the current state-of-the-art level (66.29%). This model can be used in applications with lower real-time requirements but higher accuracy requirements. By combining MAE pre-training with knowledge distillation, the backbone network can be compressed from ViT-Base to ViT-Tiny, i.e., the third facial expression recognition model. The accuracy decreases from 69.80% to 68.51%, while the number of parameters decreases from 85.8M to 5.5M, and the throughput increases from 290 images / second to 2500 images / second. Model compression achieves a significant reduction in parameters and computational cost with only a small loss in accuracy, resulting in an 8.7-fold increase in throughput. The compressed model, while meeting the real-time and throughput requirements of online services, still significantly surpasses the industry's advanced level (66.29%) in accuracy.

[0075] refer to Figure 3 This is a flowchart illustrating an expression recognition method according to an embodiment of this application; the executing entity of this expression recognition method may be, but is not limited to, a server or a terminal device. The expression recognition method includes the following steps:

[0076] S104, the expression image to be recognized is input into the third expression recognition model obtained according to the expression recognition model training method described above, and the expression recognition result of the expression image to be recognized is obtained.

[0077] In specific implementation, after obtaining the third expression recognition model through the expression recognition model training method of any of the above embodiments, the third expression recognition model can be used to recognize the expression image to be recognized.

[0078] It should be noted that the expression recognition method described herein has the beneficial effects of the corresponding expression recognition model training method embodiments, which will not be elaborated here.

[0079] Exemplary device

[0080] Based on the same inventive concept, corresponding to the facial expression recognition model training method of any of the above embodiments, this application also provides an facial expression recognition model training device.

[0081] refer to Figure 4 The facial expression recognition model training device includes:

[0082] The pre-training module 201 obtains a first expression representation model and a second expression representation model through self-supervised pre-training; wherein the number of parameters of the baseline model of the first expression representation model is greater than the number of parameters of the baseline model of the second expression representation model.

[0083] The first fine-tuning module 202 fine-tunes the first expression representation model to obtain a first expression recognition model, and performs knowledge distillation on the first expression representation model based on the first expression recognition model to obtain a second expression recognition model.

[0084] The second fine-tuning module 203 performs knowledge distillation on the second expression representation model based on the second expression recognition model to obtain the third expression recognition model.

[0085] For ease of description, the above system is described by dividing it into various modules based on their functions. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0086] The system described above is used to implement the corresponding facial expression recognition model training method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0087] Based on the same inventive concept, corresponding to the facial expression recognition method of any of the above embodiments, this application also provides an facial expression recognition device.

[0088] refer to Figure 5 The facial expression recognition model training device includes:

[0089] The recognition module 301 inputs the expression image to be recognized into the third expression recognition model obtained according to the expression recognition model training method described above, and obtains the expression recognition result of the expression image to be recognized.

[0090] For ease of description, the above system is described by dividing it into various modules based on their functions. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0091] The system described above is used to implement the corresponding facial expression recognition method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0092] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the expression recognition model training or expression recognition method described in any of the above embodiments.

[0093] Figure 6 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0094] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0095] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0096] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0097] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0098] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0099] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0100] The electronic devices described above are used to implement the corresponding facial expression recognition model training or facial expression recognition method in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0101] Exemplary program product

[0102] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the facial expression recognition model training or facial expression recognition method as described in any of the above embodiments.

[0103] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0104] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the facial expression recognition model training or facial expression recognition method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0105] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0106] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0107] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0108] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for training an facial expression recognition model, characterized in that, include: A first facial expression representation model and a second facial expression representation model are obtained through self-supervised pre-training; wherein the number of parameters of the baseline model of the first facial expression representation model is greater than the number of parameters of the baseline model of the second facial expression representation model. The first expression representation model is fine-tuned to obtain the first expression recognition model, and the first expression recognition model is knowledge distilled based on the first expression recognition model to obtain the second expression recognition model. Based on the second expression recognition model, knowledge distillation is performed on the second expression representation model to obtain a third expression recognition model, which includes: Determine the target sample data from the fine-tuned facial expression sample dataset; The target sample data is input into the second expression recognition model and the second expression representation model respectively, and the third output result of the encoder of the second expression recognition model and the fourth output result of the encoder of the second expression representation model are extracted. The loss between the third and fourth output results is calculated using a second preset loss function, and the parameters of the second facial expression representation model are adjusted based on the loss result.

2. The method according to claim 1, characterized in that, The first expression recognition model is obtained by fine-tuning the first expression representation model, specifically including: A classification layer is added after the encoder of the first expression representation model, and the first expression representation model with the added classification layer is fine-tuned based on the fine-tuned expression sample dataset to obtain the first expression recognition model.

3. The method according to claim 2, characterized in that, The first facial expression representation model, after adding a classification layer, was fine-tuned based on a fine-tuned facial expression sample dataset. Specifically, this included: Obtain the target number of each expression sample data in the fine-tuned expression sample dataset; The compensation weight for each type of facial expression sample data is determined based on the target number. The loss of the output of the first facial expression representation model after adding a classification layer is calculated using a first preset loss function and based on the compensation weights.

4. The method according to claim 1, characterized in that, Knowledge distillation is performed on the first expression representation model based on the first expression recognition model, specifically including: Determine the target sample data from the fine-tuned facial expression sample dataset; The target sample data is input into the first expression recognition model and the first expression representation model respectively, and the first output result of the encoder of the first expression recognition model and the second output result of the encoder of the first expression representation model are extracted. The loss between the first output and the second output is calculated using a second preset loss function, and the parameters of the first facial expression representation model are adjusted based on the loss result.

5. The method according to claim 1, characterized in that, The fine-tuning sample dataset used for fine-tuning the second facial expression representation model is the same as the fine-tuning sample dataset used for knowledge distillation of the first facial expression representation model.

6. The method according to claim 1, characterized in that, The first and second facial expression representation models were obtained through self-supervised pre-training, specifically including: Using an unlabeled facial expression sample dataset and based on mask autoencoder technology, a first facial expression representation model and a second facial expression representation model were obtained, respectively.

7. A facial expression recognition method, characterized in that, include: The expression image to be recognized is input into the third expression recognition model obtained by the expression recognition model training method according to any one of claims 1-6, and the expression recognition result of the expression image to be recognized is obtained.

8. A facial expression recognition model training device, characterized in that, include: The pre-training module obtains a first facial expression representation model and a second facial expression representation model through self-supervised pre-training; wherein the number of parameters of the baseline model of the first facial expression representation model is greater than the number of parameters of the baseline model of the second facial expression representation model. The first fine-tuning module fine-tunes the first expression representation model to obtain a first expression recognition model, and performs knowledge distillation on the first expression representation model based on the first expression recognition model to obtain a second expression recognition model. The second fine-tuning module performs knowledge distillation on the second expression representation model based on the second expression recognition model to obtain a third expression recognition model, which includes: Determine the target sample data from the fine-tuned facial expression sample dataset; The target sample data is input into the second expression recognition model and the second expression representation model respectively, and the third output result of the encoder of the second expression recognition model and the fourth output result of the encoder of the second expression representation model are extracted. The loss between the third and fourth output results is calculated using a second preset loss function, and the parameters of the second facial expression representation model are adjusted based on the loss result.

9. An expression recognition device, characterized in that, include: The recognition module inputs the expression image to be recognized into the third expression recognition model obtained by the expression recognition model training method according to any one of claims 1-6, and obtains the expression recognition result of the expression image to be recognized.

10. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method according to any one of claims 1 to 7.

Images