A facial expression recognition method and system based on multi-cue association learning

Abstract

The present invention provides a facial expression recognition method and system using multi-cue association learning, belonging to the field of computer vision technology. The method comprises: inputting a pre-recognized facial image into a student model and / or a teacher model for facial expression recognition; a training method comprising: cropping a global face sample image into upper and lower half face sample images; extracting clue features; obtaining an adjacency matrix corresponding to the upper half face sample image, the lower half face sample image, and the global face sample image; employing a feature-level attention mechanism to fuse associated semantics to obtain a teacher model; employing cross-entropy loss to supervise the training of the teacher model; and employing label distillation, KL divergence, and cross-entropy loss to supervise the training of the student model. The present invention utilizes associated cues to enhance the learning ability of the model and is used to solve the problem of expression recognition in natural scenes.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and more specifically, relates to a facial expression recognition method and system based on multi-cue association learning. Background Art

[0002] Automatic facial expression recognition, which aims to infer a person's underlying emotional state from facial images based on the movement of facial muscles, is a challenging computer vision task. Psychologists point out that semantic information related to facial expressions is scattered around various facial features. The upper and lower halves of the face carry different amounts of semantic information, and different combinations of semantic information convey completely different emotional meanings. More importantly, this semantic information is sensitive and fragile, easily affected by lighting changes, partial occlusions, head posture, and even identity and makeup characteristics. Therefore, there is an urgent need to design excellent facial expression recognition models that comprehensively address all of these issues.

[0003] Graph convolutional neural networks (GCNNs) are neural networks that perform convolution operations on graph data structures. They are derived from convolutional neural networks in deep learning, but can be used in non-Euclidean domains, where convolutional neural networks struggle. GCNNs inherit most of the advantages of deep learning and demonstrate strong manifold representation capabilities for graph nodes, edges, and subgraphs.

[0004] Associative learning is a feasible and promising solution. It stems from psychologists' observations of the learning processes of animals and humans. The famous Pavlov experiment demonstrated that animals have the ability to associate different stimulus signals during learning. Associationism believes that human learning ability comes from the establishment of various connections. In pattern recognition tasks, humans have the ability to associate clues and labels, and also the ability to associate individuals with similar clues. Purposefully tapping into their potential for associative learning often leads to better learning outcomes. Inspired by associative learning, some existing facial expression recognition work uses graph neural networks to establish associations between different training samples through specified clues, enabling the model to learn latent semantic features that conform to these association rules, thereby improving the model's representational capabilities.

[0005] Existing work has overlooked key characteristics of human association learning, including: 1. Human association learning is often multi-cue, not based on a single cue to establish association rules; 2. Human association learning employs a specific attention mechanism, integrating semantic knowledge learned from different association rules based on learning experience. These issues have led to limitations in the application of existing methods, resulting in low recognition accuracy. Leveraging the advantages of association learning and graph social networks can help address these issues, but there are currently no publicly available methods for further optimizing association learning and graph neural networks for multi-cue facial expression recognition. Summary of the Invention

[0006] In view of the defects of the prior art, the purpose of the present invention is to provide a facial expression recognition method and system based on multi-cue association learning, aiming to solve the problems of limitations and low recognition accuracy of existing facial expression recognition methods.

[0007] To achieve the above object, the present invention provides a facial expression recognition method based on multi-cue association learning, comprising the following steps:

[0008] Inputting the pre-recognized facial image into the student model and / or the teacher model to recognize facial expressions;

[0009] The training method of the student model and the teacher model includes the following steps:

[0010] D1: Crop the global face sample image horizontally into the upper half face sample image and the lower half face sample image;

[0011] D2: Extract clue features of the global face sample image and the upper and lower half face sample images;

[0012] D3: Calculate the correlation between the corresponding clue features of the upper half face sample image, the lower half face sample image, and the global face sample image, and obtain the corresponding adjacency matrix;

[0013] D4: Taking clue features and adjacency matrix as input, three graph convolutional neural networks are used to output related semantics, and the feature-level attention mechanism is used to fuse the related semantics to obtain the teacher model;

[0014] D5: The fused associated semantics in the teacher model are input into the classification layer, and the teacher model is trained using cross-entropy loss. At the same time, label distillation is used, and KL divergence and cross-entropy loss are used to supervise the student model training.

[0015] Among them, the student model is constructed by the fully connected layer in which the clue features are input after passing through the bottleneck layer.

[0016] Further preferably, the teacher model is constructed in two ways;

[0017] Among them, the first method is:

[0018] The clue features of the global face sample image are used as the first input of the three graph convolutional neural networks. The adjacency matrices corresponding to the global face sample image, the upper half face sample image, and the lower half face sample image are used as the second input of the first, second, and third graph convolutional neural networks respectively. The global correlation semantics are output and the feature-level attention mechanism is used to fuse the global correlation semantics to obtain the teacher model.

[0019] The second method is: the clue features corresponding to the global face sample image, the upper half face sample image and the lower half face sample image are used as the first input of the first, second and third graph convolutional neural networks respectively, and the adjacency matrices corresponding to the global face sample image, the upper half face sample image and the lower half face sample image are used as the second input of the first, second and third graph convolutional neural networks respectively, and the global association semantics, the upper half face sample association semantics and the lower half face sample association semantics are output respectively, and the feature-level attention mechanism is used to fuse the global association semantics, the upper half face sample association semantics and the lower half face sample association semantics to obtain the teacher model.

[0020] Further preferably, the method for extracting clue features is: using the LBP operator for feature extraction; or using a public and trained face model to directly extract deep features; or using labeled samples to fine-tune a pre-trained face model to obtain a deep model, and then using the deep model to extract deep embedded features.

[0021] Further preferably, the cross entropy loss function of the teacher model is:

[0022]

[0023] in, Represents the associated semantics F(x) after fusion in the teacher model i ) Sample label distribution predicted by the fully connected layer; y i represents the true distribution of the sample.

[0024] Further preferably, the total loss function of the student model, including the distillation loss function and the student model cross entropy loss function, is specifically:

[0025]

[0026]

[0027] in, represents the cross entropy loss of the student model, represents the probability distribution predicted by the student model; is the distillation loss; f(·) represents the softmax activation function, and T represents the distillation temperature; Represents the associated semantics F(x) after fusion in the teacher model i ) Sample label distribution predicted by the fully connected layer.

[0028] In another aspect, the present invention provides a facial expression recognition system using multi-cue association learning, comprising:

[0029] The student model expression recognition module is used to input the pre-recognized facial image into the student model storage module to recognize the facial expression;

[0030] The teacher model expression recognition module is used to input the pre-recognized facial image into the teacher model storage module to recognize the facial expression;

[0031] A global face sample image preprocessing module is used to crop the collected global face sample image into an upper half face sample image and a lower half face sample image along the horizontal direction;

[0032] A feature extraction module is used to extract clue features of the global face sample image and the upper half face and lower half face sample images;

[0033] An adjacency matrix acquisition module is used to calculate the correlation between the corresponding clue features of the upper half face sample image, the lower half face sample image and the global face sample image, and obtain the adjacency matrix corresponding to the upper half face sample image, the lower half face sample image and the global face sample image;

[0034] The teacher model construction module is used to take clue features and adjacency matrix as input, use three graph convolutional neural networks to output related semantics, and use feature-level attention mechanism to fuse the related semantics to obtain the teacher model;

[0035] The model training module is used to input the fused semantic associations in the teacher model into the classification layer and supervise the training of the teacher model using the teacher model's cross-entropy loss function. At the same time, label distillation is used to supervise the training of the student model using KL divergence and the student model's cross-entropy loss.

[0036] Among them, the student model is constructed by the fully connected layer in which the clue features are input after passing through the bottleneck layer.

[0037] Further preferably, the teacher model is constructed in two ways;

[0038] Among them, the first method is:

[0039] The clue features of the global face sample image are used as the first input of the three graph convolutional neural networks. The adjacency matrices corresponding to the global face sample image, the upper half face sample image, and the lower half face sample image are used as the second input of the first, second, and third graph convolutional neural networks respectively. The global correlation semantics are output and the feature-level attention mechanism is used to fuse the global correlation semantics to obtain the teacher model.

[0040] The second method is: the clue features corresponding to the global face sample image, the upper half face sample image and the lower half face sample image are used as the first input of the first, second and third graph convolutional neural networks respectively, and the adjacency matrices corresponding to the global face sample image, the upper half face sample image and the lower half face sample image are used as the second input of the first, second and third graph convolutional neural networks respectively, and the global association semantics, the upper half face sample association semantics and the lower half face sample association semantics are output respectively, and the feature-level attention mechanism is used to fuse the global association semantics, the upper half face sample association semantics and the lower half face sample association semantics to obtain the teacher model.

[0041] Further preferably, the cross entropy loss function of the teacher model is:

[0042]

[0043] in, Represents the associated semantics F(x) after fusion in the teacher model i ) Sample label distribution predicted by the fully connected layer; y i represents the true distribution of the sample.

[0044] Further preferably, the total loss function of the student model includes the distillation loss function and the cross entropy loss function of the student model, specifically:

[0045]

[0046]

[0047] in, represents the cross entropy loss of the student model, represents the probability distribution predicted by the student model; is the distillation loss; f(·) represents the softmax activation function, and T represents the distillation temperature; Represents the associated semantics F(x) after fusion in the teacher model i ) Sample label distribution predicted by the fully connected layer.

[0048] In general, the above technical solutions conceived by the present invention have the following advantages compared with the prior art:

[0049] Beneficial effects:

[0050] The present invention provides a facial expression recognition method and system based on multi-cue association learning, which divides facial expressions into the upper and lower halves of the face to guide association learning based on local cues. While effectively dealing with the problem of local occlusion, it better utilizes different association cues to enhance the learning ability of the model. A multi-cue association learning method based on graph neural network is proposed to solve the problem of expression recognition in natural scenes.

[0051] The present invention provides a facial expression recognition method and system based on multi-cue association learning, in which a feature-level attention mechanism effectively integrates the knowledge of multi-cue association learning, making the model more consistent with the human association learning mechanism, thereby better coping with the complex and changing challenges in natural scenes.

[0052] The present invention provides a facial expression recognition method and system based on multi-cue association learning, in which knowledge distillation is used to further optimize the student model and teacher model. On the one hand, knowledge distillation can regard association learning as the regularization peak of traditional convolutional neural networks, reducing the computing resources required by the model during the testing phase; on the other hand, the collaborative training of the teacher model and the student model can further optimize the backbone network of the model, thereby further improving the performance of the teacher model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of a facial expression recognition method using multi-cue association learning provided by an embodiment of the present invention;

[0054] Figure 2 is a schematic diagram of facial expression data preprocessed according to an embodiment of the present invention;

[0055] Figure 3 It is a structural diagram of the teacher model and the student model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] like Figure 1 As shown, on the one hand, an embodiment of the present invention provides a multi-cue associated facial expression recognition method, comprising the steps of:

[0058] S101: Global face sample image preprocessing

[0059] Since the semantic information carried by the upper and lower halves of a person's face has its own association characteristics (different expressions overlap in AUs (Action Units) on the upper half of the face, especially AU1, AU4, and AU5, while they are mostly mutually exclusive on the lower half of the face), and the occlusion of the upper and lower halves of the face has different application scenarios (for example, occluding the upper half of the face can be used for emotion recognition when wearing VR glasses, while occluding the lower half of the face can be used for emotion recognition when wearing a mask), this invention uses the upper half of the face, the lower half of the face, and the whole face as association cues to guide multi-cue association learning;

[0060] Furthermore, before the global face sample image is input into the next step for feature extraction, the sample is preprocessed; that is, Figure 2 As shown, the global face sample image is evenly cropped into two parts along the horizontal direction, the upper part is the upper half of the face, and the lower part is the lower half of the face. The original image is the global face;

[0061] S102: Feature extraction, used to extract the global face of each sample in the global face sample image set and three clues of the upper and lower halves of the face;

[0062] Furthermore, the feature extraction of the sample preferably adopts one of the following three methods;

[0063] a. Feature extraction method using LBP operator

[0064] LBP is an operator used to describe the local texture features of an image. It has significant advantages such as rotation invariance and grayscale invariance. The calculation of the LBP operator is as follows: in a 3*3 window, the grayscale value of the center pixel of the window is used as the threshold, and the grayscale values ​​of the eight surrounding pixels are compared with it. If it is greater than the grayscale value of the center pixel, it is marked as 1, otherwise it is 0. In this way, after comparison, the 8 pixels in the 3*3 neighborhood can generate an 8-bit binary number; it is usually converted into a decimal number, namely the LBP code, which has a total of 256 types; finally, the LBP value of the center pixel of the window is obtained, and this value is used to reflect the texture information of the area; it can be expressed in the formula:

[0065]

[0066] Where I(p) represents the grayscale value of the pth pixel in the window except the center pixel, I(c) represents the grayscale value of the center pixel, and s(·) is the threshold function, which is expressed as follows:

[0067]

[0068] Since LBP records the difference between the neighborhood pixels and the central pixel, when the illumination changes and the grayscale values ​​of the pixels in the window increase or decrease synchronously, the LBP value does not change significantly, so LBP is not sensitive to illumination changes.

[0069] b. A deep feature extraction method that directly extracts global face sample image features using a public and trained face model;

[0070] In this feature extraction method of this embodiment, the ResNet18 network model is used to read the network parameters of the pre-trained face model Ms-Celeb-1M, all global face sample images are directly input into the ResNet18 network, and the 512-dimensional vector before the last fully connected layer in the ResNet18 network is output as the deep feature of the data sample in this embodiment;

[0071] c. Fine-tune the pre-trained face model Ms-Celeb-1M using labeled samples to obtain a deep model. Finally, use the obtained deep model to extract deep embedding features of all global face sample images;

[0072] In this feature extraction method of this embodiment, the ResNet18 network model is also used to read the network parameters of the pre-trained face model Ms-Celeb-1M. All labeled samples are used as the training set data of the ResNet18 model, and the parameters of the network in the face model are fine-tuned to obtain the fine-tuned deep model of the data samples used in this embodiment. Then, all data samples are directly input into the fine-tuned deep model of this feature extraction method, and the 512-dimensional vector before the last fully connected layer in the entire network is output as the deep embedding feature of the current data sample.

[0073] The features obtained by the above three feature extraction methods can all be used as the feature vectors of the samples in this embodiment;

[0074] S103: association learning, used to extract association semantics based on the upper half of the face, association semantics based on the lower half of the face, and association semantics based on the entire face;

[0075] In association learning, the association between samples can be To represent; each sample As a node on the graph, the edge a between any two samples ij ∈ε represents the correlation between samples; a ij The larger the weight is, the greater the correlation between samples is; if a ij If the weight is 0, it is considered that there is no correlation between the two samples;

[0076] like Figure 3 As shown in the graph convolutional neural network, the teacher model is based on the semantic features provided by the nodes and the adjacency matrix representing the relationship between the nodes. To train the network parameters W, when the graph convolutional neural network is trained in batches, n represents the size of the mini-batch; each element in the adjacency matrix can be represented by a ij To express; given a clue c, the feature corresponding to the i-th sample is expressed as The calculation formula of the adjacency matrix is ​​expressed as follows:

[0077]

[0078] Among them, κ is an empirical parameter used to control the number of sample pairs with associated relationships; when there are different associated clues, multiple adjacency matrices can be obtained; the present invention uses three clues, the upper half of the face, the lower half of the face, and the whole face, to construct the adjacency matrix; x i and x j represents any two samples in a training batch, Represents the clue features of the global face extracted by the backbone network; and Respectively represent the clue features of the upper and lower half of the face extracted by the backbone network;

[0079] Based on the adjacency matrix constructed according to different clues, an independent graph convolutional neural network can be used to learn the associated knowledge guided by it. Therefore, the present invention configures three GCN networks. Among them, there are two inputs in the graph convolutional neural network. Each network contains two graph convolution layers, one input is the clue feature, and the other input is the adjacency matrix. The calculation formula of the adjacency matrix is ​​expressed as follows:

[0080]

[0081] in, represents the normalized adjacency matrix, which is calculated as represents the identity matrix; W is the trainable parameter of the graph convolutional network; σ((x)=Relu(x)=max(0,x); H (h) Represents the input of the h-th layer of convolutional neural network; H (1) Sample clue features provided to the bottleneck layer of the convolutional neural network backbone;

[0082] Furthermore, the distance between a sample and other samples is preferably calculated using the Euclidean distance; however, other distance calculation formulas that represent differences may also be used to achieve this;

[0083] H (h) There are two construction methods, specifically:

[0084] In one embodiment, clue features corresponding to the global face sample image are input into the three graph convolutional neural networks, namely

[0085]

[0086] in, Represents the features of the global face extracted by the backbone network; based on this method, the three graph convolutional neural networks also output global correlation semantics;

[0087] In another embodiment, three graph convolutional neural networks are respectively input with clue features corresponding to the construction of the adjacency matrix, namely:

[0088]

[0089] and Represent the clue features extracted from the upper and lower half of the face respectively; in this method, the three graph convolutional neural networks output a global association semantics and an upper and lower half face sample association semantics respectively;

[0090] In the above two embodiments, the adjacency matrix used by the corresponding GCN is the same, that is, no matter how the c-th GCN generates H(h), its adjacency matrix is ​​Ac;

[0091] Regardless of the above implementation methods, the attention mechanism introduced in the next step is used for feature fusion, so that the associated knowledge guided by different clues is integrated together;

[0092] S104: Associative semantic fusion, used to fuse the associated semantics learned from the three cues to complete the construction of the teacher model;

[0093] This paper uses the feature-level attention mechanism to analyze the features output by different graph neural networks. Perform adaptive fusion to simulate human attention learning mechanism;

[0094] Specifically, the attention mechanism can be represented by a trainable fully connected layer that Mapped to an attention weight α c , all features are aggregated according to the attention weights to obtain the final sample features:

[0095]

[0096] Among them, α c =G(F(x i ) Tq), G(·) is the sigmoid function, and q is the trainable network parameter of the fully connected layer. The fused associated semantics can be input into the classification layer and supervised training is performed using cross entropy loss:

[0097]

[0098] in, Represents the feature F(x i ) Sample label distribution predicted by the fully connected layer; y i Represents the true distribution of samples; through the attention mechanism, the knowledge obtained from different related clues is organically integrated, making the model more capable of responding to different challenges. For example, when the upper half of the face is occluded or highly confused, the clues based on the lower half of the face will receive a greater fusion weight through the attention mechanism, thus playing a major role in the recognition process, and vice versa;

[0099] S105: Use knowledge distillation to further optimize the model;

[0100] In the framework of the present invention, Figure 3 As shown in the figure, the two branches derived from the backbone network correspond to the teacher model and the student model respectively; the teacher model is a multi-cue guided associative learning, while the student model is a traditional unrelated learning; knowledge distillation aims to transfer the knowledge of the teacher model to the student model; the student model is the fully connected layer connected after the bottleneck layer of the convolutional neural network used in the backbone network;

[0101] Specifically, the present invention adopts label distillation and uses KL divergence and cross entropy loss to supervise the training of the student model:

[0102]

[0103]

[0104] in, represents the cross entropy loss of the student model, represents the probability distribution predicted by the student model; is the distillation loss; f(·) represents the softmax activation function, and T represents the distillation temperature;

[0105] It should be emphasized that although the student model only uses global features as a clue and does not perform associative learning, its performance is inferior to that of the teacher model; however, as two branches of the backbone network, the two have a synergistic effect on the optimization of the backbone network even without the addition of distillation loss. Therefore, the introduction of the student model can optimize the backbone network and thus improve the performance of the teacher model; and with the addition of distillation loss, the teacher model can further promote the performance of the student model, thus achieving a mutually reinforcing effect.

[0106] In summary, the present invention needs to optimize the teacher model and the student model simultaneously during the training process, so the total loss of the model is Specifically, it can be expressed as:

[0107]

[0108] In the testing phase, different branches can be selected for prediction according to the actual situation. For environments with limited computing resources, the teacher model can be removed and the prediction results of the student model can be used alone. For applications that require extreme performance, the prediction results of the teacher model can be used for expression classification decisions. The present invention will compare the differences in prediction performance of the two branches in detail in the experimental part.

[0109] In one embodiment, the RAF-DB (Real-world Affective Faces Database) expression database was used. The database contains 29,672 facial images collected from the Internet, and the expressions were annotated by 315 staff members (university students and faculty members). It contains a total of 7 expressions: anger, disgust, fear, happiness, sadness, surprise, and neutral.

[0110] The present invention selected all facial expression images and conducted experiments using training and test sets divided according to the expression library. Using a method that inputted global features into three graph neural networks, the accuracy of expression recognition reached 88.43%. Using a method that inputted global features and features of the upper and lower halves of the face into three separate graph neural networks, the accuracy reached 89.62%. Further optimizing the model using knowledge distillation yielded the best expression recognition accuracy of 90.66%.

[0111] In another embodiment, the FER+ (Hard-Label) expression database is used; this expression database is an extension of the original FER dataset, in which facial expression images are re-labeled into one of eight emotion types: neutral, happy, surprised, sad, angry, disgusted, fearful, and contemptuous;

[0112] The present invention selected all facial expression images in the dataset for training; when global features were input into three graph neural networks, the expression recognition accuracy obtained was 87.68%; when global features and features of the upper and lower halves of the face were input into three graph neural networks respectively, the expression recognition accuracy obtained was 88.48%; and after further optimizing the model using the knowledge distillation method, the best expression recognition accuracy was 89.48%.

[0113] In another aspect, the present invention provides a facial expression recognition system using multi-cue association learning, comprising:

[0114] The student model expression recognition module is used to input the pre-recognized facial image into the student model storage module to recognize the facial expression;

[0115] The teacher model expression recognition module is used to input the pre-recognized facial image into the teacher model storage module to recognize the facial expression;

[0116] A global face sample image preprocessing module is used to crop the collected global face sample image into an upper half face sample image and a lower half face sample image along the horizontal direction;

[0117] A feature extraction module is used to extract clue features of the global face sample image and the upper half face and lower half face sample images;

[0118] An adjacency matrix acquisition module is used to calculate the correlation between the corresponding clue features of the upper half face sample image, the lower half face sample image and the global face sample image, and obtain the adjacency matrix corresponding to the upper half face sample image, the lower half face sample image and the global face sample image;

[0119] The teacher model construction module is used to take clue features and adjacency matrix as input, use three graph convolutional neural networks to output related semantics, and use feature-level attention mechanism to fuse the related semantics to obtain the teacher model;

[0120] The model training module is used to input the fused semantic associations in the teacher model into the classification layer, and use the cross-entropy loss function of the teacher model to supervise the training of the teacher model; at the same time, label distillation is adopted, and the KL divergence and the cross-entropy loss of the student model are used to supervise the training of the student model.

[0121] Further preferably, the teacher model is constructed in two ways;

[0122] Among them, the first method is:

[0123] The clue features of the global face sample image are used as the first input of the three graph convolutional neural networks. The adjacency matrices corresponding to the global face sample image, the upper half face sample image, and the lower half face sample image are used as the second input of the first, second, and third graph convolutional neural networks respectively. The global correlation semantics are output and the feature-level attention mechanism is used to fuse the global correlation semantics to obtain the teacher model.

[0124] The second method is: the clue features corresponding to the global face sample image, the upper half face sample image and the lower half face sample image are used as the first input of the first, second and third graph convolutional neural networks respectively, and the adjacency matrices corresponding to the global face sample image, the upper half face sample image and the lower half face sample image are used as the second input of the first, second and third graph convolutional neural networks respectively, and the global association semantics, the upper half face sample association semantics and the lower half face sample association semantics are output respectively, and the feature-level attention mechanism is used to fuse the global association semantics, the upper half face sample association semantics and the lower half face sample association semantics to obtain the teacher model.

[0125] Further preferably, the cross entropy loss function of the teacher model is:

[0126]

[0127] in, Represents the associated semantics F(x) after fusion in the teacher model i ) Sample label distribution predicted by the fully connected layer; y i represents the true distribution of the sample.

[0128] Further preferably, the total loss function of the student model includes the distillation loss function and the cross entropy loss function of the student model, specifically:

[0129]

[0130]

[0131] in, represents the cross entropy loss of the student model, represents the probability distribution predicted by the student model; is the distillation loss; f(·) represents the softmax activation function, and T represents the distillation temperature; Represents the associated semantics F(x) after fusion in the teacher model i ) Sample label distribution predicted by the fully connected layer.

[0132] More specifically, knowledge distillation includes:

[0133] The two branches derived from the backbone network correspond to the teacher model and the student model respectively; the teacher model is a multi-cue guided associative learning, while the student model is a traditional unrelated learning; knowledge distillation aims to transfer the knowledge of the teacher model to the student model;

[0134] Further preferably, the teacher model and the student model need to be optimized simultaneously during the training process; in the testing phase, different branches can be selected for prediction results according to actual conditions; for environments with limited computing resources, the teacher model can be removed and the prediction results of the student model can be used alone; for applications that pursue extreme performance, the prediction results of the teacher model can be selected for expression classification decisions.

[0135] The implementation principle and technical effects of the system are similar to the above method and will not be repeated here.

[0136] The present invention also provides a storage medium storing a computer program, which is executed by a processor to implement the technical solution of any of the aforementioned facial expression recognition methods based on expression intensity changes. The implementation principles and technical effects are similar to those of the aforementioned methods and will not be further elaborated here.

[0137] It must be noted that in any of the above embodiments, the method is not necessarily executed in sequence according to the sequence number. As long as it cannot be inferred from the execution logic that it must be executed in a certain order, it means that it can be executed in any other possible order.

[0138] In summary, the present invention has the following advantages compared with the prior art:

[0139] The present invention provides a facial expression recognition method and system based on multi-cue association learning, which divides facial expressions into the upper and lower halves of the face to guide association learning based on local cues. While effectively dealing with the problem of local occlusion, it better utilizes different association cues to enhance the learning ability of the model. A multi-cue association learning method based on graph neural network is proposed to solve the problem of expression recognition in natural scenes.

[0140] The present invention provides a facial expression recognition method and system based on multi-cue association learning, in which a feature-level attention mechanism effectively integrates the knowledge of multi-cue association learning, making the model more consistent with the human association learning mechanism, thereby better coping with the complex and changing challenges in natural scenes.

[0141] The present invention provides a facial expression recognition method and system based on multi-cue association learning, in which knowledge distillation is used to further optimize the student model and teacher model. On the one hand, knowledge distillation can regard association learning as the regularization peak of traditional convolutional neural networks, reducing the computing resources required by the model during the testing phase; on the other hand, the collaborative training of the teacher model and the student model can further optimize the backbone network of the model, thereby further improving the performance of the teacher model.

[0142] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A facial expression recognition method based on multi-cue association learning, characterized in that: The following steps are involved: Inputting the pre-recognized facial image into the student model and / or the teacher model to recognize facial expressions; The training method of the student model and the teacher model includes the following steps: D1: Crop the global face sample image horizontally into the upper half face sample image and the lower half face sample image; D2: Extract clue features of the global face sample image and the upper and lower half face sample images; D3: Calculate the correlation between the corresponding clue features of the upper half face sample image, the lower half face sample image, and the global face sample image, and obtain the corresponding adjacency matrix; D4: Taking clue features and adjacency matrix as input, three graph convolutional neural networks are used to output related semantics, and the feature-level attention mechanism is used to fuse the related semantics to obtain the teacher model; D5: The fused associated semantics in the teacher model are input into the classification layer, and the teacher model is trained using cross-entropy loss. Label distillation is also used to supervise the student model training using KL divergence and cross-entropy loss. The student model is constructed by inputting clue features into the fully connected layer after the bottleneck layer. The methods for extracting clue features are: using the LBP operator for feature extraction; or using a public and trained face model to directly extract deep features; or using labeled samples to fine-tune the pre-trained face model to obtain a deep model, and then using the deep model to extract deep embedded features.

2. The facial expression recognition method according to claim 1, wherein The teacher model is constructed in two ways: Among them, the first method is: The clue features of the global face sample image are used as the first input of the three graph convolutional neural networks. The adjacency matrices corresponding to the global face sample image, the upper half face sample image, and the lower half face sample image are used as the second input of the first, second, and third graph convolutional neural networks respectively. The global correlation semantics are output and the feature-level attention mechanism is used to fuse the global correlation semantics to obtain the teacher model. The second method is: the clue features corresponding to the global face sample image, the upper half face sample image and the lower half face sample image are used as the first input of the first, second and third graph convolutional neural networks respectively, and the adjacency matrices corresponding to the global face sample image, the upper half face sample image and the lower half face sample image are used as the second input of the first, second and third graph convolutional neural networks respectively, and the global association semantics, the upper half face sample association semantics and the lower half face sample association semantics are output respectively, and the feature-level attention mechanism is used to fuse the global association semantics, the upper half face sample association semantics and the lower half face sample association semantics to obtain the teacher model.

3. The facial expression recognition method according to claim 1 or 2, wherein: The cross entropy loss function of the teacher model is: in, Represents the associated semantics after fusion in the teacher model Sample label distribution predicted by the fully connected layer; represents the true distribution of the sample.

4. The facial expression recognition method according to claim 3, wherein The total loss function of the student model includes the distillation loss function and the cross entropy loss function of the student model, specifically: in, represents the cross entropy loss of the student model, represents the probability distribution predicted by the student model; is the distillation loss; represents the softmax activation function, Indicates distillation temperature; Represents the associated semantics after fusion in the teacher model Sample label distribution predicted by the fully connected layer.

5. A facial expression recognition system based on multi-cue association learning, characterized in that: include: The student model expression recognition module is used to input the pre-recognized facial image into the student model storage module to recognize the facial expression; The teacher model expression recognition module is used to input the pre-recognized facial image into the teacher model storage module to recognize the facial expression; A global face sample image preprocessing module is used to crop the collected global face sample image into an upper half face sample image and a lower half face sample image along the horizontal direction; A feature extraction module is used to extract clue features of the global face sample image and the upper half face and lower half face sample images; An adjacency matrix acquisition module is used to calculate the correlation between the corresponding clue features of the upper half face sample image, the lower half face sample image and the global face sample image, and obtain the adjacency matrix corresponding to the upper half face sample image, the lower half face sample image and the global face sample image; The teacher model construction module is used to take clue features and adjacency matrix as input, use three graph convolutional neural networks to output related semantics, and use feature-level attention mechanism to fuse the related semantics to obtain the teacher model; The model training module is used to input the fused semantic associations in the teacher model into the classification layer and supervise the training of the teacher model using the teacher model's cross-entropy loss function. At the same time, label distillation is used to supervise the training of the student model using KL divergence and the student model's cross-entropy loss. Among them, the student model is constructed by the fully connected layer after the clue features pass through the bottleneck layer; The methods for extracting clue features are: using LBP operator for feature extraction; or using a public and trained face model to directly extract deep features; Alternatively, the pre-trained face model can be fine-tuned using labeled samples to obtain a deep model, and then the deep model can be used to extract deep embedding features.

6. The facial expression recognition system according to claim 5, characterized in that There are two ways to construct the teacher model; Among them, the first method is: The clue features of the global face sample image are used as the first input of the three graph convolutional neural networks. The adjacency matrices corresponding to the global face sample image, the upper half face sample image, and the lower half face sample image are used as the second input of the first, second, and third graph convolutional neural networks respectively. The global correlation semantics are output and the feature-level attention mechanism is used to fuse the global correlation semantics to obtain the teacher model. The second method is: the clue features corresponding to the global face sample image, the upper half face sample image and the lower half face sample image are used as the first input of the first, second and third graph convolutional neural networks respectively, and the adjacency matrices corresponding to the global face sample image, the upper half face sample image and the lower half face sample image are used as the second input of the first, second and third graph convolutional neural networks respectively, and the global association semantics, the upper half face sample association semantics and the lower half face sample association semantics are output respectively, and the feature-level attention mechanism is used to fuse the global association semantics, the upper half face sample association semantics and the lower half face sample association semantics to obtain the teacher model.

7. The facial expression recognition system according to claim 5 or 6, characterized in that: The cross entropy loss function of the teacher model is: in, Represents the associated semantics after fusion in the teacher model Sample label distribution predicted by the fully connected layer; represents the true distribution of the sample.

8. The facial expression recognition system according to claim 7, wherein: The total loss function of the student model includes the distillation loss function and the cross entropy loss function of the student model, specifically: in, represents the cross entropy loss of the student model, represents the probability distribution predicted by the student model; is the distillation loss; represents the softmax activation function, Indicates distillation temperature; Represents the associated semantics after fusion in the teacher model Sample label distribution predicted by the fully connected layer.

Images