Micro-expression recognition method and system based on neural network model and stacking method

Abstract

The invention discloses a micro-expression recognition method and system based on neural network models and a stacking method, and the method comprises the steps: carrying out the iterative training of five neural network models into five training models, respectively inputting an expansion data set into the five training models, and obtaining stable training features; the five training models are stacked into a meta-model by using a stacking method, the meta-model is further trained into a comprehensive meta-model by using the stable training features, and the comprehensive meta-model effectively integrates the advantages of each neural network model, can cover most expression data, and improves the expression efficiency. By adding the self-defined layer or model for fine tuning the model, the expression recognition method can be better suitable for expression recognition tasks in different scenes, the expression recognition capacity for coping with complex conditions is improved, and the expression recognition accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of expression recognition, and in particular to a micro-expression recognition method and system based on a neural network model and a stacking method. Background Art

[0002] In the era of artificial intelligence, facial expression recognition is a research hotspot. Expression recognition is the prerequisite for computers to understand human emotions. How to realize the anthropomorphism of computers so that they can adaptively provide a friendly operating environment for communication objects based on the surrounding environment and the state of the object has become the goal of the next generation of human-computer interaction development.

[0003] There are many types of facial micro-expressions, and they are difficult to recognize. Algorithms such as convolutional neural networks, deep belief networks, and multi-layer perceptrons are widely used in the field of facial expression recognition. There are many types of convolutional neural networks, and different convolutional neural networks are suitable for different types of expression recognition tasks. However, in complex situations, when there are many types of facial micro-expressions, there is a problem of inaccurate recognition results. Summary of the invention

[0004] The present invention provides a micro-expression recognition method and system based on a neural network model and a stacking method to overcome the above technical problems.

[0005] In order to achieve the above object, the technical solution of the present invention is:

[0006] A micro-expression recognition method based on a neural network model and a stacking method comprises the following steps:

[0007] S1: inputting a facial expression data set, and performing normalization processing on the facial expression data set;

[0008] S2: Divide the facial expression dataset into a training set and a test set;

[0009] S3: inputting the training set into five neural network models for independent training respectively, to obtain five corresponding training models;

[0010] S4: expanding the training set to form an expanded data set;

[0011] S5: Substituting the expanded data set into five corresponding training models for training respectively, and obtaining corresponding training features;

[0012] S6: Determine whether the training feature is stable. If not, use the expansion processing method to expand the expansion data set multiple times, and iterate the five training models multiple times until the stable training feature is output;

[0013] S7: Train the five training models to obtain a meta-model using a stacking method;

[0014] S8: averaging the stable training features and inputting them into the meta-model for training to obtain a comprehensive meta-model;

[0015] S9: Input the picture set to be identified into the comprehensive meta-model, and output the facial expressions after the selection.

[0016] Furthermore, the five basic models are simple_CNN, simpler_CNN, tiny_XCEPTION, mini_XCEPTION and big_XCEPTION convolutional neural network models.

[0017] Furthermore, in S3, the method of inputting the training set into simple_CNN, simpler_CNN, tiny_XCEPTION, mini_XCEPTION and big_XCEPTION for independent training is:

[0018] S3.1: performing face recognition on the face features in the training set, and selecting the recognized face image with a candidate frame;

[0019] S3.2: The face images selected by the candidate boxes are input into simple_CNN, simpler_CNN, tiny_XCEPTION, mini_XCEPTION and big_XCEPTION respectively for independent training to obtain five corresponding training models.

[0020] Furthermore, in S3.1, the method of performing face recognition on the face features in the training set and selecting the recognized face image with a candidate frame is:

[0021] S3.1.1: Input the test set, and cyclically scale the images in the test set, scaling each time according to a set scaling factor until the images in the test set are scaled to a specified size, and save the images of the same image in the test set after each scaling process together to form an image pyramid;

[0022] S3.1.2: Input the image pyramid into the full convolution network P-Net, output a 1x1x32 feature map, and use the feature map as detection images for three detection branches, respectively. The three detection branches are face classification, bounding box regression, and face feature point positioning. The full convolution network P-Net outputs two arrays, cls_cls_map and reg. The cls_cls_map represents the probability of a face or a non-face, and the reg predicts the position difference of the face position relative to the input image.

[0023] S3.1.3: Set a loss threshold, define loss functions and weights for the three detection branches respectively, calculate the loss values ​​of the three detection branches respectively and perform weighted summation on the three loss values, calculate a total loss value, compare the total loss value with the loss threshold, and when the total loss value is greater than the loss threshold, adjust the weights of the convolutional layer and the fully connected layer, and optimize the total loss value until the total loss value is less than the loss threshold;

[0024] S3.1.4: Set a screening threshold. When the probability of a face in cls_cls_map at a certain position exceeds the screening threshold, the position is determined to contain a face. The position containing the face is combined with the position information in reg to obtain the coordinates of the potential face candidate frame.

[0025] S3.1.5: Use the NMS algorithm to remove the duplicate potential face candidate frames, select the position with the highest face probability from the cls_cls_map, and use the reg information of the position with the highest face probability and the pixel coordinates of the original image to obtain the final candidate frame coordinates (x1, y1, x2, y2), where x1, y1 represents the coordinates of the upper left corner of the candidate frame, and x2, y2 represents the coordinates of the lower right corner of the candidate frame;

[0026] S3.1.6: Output the coordinates of the candidate box.

[0027] Furthermore, in S4, the expansion processing method includes but is not limited to changing grayscale, local occlusion, image mirroring and rotation.

[0028] Furthermore, in S7, the method of training the five training models to obtain a meta-model by using a stacking method comprises the following steps:

[0029] S7.1: retain the weight of a base model among the five training models, where the base model is used to learn common image features, including but not limited to edges and textures of the images in the training set;

[0030] S7.2: Add convolutional layers and fully connected layers to the five training models to adjust the weights of expression features according to the target training task;

[0031] S7.3: Randomly stack the five training models and the convolutional layer and the fully connected layer together to form a meta-model;

[0032] S7.4: A loss function, an optimizer and an evaluation index are defined in the metamodel, wherein the loss function is used to measure the difference between the output of the metamodel and the true value, the optimizer is used to minimize the loss function, and the evaluation index is used to measure the performance of the metamodel.

[0033] Further, a system based on the micro-expression recognition method based on a neural network model and a stacking method includes a training set selection module, a video input module, a picture input module, a preprocessing module, an output module, a memory and a processor;

[0034] The video input module is used to input video materials containing facial expressions;

[0035] The preprocessing module is used to process the 3D model in the video material into a 2D picture and save it as a set of pictures to be identified;

[0036] The picture input module is used to input pictures containing facial expressions and save them as the picture set to be identified;

[0037] The processor has a built-in processing program to execute the student status analysis method based on micro-expression recognition according to any one of claims 1 to 6, and marks the expressions recognized in the set of pictures to be recognized in the form of a selection box;

[0038] The training set selection module is used to select the facial expression data set and input the facial expression data set into the processor;

[0039] The output module is used to output the expressions marked in the form of selection boxes.

[0040] Beneficial effects: The present invention discloses a micro-expression recognition method and system based on a neural network model and a stacking method, which compiles five independent convolutional neural network training models together using a stacking method to form a comprehensive meta-model. The comprehensive meta-model effectively integrates the advantages of each basic model and can cover most of the expression data. By adding a custom layer or model for fine-tuning the model, it can be better suitable for expression recognition tasks in different scenarios, improve the ability to cope with complex situations, and improve the accuracy of expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 The present invention discloses a flow chart of a micro-expression recognition method based on a neural network model and a stacking method. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] This embodiment provides a micro-expression recognition method based on a neural network model and a stacking method. Figure 1 As shown, the following steps are included:

[0045] S1: input a facial expression dataset, the facial expression dataset comes from FER2013, CK+ and JAFFE, and normalize the facial expression dataset;

[0046] S2: Divide the facial expression dataset into a training set and a test set;

[0047] S3: inputting the training set into five neural network models for independent training respectively, to obtain five corresponding training models;

[0048] S4: expanding the training set to form an expanded data set;

[0049] S5: Substituting the expanded data set into five corresponding training models for training respectively, and obtaining corresponding training features;

[0050] S6: Determine whether the training feature is stable. If not, use the expansion processing method to expand the expansion data set multiple times, and iterate the five training models multiple times until the stable training feature is output;

[0051] S7: Train the five training models to obtain a meta-model using a stacking method;

[0052] S8: averaging the stable training features and inputting them into the meta-model for training to obtain a comprehensive meta-model, wherein the comprehensive meta-model has stronger generalization ability and performance;

[0053] S9: Input the picture set to be identified into the comprehensive meta-model, and output the facial expressions after the selection.

[0054] In this embodiment, five independent convolutional neural network training models are compiled together using a stacking method to form a comprehensive meta-model. The comprehensive meta-model effectively integrates the advantages of each basic model, can cover most of the expression data, and by adding customized layers or models for fine-tuning the model, it can be better suitable for expression recognition tasks in different scenarios, improve the ability to cope with complex situations, and improve the accuracy of expression recognition.

[0055] In a specific embodiment, the five basic models are simple_CNN, simpler_CNN, tiny_XCEPTION, mini_XCEPTION and big_XCEPTION convolutional neural network models, respectively. The simple_CNN model is a relatively simple CNN model, and the extracted model normalization can be used as an important reference. The simpler_CNN model uses a larger convolution kernel and a smaller stride, and extracts more model features. The tiny_XCEPTION model contains multiple deeply separable convolutional layers, which is more advantageous for small image recognition. The mini_XCEPTION model uses batch normalization and residual connections to improve image recognition performance. Compared with the previous four models, big_XCEPTION has deeper layers, which helps to learn more complex features. Combining these five neural network models together can adapt to expression recognition tasks of various types and scenarios.

[0056] In a specific embodiment, the method of inputting the training set into simple_CNN, simpler_CNN, tiny_XCEPTION, mini_XCEPTION and big_XCEPTION for independent training in S3 is:

[0057] S3.1: performing face recognition on the face features in the training set, and selecting the recognized face image with a candidate frame;

[0058] S3.2: The face images selected by the candidate boxes are input into simple_CNN, simpler_CNN, tiny_XCEPTION, mini_XCEPTION and big_XCEPTION respectively for independent training to obtain five corresponding training models.

[0059] In a specific embodiment, the method of performing face recognition on the face features in the training set and selecting the recognized face image with a candidate frame in S3.1 is:

[0060] S3.1.1: Input the test set, and cyclically scale the test set images, each scaling factor being 0.8, until the width or height of the scaled test set images is less than 12 pixels, and save the process images of the same test set image after each scaling together to form an image pyramid, wherein each layer of the image pyramid represents a version of the original image at different scales;

[0061] S3.1.2: Input the image pyramid into the full convolution network P-Net, output a 1x1x32 feature map, and use the feature map as detection images for three detection branches, respectively. The three detection branches are face classification, bounding box regression, and face feature point positioning. The full convolution network P-Net outputs two arrays, cls_cls_map and reg. The cls_cls_map represents the probability of a face or a non-face, and the reg predicts the position difference of the face position relative to the input image.

[0062] S3.1.3: Set the loss threshold, define the loss function and weight for the three detection branches respectively. The face classification is to distinguish whether each candidate window is a face or a non-face. The face classification uses the cross entropy loss function, which is often used for binary classification problems. The weight is set to 1, indicating a relatively high weight. The frame regression is used to regress the position difference of the candidate window classified as a face relative to the input image. The frame regression uses the sum of squares loss function to measure the difference between the predicted value and the true value in the regression task. The weight is set to 0.5, indicating a relatively low weight. The face feature point positioning loss is used to predict the position of the five feature points of the face in the candidate window. The face feature point positioning uses the sum of squares loss function to measure the difference between the predicted feature point position and the position of the calibrated data. The weight is usually set to 0.5, indicating a relatively low weight. Calculate the loss values ​​of the three detection branches respectively and perform weighted summation on the three loss values ​​to calculate the total loss value. The final total loss is the weighted sum of the three losses, with weights of 1, 0.5, and 0.5 respectively. The setting of these weights is an empirical choice, the purpose of which is to maintain a relative balance while comprehensively considering the three tasks. Through such a loss function design, P-Net can simultaneously consider the three tasks of face classification, bounding box regression and feature point positioning during the training process, improve the accuracy and robustness of face detection, compare the total loss value and the loss threshold, and when the total loss value is greater than the loss threshold, adjust the weights of the convolutional layer and the fully connected layer to optimize the total loss value until the total loss value is less than the loss threshold;

[0063] S3.1.4: Set a screening threshold, which is used to determine which candidate frames should be retained. When the probability of a face in cls_cls_map at a certain position exceeds the screening threshold, the position is determined to contain a face. The position containing the face is combined with the position information in reg to obtain the coordinates of the potential face candidate frame;

[0064] S3.1.5: Use the NMS algorithm to remove the duplicate potential face candidate frames. The NMS algorithm is a non-maximum suppression algorithm. The goal of NMS is to remove candidate frames with high overlap, retain the element with the maximum probability value as the final face frame, select the position with the highest face probability from the cls_cls_map, and use the reg information of the position with the highest face probability and the pixel coordinates of the original image to obtain the final candidate frame coordinates (x1, y1, x2, y2), where x1, y1 represent the coordinates of the upper left corner of the candidate frame, and x2, y2 represent the coordinates of the lower right corner of the candidate frame;

[0065] S3.1.6: Output the coordinates of the candidate box.

[0066] In a specific embodiment, in S4, the expansion processing method includes but is not limited to changing grayscale, local occlusion, image mirroring and rotation.

[0067] In a specific embodiment, in S7, the method of training the five training models to obtain a meta-model by using a stacking method comprises the following steps:

[0068] S7.1: retaining the weights of the base model in the five training models, wherein the base model is used to learn common image features, including but not limited to edges and textures of the images in the training set. Since these common features are useful for image processing and recognition tasks, it is desirable to retain these common features in transfer learning to prevent updating these weights in subsequent training;

[0069] S7.2: According to the target training task, convolutional layers and fully connected layers are added to the five training models to adjust the weights of specific facial features, such as the degree of squinting of eyes, the curvature of mouth corners and the degree of opening; the convolutional layers and fully connected layers are used to fine-tune the five training models to adapt to specific tasks or problems. Some facial expressions are highly similar and easily cause confusion. For example, in some scenarios, surprise and anger are easily confused. By adjusting the weights of specific facial expressions, it is helpful to distinguish similar facial expressions;

[0070] S7.3: Randomly stack the five training models and the convolutional layer and the fully connected layer together to form a meta-model;

[0071] S7.4: A loss function, an optimizer, and an evaluation index are defined in the metamodel, wherein the loss function is used to measure the difference between the output of the metamodel and the true value, the optimizer is used to minimize the loss function so that the model can generate predictions consistent with the actual situation, and the evaluation index is used to measure the performance of the metamodel, providing a more intuitive performance measurement;

[0072] The metamodel can be used for feature extraction. The extended data set is input into the metamodel for feature extraction. Preliminary feature extraction is performed through the shallow or initial layer of the metamodel to obtain the original features of the extended data set. The original features include pixels, colors, edges, and textures. Higher-level abstract features can also be extracted. The abstract features include shapes, objects, and patterns. The extraction process can enhance the expressive power of the model, enabling it to better understand the different levels of information in the input data, which helps the model to more accurately capture the essence of the data. The model has better generalization performance when processing new, unseen data. The model can learn more general feature representations, not just for specific samples of training data.

[0073] In a specific embodiment, a system based on the micro-expression recognition method based on a neural network model and a stacking method includes a training set selection module, a video input module, a picture input module, a preprocessing module, an output module, a memory and a processor;

[0074] The video input module is used to input video materials containing facial expressions;

[0075] The preprocessing module is used to process the 3D model in the video material into a 2D picture and save it as a set of pictures to be identified;

[0076] The picture input module is used to input pictures containing facial expressions and save them as the picture set to be identified;

[0077] The processor has a built-in processing program to execute the student status analysis method based on micro-expression recognition, and marks the expressions recognized in the set of pictures to be recognized in the form of selection boxes;

[0078] The training set selection module is used to select the facial expression data set and input the facial expression data set into the processor. When the accuracy of the expression recognition result is not high, the training set selection module can be used to replace the facial expression data set with a higher similarity to the target recognition task to train the five neural network models and the training model to improve the expression recognition performance of the model;

[0079] The output module is used to mark the recognized expression in the form of a selection box and then output it.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A micro-expression recognition method based on a neural network model and a stacking method, characterized in that: The following steps are involved: S1: inputting a facial expression data set, and performing normalization processing on the facial expression data set; S2: Divide the facial expression dataset into a training set and a test set; S3: inputting the training set into five neural network models for independent training respectively, to obtain five corresponding training models; S4: expanding the training set to form an expanded data set; S5: Substituting the expanded data set into five corresponding training models for training respectively, and obtaining corresponding training features; S6: Determine whether the training feature is stable. If not, use the expansion processing method to expand the expansion data set multiple times, and iterate the five training models multiple times until the stable training feature is output; S7: Train the five training models to obtain a meta-model using a stacking method; S8: averaging the stable training features and inputting them into the meta-model for training to obtain a comprehensive meta-model; S9: Input the picture set to be recognized into the comprehensive meta-model, and output the facial expressions after the selection.

2. A micro-expression recognition method based on a neural network model and a stacking method according to claim 1, characterized in that: The five basic models are simple_CNN, simpler_CNN, tiny_XCEPTION, mini_XCEPTION and big_XCEPTION convolutional neural network models.

3. A micro-expression recognition method based on a neural network model and a stacking method according to claim 2, characterized in that: The method of inputting the training set into simple_CNN, simpler_CNN, tiny_XCEPTION, mini_XCEPTION and big_XCEPTION for independent training in S3 is: S3.1: performing face recognition on the face features in the training set, and selecting the recognized face image with a candidate frame; S3.2: The face images selected by the candidate boxes are input into simple_CNN, simpler_CNN, tiny_XCEPTION, mini_XCEPTION and big_XCEPTION respectively for independent training to obtain five corresponding training models.

4. A micro-expression recognition method based on a neural network model and a stacking method according to claim 3, characterized in that: In S3.1, face recognition is performed on the face features in the training set, and the method of selecting the recognized face image with a candidate frame is: S3.1.1: Input the test set, and cyclically scale the images in the test set, scaling each time according to a set scaling factor until the images in the test set are scaled to a specified size, and save the images of the same image in the test set after each scaling process together to form an image pyramid; S3.1.2: Input the image pyramid into the full convolution network P-Net, output a 1x1x32 feature map, and use the feature map as detection images for three detection branches, respectively. The three detection branches are face classification, bounding box regression, and face feature point positioning. The full convolution network P-Net outputs two arrays, cls_cls_map and reg. The cls_cls_map represents the probability of a face or a non-face, and the reg predicts the position difference of the face position relative to the input image. S3.1.3: Set a loss threshold, define loss functions and weights for the three detection branches respectively, calculate the loss values ​​of the three detection branches respectively and perform weighted summation on the three loss values, calculate a total loss value, compare the total loss value with the loss threshold, and when the total loss value is greater than the loss threshold, adjust the weights of the convolutional layer and the fully connected layer, and optimize the total loss value until the total loss value is less than the loss threshold; S3.1.4: Set a screening threshold. When the probability of a face in cls_cls_map at a certain position exceeds the screening threshold, the position is determined to contain a face. The position containing the face is combined with the position information in reg to obtain the coordinates of the potential face candidate frame. S3.1.5: Use the NMS algorithm to remove the duplicate potential face candidate frames, select the position with the highest face probability from the cls_cls_map, and use the reg information of the position with the highest face probability and the pixel coordinates of the original image to obtain the final candidate frame coordinates (x1, y1, x2, y2), where x1, y1 represents the coordinates of the upper left corner of the candidate frame, and x2, y2 represents the coordinates of the lower right corner of the candidate frame; S3.1.6: Output the coordinates of the candidate box.

5. The micro-expression recognition method based on a neural network model and a stacking method according to claim 1, characterized in that: In S4, the expansion processing method includes but is not limited to changing grayscale, local occlusion, image mirroring and rotation.

6. The micro-expression recognition method based on a neural network model and a stacking method according to claim 1, characterized in that: In S7, the method of training the five training models to obtain a meta-model by using a stacking method comprises the following steps: S7.1: retain the weight of a base model among the five training models, where the base model is used to learn common image features, including but not limited to edges and textures of the images in the training set; S7.2: Add convolutional layers and fully connected layers to the five training models to adjust the weights of expression features according to the target training task; S7.3: Randomly stack the five training models and the convolutional layer and the fully connected layer together to form a meta-model; S7.4: A loss function, an optimizer and an evaluation index are defined in the metamodel, wherein the loss function is used to measure the difference between the output of the metamodel and the true value, the optimizer is used to minimize the loss function, and the evaluation index is used to measure the performance of the metamodel.

7. A system based on the micro-expression recognition method based on a neural network model and a stacking method according to claims 1-6, characterized in that: It includes a training set selection module, a video input module, an image input module, a preprocessing module, an output module, a memory and a processor; The video input module is used to input video materials containing facial expressions; The preprocessing module is used to process the 3D model in the video material into a 2D picture and save it as a set of pictures to be identified; The picture input module is used to input pictures containing facial expressions and save them as the picture set to be identified; The processor has a built-in processing program to execute the student status analysis method based on micro-expression recognition according to any one of claims 1 to 6, and marks the expressions recognized in the set of pictures to be recognized in the form of a selection box; The training set selection module is used to select the facial expression data set and input the facial expression data set into the processor; The output module is used to output the expressions marked in the form of selection boxes.

Images