Facial expression data calculation method based on facial image
By using face expression recognition technology to adopt methods such as face detection, feature extraction, transfer learning, multimodal information fusion, adversarial sample training and improving the robustness of the algorithm, the problem of poor recognition effect in the uncontrolled environment is solved, and higher accuracy and robustness are achieved.
Patent Information
- Application Number
- CN202411658911.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-19
- Publication Date
- 2025-03-11
AI Technical Summary
Existing facial expression recognition technology is difficult to learn real expression characteristics in uncontrolled environments, and is affected by light, occlusion and posture changes, resulting in a decrease in recognition effect and poor repeatability of results.
The facial expression data calculation method based on facial images is adopted, including face detection, feature extraction, transfer learning, multimodal information fusion, adversarial sample training and improving algorithm robustness, and other steps to improve image quality under low light conditions through deep learning and image enhancement algorithms.
It significantly improves the accuracy and robustness of facial expression data calculations, and can more effectively identify and handle the impact of factors such as lighting, viewing angle and occlusion on the recognition results.
Smart Images
Figure CN119600663A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a facial expression data calculation method based on facial images. Background Art
[0002] The basic principle of facial expression recognition is to determine a person's emotional state by analyzing facial expressions and movements through computer vision and machine learning technology. With the development of deep learning technology, the accuracy and reliability of facial expression recognition continue to improve, making this technology more and more widely used in various fields.
[0003] However, general methods suffer from the poor quality of annotation of expression data collected in uncontrolled environments, making it difficult for the model to learn the true expression features. In practical applications, faces may be occluded or postures may change, resulting in reduced recognition effects. Comparisons between different methods may be unfair, resulting in poor repeatability of the results. Summary of the invention
[0004] The purpose of the present invention is to provide a method for calculating facial expression data based on facial images in order to solve the above problems, thereby solving the problems mentioned in the background technology.
[0005] In order to solve the above problems, the present invention provides a technical solution:
[0006] A method for calculating facial expression data based on facial images comprises the following steps:
[0007] S1: Face detection: locate the face part by predicting a rectangular bounding box. Commonly used algorithms include Haar feature + AdaBoost, HOG feature + SVM, deep learning, etc.
[0008] S2: Feature extraction: Extract key features, such as the shape and position of eyes and mouth. Commonly used feature extraction methods include LBP, HOG and deep features.
[0009] S3: Use transfer learning strategy: transfer the object recognition model or face recognition model to the expression recognition task and use a large-scale face recognition database for training;
[0010] S4: Introducing multimodal information: Using multimodal information such as infrared images and thermal infrared images to solve illumination and occlusion problems;
[0011] S5: Adversarial sample training: By introducing adversarial samples for training, the algorithm can be made more robust in the face of adversarial attacks;
[0012] S6: Improve algorithm robustness: Continuously optimize the algorithm, strengthen model training, and improve the recognition accuracy and stability of the algorithm under various lighting, viewing angles and occlusion conditions;
[0013] S7: Improve annotation quality: Crowdsourcing annotation of data collected in uncontrolled environments can increase the reliability of annotations, although the number of annotations is limited;
[0014] S8: Expression classification: The extracted features are input into the classifier. Commonly used classifiers include SVM, decision tree and neural network.
[0015] As a preferred solution of the present invention, the feature extraction according to S2 mainly includes the following methods:
[0016] Principal component analysis: By mapping high-dimensional data into low-dimensional space, the variance of the retained data is maximized;
[0017] Particle swarm algorithm: Particles follow the optimal particles in the solution space to search, and do not require gradient information, which is suitable for processing real number optimization problems;
[0018] Genetic algorithm: Based on the natural selection and genetic ideas of evolution, it improves the solution of problems by simulating the genetic behavior of organisms;
[0019] Convolutional Neural Network: In deep learning technology, CNN is widely used for feature extraction, which can automatically extract meaningful features from images;
[0020] Gabor wavelet feature operator and uniform local binary pattern: used to extract texture features of images, combined with geometric feature point positioning technology (such as active shape model ASM).
[0021] As a preferred solution of the present invention, the step of introducing multimodal information in S4 includes the following steps:
[0022] S41: Optimize data collection and annotation methods: Use methods such as automated annotation and semi-supervised learning to reduce the cost of data collection and annotation;
[0023] S42: Optimizing fusion algorithms and classifiers: Using deep learning and other methods to optimize fusion algorithms and classifiers;
[0024] S43: Consider the information conflict and redundancy problems between multiple modalities: Use methods such as information fusion to deal with the information conflict and redundancy problems between different modalities.
[0025] As a preferred solution of the present invention, the adversarial sample training in S5 mainly includes the following steps:
[0026] S51: Deep feature fusion: Through in-depth research on the importance of color feature information of face images for face detection, a deep feature fusion network structure is constructed using a deep convolutional neural network;
[0027] S52: Mirror feature projection: First, characterize the mirror feature space corresponding to the real image, learn the projection of real data and fake data, then train the SVM model based on the real projection, and use 3D mask projection and printed photo projection as the anti-spoofing model for detection simulation;
[0028] S53: Image quality assessment: Compare the original image with the processed image based on a combination of image quality metrics;
[0029] S54: Adversarial sample generation: Attacking the face recognition system by adding adversarial perturbations of specific shapes to the original face samples to achieve the purpose of misleading or pretending to attack.
[0030] As a preferred solution of the present invention, the method of improving the robustness of the algorithm according to S6 mainly includes the following steps:
[0031] S61: Model Optimization:
[0032] Parameter adjustment: adjust key parameters such as model hyperparameters and number of iterations, and find the optimal combination through methods such as cross-validation;
[0033] Ensemble learning: Ensemble learning using multiple complementary models;
[0034] Model regularization: by adding regularization terms (such as L1, L2 regularization);
[0035] S62: Data enhancement:
[0036] Data augmentation: Generate more samples to increase the amount of data by using operations such as image rotation, cropping, and scaling;
[0037] Noise injection: Adding appropriate amount of noise to the data to help the model learn a wider feature distribution;
[0038] S63: Specific technical means:
[0039] Dropout and BatchNormalization: In neural networks, these techniques are added to improve the robustness of the model;
[0040] Regularization term in least squares method: Regularization term is added to least squares method to prevent overfitting;
[0041] Sparsity constraints in maximum likelihood estimation: Sparsity constraints are added to maximum likelihood estimation to improve the robustness of the model.
[0042] As a preferred solution of the present invention, the S42: optimizing the fusion algorithm and the classifier is divided into optimizing the fusion algorithm and the separator;
[0043] Optimize fusion algorithm: By fusing multiple features, the algorithm can improve the accuracy and robustness of expression recognition;
[0044] Separator: In facial expression recognition, it is responsible for matching the extracted features with known expression categories to output the final recognition result.
[0045] As a preferred embodiment of the present invention, the application fields of the facial expression recognition include:
[0046] Human-computer interaction: By recognizing the user's facial expressions, the machine can better understand the user's emotions, thereby providing a more humanized interactive experience;
[0047] Intelligent monitoring: In the monitoring system, facial expression recognition can help analyze the emotional state of the crowd and improve the intelligence level of the monitoring system;
[0048] Emotion analysis: In psychology and sociology research, facial expression recognition can help analyze people’s emotional changes and provide important data support for research.
[0049] As a preferred solution of the present invention, the related technical principles include:
[0050] Convolutional Neural Network: CNN is a deep learning model that extracts image features through convolution operations and is suitable for tasks such as image classification and object detection;
[0051] Attention Mechanism: When processing complex images, the attention mechanism can help the model focus more on key areas.
[0052] The beneficial effects of the present invention are: by setting up and using a transfer learning strategy, introducing multimodal information, adversarial sample training and improving algorithm robustness, more features are extracted through deep learning and other technologies, and image enhancement algorithms are combined to improve image quality under low light conditions, the accuracy and robustness of facial expression data calculation based on facial images can be significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] For ease of explanation, the present invention is described in detail with reference to the following specific implementations and the accompanying drawings.
[0054] Figure 1 is a flow chart of the facial expression data calculation method of the present invention;
[0055] Figure 2 It is a flow chart of improving the robustness of the algorithm of the present invention. DETAILED DESCRIPTION
[0056] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0057] Embodiment 1:
[0058] See also Figure 1-Figure 2 The present invention provides a technical solution: a method for calculating facial expression data based on facial images, comprising the following steps:
[0059] S1: Face detection: locate the face part by predicting a rectangular bounding box. Commonly used algorithms include Haar feature + AdaBoost, HOG feature + SVM, deep learning, etc.
[0060] S2: Feature extraction: Extract key features, such as the shape and position of eyes and mouth. Commonly used feature extraction methods include LBP, HOG and deep features.
[0061] S3: Use transfer learning strategy: transfer object recognition model or face recognition model to expression recognition task, and use large-scale face recognition database for training to improve the generalization ability of expression recognition;
[0062] S4: Introducing multimodal information: Using multimodal information such as infrared images and thermal infrared images to solve illumination and occlusion problems and improve the robustness of recognition;
[0063] S5: Adversarial sample training: By introducing adversarial samples for training, the algorithm can be made more robust in the face of adversarial attacks;
[0064] S6: Improve algorithm robustness: Continuously optimize the algorithm, strengthen model training, and improve the recognition accuracy and stability of the algorithm under various lighting, viewing angles and occlusion conditions;
[0065] S7: Improve annotation quality: Crowdsourcing annotation of data collected in uncontrolled environments can increase the reliability of annotations, although the number of annotations is limited;
[0066] S8: Expression classification: The extracted features are input into the classifier. Commonly used classifiers include SVM, decision tree and neural network.
[0067] In human-computer interaction, facial expression recognition can help us better understand the user's emotional state and provide more personalized services. In psychological research, facial expression recognition can help researchers analyze the emotional state of individuals and further explore the patterns and mechanisms of human emotional expression. In addition, in the field of education and training, facial expression recognition technology can also be used to evaluate students' emotional reactions and learning outcomes.
[0068] Embodiment 2:
[0069] The feature extraction method according to S2 mainly includes the following methods:
[0070] Principal component analysis: By mapping high-dimensional data into low-dimensional space, the variance of the data is maximized, thereby reducing the data dimension while retaining important feature information;
[0071] Particle swarm algorithm: Particles follow the optimal particles in the solution space to search, and do not require gradient information, which is suitable for processing real number optimization problems;
[0072] Genetic algorithm: Based on the natural selection and genetic ideas of evolution, it improves the solution of the problem by simulating the genetic behavior of organisms to obtain the optimal solution;
[0073] Convolutional Neural Network: In deep learning technology, CNN is widely used for feature extraction, which can automatically extract meaningful features from images, such as the shape, color, and texture of objects;
[0074] Gabor wavelet feature operator and uniform local binary pattern: used to extract texture features of images. Combined with geometric feature point positioning technology (such as active shape model ASM), key feature points of the face, such as the eyes, nose and mouth, can be accurately located.
[0075] It has broad application prospects and important research value in the field of computer vision, especially in the fields of autonomous driving, security monitoring, medical diagnosis and human-computer interaction.
[0076] Embodiment 3:
[0077] According to the introduction of multimodal information in S4, there are the following steps:
[0078] S41: Optimize data collection and annotation methods: Use methods such as automated annotation and semi-supervised learning to reduce the cost of data collection and annotation;
[0079] S42: Optimize fusion algorithms and classifiers: Use deep learning and other methods to optimize fusion algorithms and classifiers to improve recognition accuracy and robustness;
[0080] S43: Consider the information conflict and redundancy problems between multiple modalities: Use methods such as information fusion to deal with the information conflict and redundancy problems between different modalities to improve the accuracy and robustness of recognition.
[0081] The multimodal fusion recognition method in face recognition is a technology with potential, which can improve the accuracy and robustness of recognition, and can also better adapt to complex application scenarios. With the continuous development of technology and the continuous expansion of application scenarios, the multimodal fusion recognition method will play an increasingly important role in the field of face recognition.
[0082] Embodiment 4:
[0083] The adversarial sample training according to S5 mainly includes the following steps:
[0084] S51: Deep feature fusion: Through in-depth research on the importance of color feature information of face images for face detection, a deep feature fusion network structure is constructed using a deep convolutional neural network to effectively train relevant face anti-spoofing data;
[0085] S52: Mirror feature projection: First, characterize the mirror feature space corresponding to the real image, learn the projection of real data and fake data, then train the SVM model based on the real projection, and use 3D mask projection and printed photo projection as the anti-spoofing model for detection simulation;
[0086] S53: Image quality assessment: Compare the original image with the processed image based on a combination of image quality metrics to assess the quality and authenticity of the image;
[0087] S54: Adversarial sample generation: Attack the face recognition system by adding adversarial perturbations of specific shapes to the original face samples to achieve the purpose of misleading or pretending to attack. Specific methods include using facial key points to construct masks of specific shapes to generate adversarial perturbations, designing adversarial loss functions and print score loss functions to reduce print color differences, and reproducing the generation of adversarial samples in the physical domain.
[0088] By introducing adversarial samples during the training process, the model can be better identified and defended against potential attacks, thereby improving the robustness and security of the model.
[0089] Embodiment 5:
[0090] The method of improving the robustness of the algorithm according to S6 mainly includes the following steps:
[0091] S61: Model Optimization:
[0092] Parameter adjustment: adjust key parameters such as model hyperparameters and number of iterations, and find the optimal combination through methods such as cross-validation;
[0093] Ensemble learning: Ensemble learning is performed using multiple complementary models to reduce bias and variance while improving the generalization ability of the algorithm;
[0094] Model regularization: By adding regularization terms (such as L1 and L2 regularization), the complexity of the model is reduced, the risk of overfitting is reduced, and robustness is improved;
[0095] S62: Data enhancement:
[0096] Data augmentation: Use operations such as image rotation, cropping, and scaling to generate more samples to increase the amount of data and improve the training effect of the model;
[0097] Noise injection: Adding an appropriate amount of noise to the data helps the model learn a wider range of feature distributions, thereby enhancing the algorithm’s robustness to input changes;
[0098] S63: Specific technical means:
[0099] Dropout and BatchNormalization: In neural networks, these techniques are added to improve the robustness of the model;
[0100] Regularization term in least squares method: Regularization term is added to least squares method to prevent overfitting and thus improve the robustness of the model;
[0101] Sparsity constraints in maximum likelihood estimation: Adding sparsity constraints in maximum likelihood estimation can improve the robustness of the model.
[0102] It can effectively improve the robustness of the algorithm, making it more stable and reliable in practical applications.
[0103] Feature selection is the process of selecting the most important and relevant feature subsets from the original data. It can help eliminate redundant information and noise, improve the training effect and robustness of the model, train multiple sub-models by randomly selecting sample sets, and then integrate these sub-models through voting or averaging, adjust model parameters, and select the optimal model. The robustness of parameter estimation can be improved by adding regularization terms, sparsity constraints and other techniques to prevent overfitting and improve the robustness of the model.
[0104] Embodiment 6:
[0105] The S42: optimizing the fusion algorithm and the classifier is divided into optimizing the fusion algorithm and the separator;
[0106] Optimizing the fusion algorithm: By fusing multiple features, the algorithm can improve the accuracy and robustness of expression recognition, thereby improving the accuracy of recognition;
[0107] Separator: In facial expression recognition, it is responsible for matching the extracted features with known expression categories to output the final recognition results. An efficient classifier can quickly match features, reduce computing time, and improve recognition efficiency.
[0108] Through preprocessing methods such as filtering, optimizing the fusion algorithm can reduce the noise in the image, improve the image quality, and thus improve the recognition effect. The classifier can adapt to different conditions such as lighting and posture, and improve the robustness of the recognition system.
[0109] Embodiment 7:
[0110] The application areas of facial expression recognition include:
[0111] Human-computer interaction: By recognizing the user's facial expressions, the machine can better understand the user's emotions, thereby providing a more humanized interactive experience;
[0112] Intelligent monitoring: In the monitoring system, facial expression recognition can help analyze the emotional state of the crowd and improve the intelligence level of the monitoring system;
[0113] Emotion analysis: In psychology and sociology research, facial expression recognition can help analyze people’s emotional changes and provide important data support for research.
[0114] The relevant technical principles include:
[0115] Convolutional Neural Network: CNN is a deep learning model that extracts image features through convolution operations and is suitable for tasks such as image classification and object detection;
[0116] Attention mechanism: When processing complex images, the attention mechanism can help the model focus more on key areas and improve recognition accuracy.
[0117] Together they form the basic framework for calculating facial expression data based on facial images and have been widely used in many fields.
[0118] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for calculating facial expression data based on facial images, characterized in that: The following steps are involved: S1: Face detection: locate the face part by predicting a rectangular bounding box. Commonly used algorithms include Haar feature + AdaBoost, HOG feature + SVM, deep learning, etc. S2: Feature extraction: Extract key features, such as the shape and position of eyes and mouth. Commonly used feature extraction methods include LBP, HOG and deep features. S3: Use transfer learning strategy: transfer the object recognition model or face recognition model to the expression recognition task and use a large-scale face recognition database for training; S4: Introducing multimodal information: Using multimodal information such as infrared images and thermal infrared images to solve illumination and occlusion problems; S5: Adversarial sample training: By introducing adversarial samples for training, the algorithm can be made more robust in the face of adversarial attacks; S6: Improve algorithm robustness: Continuously optimize the algorithm, strengthen model training, and improve the recognition accuracy and stability of the algorithm under various lighting, viewing angles and occlusion conditions; S7: Improve annotation quality: Crowdsourcing annotation of data collected in uncontrolled environments can increase the reliability of annotations, although the number of annotations is limited; S8: Expression classification: The extracted features are input into the classifier. Commonly used classifiers include SVM, decision tree and neural network.
2. A facial expression data calculation method based on facial images according to claim 1, characterized in that, The feature extraction method according to S2 mainly includes the following methods: Principal component analysis: By mapping high-dimensional data into low-dimensional space, the variance of the retained data is maximized; Particle swarm algorithm: Particles follow the optimal particles in the solution space to search, and do not require gradient information, which is suitable for processing real number optimization problems; Genetic algorithm: Based on the natural selection and genetic ideas of evolution, it improves the solution of problems by simulating the genetic behavior of organisms; Convolutional Neural Network: In deep learning technology, CNN is widely used for feature extraction, which can automatically extract meaningful features from images; Gabor wavelet feature operator and uniform local binary pattern: used to extract texture features of images, combined with geometric feature point positioning technology (such as active shape model ASM).
3. A facial expression data calculation method based on facial images according to claim 1, characterized in that, According to the introduction of multimodal information in S4, there are the following steps: S41: Optimize data collection and annotation methods: Use methods such as automated annotation and semi-supervised learning to reduce the cost of data collection and annotation; S42: Optimizing fusion algorithms and classifiers: Using deep learning and other methods to optimize fusion algorithms and classifiers; S43: Consider the information conflict and redundancy problems between multiple modalities: Use methods such as information fusion to deal with the information conflict and redundancy problems between different modalities.
4. A facial expression data calculation method based on facial images according to claim 1, characterized in that, The adversarial sample training according to S5 mainly includes the following steps: S51: Deep feature fusion: Through in-depth research on the importance of color feature information of face images for face detection, a deep feature fusion network structure is constructed using a deep convolutional neural network; S52: Mirror feature projection: First, characterize the mirror feature space corresponding to the real image, learn the projection of real data and fake data, then train the SVM model based on the real projection, and use 3D mask projection and printed photo projection as the anti-spoofing model for detection simulation; S53: Image quality assessment: Compare the original image with the processed image based on a combination of image quality metrics; S54: Adversarial sample generation: Attacking the face recognition system by adding adversarial perturbations of specific shapes to the original face samples to achieve the purpose of misleading or pretending to attack.
5. A facial expression data calculation method based on facial images according to claim 1, characterized in that, The method of improving the robustness of the algorithm according to S6 mainly includes the following steps: S61: Model Optimization: Parameter adjustment: adjust key parameters such as model hyperparameters and number of iterations, and find the optimal combination through methods such as cross-validation; Ensemble learning: Ensemble learning using multiple complementary models; Model regularization: by adding regularization terms (such as L1, L2 regularization); S62: Data enhancement: Data augmentation: Generate more samples to increase the amount of data by using operations such as image rotation, cropping, and scaling; Noise injection: Adding appropriate amount of noise to the data to help the model learn a wider feature distribution; S63: Specific technical means: Dropout and BatchNormalization: In neural networks, these techniques are added to improve the robustness of the model; Regularization term in least squares method: Regularization term is added to least squares method to prevent overfitting; Sparsity constraints in maximum likelihood estimation: Sparsity constraints are added to maximum likelihood estimation to improve the robustness of the model.
6. A facial expression data calculation method based on facial images according to claim 3, characterized in that, The S42: optimizing the fusion algorithm and the classifier is divided into optimizing the fusion algorithm and the separator; Optimize fusion algorithm: By fusing multiple features, the algorithm can improve the accuracy and robustness of expression recognition; Separator: In facial expression recognition, it is responsible for matching the extracted features with known expression categories to output the final recognition result.
7. A facial expression data calculation method based on facial images according to claims 1-6, characterized in that: The application areas of facial expression recognition include: Human-computer interaction: By recognizing the user's facial expressions, the machine can better understand the user's emotions, thereby providing a more humanized interactive experience; Intelligent monitoring: In the monitoring system, facial expression recognition can help analyze the emotional state of the crowd and improve the intelligence level of the monitoring system; Emotion analysis: In psychology and sociology research, facial expression recognition can help analyze people’s emotional changes and provide important data support for research.
8. A facial expression data calculation method based on facial images according to claim 7, characterized in that, The relevant technical principles include: Convolutional Neural Network: CNN is a deep learning model that extracts image features through convolution operations and is suitable for tasks such as image classification and object detection; Attention Mechanism: When processing complex images, the attention mechanism can help the model focus more on key areas.
Citation Information
Cited By
Image sentiment analysis method fusing multi-dimensional features
CN120656223A