Facial expression recognition and emotion tracking method based on compound optimization dataset

Abstract

The present invention discloses a method for facial expression recognition and emotion tracking based on a complex optimized dataset, comprising the following steps: S1: dividing and constructing facial expression data according to functional classification; S2: using a confidence learning algorithm to perform complex analysis and assist in artificial mixing to process the corresponding functional datasets, thereby obtaining datasets with low noise rates; S3: training a CNN model based on each dataset to obtain real-time recognition results of facial expression components; and S4: designing an emotion tracking algorithm based on the single-frame expression recognition results to obtain emotion tracking results. Based on an existing large sample dataset, the present invention utilizes a confidence learning algorithm to perform complex analysis and assist in artificial mixing to process data, thereby resolving the issues of a large proportion of noise data in the original dataset and difficulty in recognition, thereby improving the accuracy of expression recognition. Based on this, a corresponding algorithm is designed to achieve functional scenario applications such as emotion tracking.

Description

Technical Field

[0001] The invention relates to an expression recognition and emotion tracking method for an artificial and compound confidence learning improved data set, belonging to the technical field of computer vision. Background Art

[0002] Facial expression recognition is a key application of computer vision technology and plays a crucial role in enabling emotional interaction in robots. Convolutional neural networks, the leading network model for facial expression recognition, have seen continuous development in recent years, with models boasting superior acceleration performance. However, existing large-scale facial expression datasets suffer from low human recognition rates and contain a significant amount of noise that interferes with the learning process, resulting in low model accuracy. Furthermore, with the increasing application of these features, certain user needs for psychological analysis in human-machine emotional interaction cannot be adequately met through facial expressions alone. Summary of the Invention

[0003] In order to improve related technical problems, the present invention proposes an expression recognition and emotion tracking method based on a complex optimized data set. On the basis of the existing large sample data set, the confidence learning algorithm is used to perform complex analysis assisted by manual mixing to complete cleaning and revision, solve the problem of low model learning quality caused by a large number of noise samples, improve recognition accuracy, and deepen functional application scenarios to achieve effective emotion tracking.

[0004] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0005] A method for facial expression recognition and emotion tracking based on a complex optimized data set, characterized by comprising the following steps:

[0006] S1: Divide and construct facial expression data according to functional classification;

[0007] S2: Use the confidence learning algorithm to perform complex analysis and assist in artificial mixing to process the corresponding functional data sets respectively to obtain a data set with low noise rate;

[0008] S3: Based on each data set, the CNN model is trained to obtain real-time recognition results of facial expression components;

[0009] S4: Based on the single-frame expression recognition results, an emotion tracking algorithm is designed to obtain the emotion tracking results.

[0010] Preferably, the specific steps of step S1 are as follows:

[0011] S11: Construct a large-sample training dataset of facial expressions;

[0012] S12: Build a test function dataset, redistribute various types of samples, and perform necessary creation and preprocessing.

[0013] Preferably, in step S11, a large sample data set of tens of thousands and its control data containing artificial information are obtained, the reference information is screened and processed, and a manual processing control group is created through a semi-automatic algorithm and further cleaning and global revision. The reference information is screened and processed, and the screening information field tag set is:

[0014]

[0015] Normalize the confidence of each field, and the confidence of the jth field of the i-th sample is recorded as c ij , filter the data:

[0016]

[0017] Preliminary manual preprocessing of noise sample set:

[0018]

[0019] Further randomly detect and discard x cases of expression class i whose deviation exceeds the threshold R:

[0020]

[0021] where N i The semi-automatic algorithm was globally revised to create a manual control group and revise the sample set:

[0022]

[0023] Preferably, one of the test function data sets in step S12 is reallocated from the video sequence related data set and preprocessed in a unified format with the training data set, and the test function data set is supplemented and created for special objects.

[0024] Preferably, the specific steps of step S2 are as follows:

[0025] S21: Establish a control group data set for confidence learning data processing and analyze the results of single confidence learning processing;

[0026] S22: Combine manual and single-step confidence learning processing methods, and conduct complex analysis processing after testing the training effect.

[0027] Preferably, the single confidence learning processing result in step S21 is used to preliminarily analyze and evaluate the overall quality of the data set, and to determine whether to perform a double processing based on the training effect, as a control group. Assume that the noise label detection rate x of the nth confidence learning is e , detection accuracy x ec , non-detection rate y e , confidence effect threshold R n , then the confidence learning effect is:

[0028]

[0029] If r1(1+r2)≤1, there must be an effect; otherwise, the probability of effectiveness is:

[0030]

[0031] When E n ≥R n , continue with the compound analysis.

[0032] Preferably, the fusion method of the manual and single-mode belief learning processing methods described in step S22 is based on the intersection and union and whether to retain the revised results as the main ideas.

[0033] Preferably, the specific steps of step S3 are as follows:

[0034] Select a lightweight model framework and, based on this, perform necessary manual pruning to further improve acceleration or network expansion to enhance learning capabilities, weighing model performance. Perform image enhancement on each dataset to improve robustness. Use the cross-entropy loss function to train the expression recognition model. Select the optimal model, perform forward propagation, and use the softmax function to obtain real-time recognition output of facial expression components.

[0035] Preferably, the specific steps of step S4 are as follows:

[0036] S41: Verify the effectiveness of the application of single-frame special objects, and set the meta-time window according to the time habits of continuous facial emotional expression as the basis for instantaneous emotional intention feedback;

[0037] S42: Track the changes in emotional components in real time within a dynamic timeframe as a basis for psychological analysis.

[0038] Preferably, the single-frame application special object validity check described in step S41 is performed based on a specific application scenario, such as intention feedback in the emotional interaction of an elderly care robot, using the special object supplementary dataset described in step S12. A meta-time window is set based on the temporal characteristics of continuous facial emotional expression. The meta-time window emotion output result is:

[0039]

[0040] Where t is the length of the meta-time window, x is the sampling frequency, P ij For the The output result of class emotion at j sampling points, Output result for the lth meta-time window, c l Identify the sentiment for the lth meta-time window.

[0041] Preferably, the real-time tracking of emotional components within the dynamic time in step S42 is completed based on the positioning of various emotional element time windows obtained in step S41. Positioning is:

[0042]

[0043] In the dynamic test time window, whose length is T, the sentiment tracking is:

[0044]

[0045] in For the The component ratio of a sentiment class in a dynamically changing test time window of length T.

[0046] Beneficial effects: The present invention provides a method for facial expression recognition and emotion tracking based on a complex optimized data set, which has the following advantages:

[0047] 1. The expression recognition convolutional neural network used in this invention is based on the mini_Xception convolutional neural network framework, and the network is doubled expanded on this basis, which retains excellent acceleration performance while improving learning ability, thereby improving experimental efficiency and reducing the probability of underfitting of the network. At the same time, the softmax classification is used to improve the recognition accuracy and provide an application basis.

[0048] 2. The confidence learning algorithm used in the present invention uses a complex analysis-assisted manual mixing method to process the corresponding functional data set, which not only reduces the recognition bias of manual processing, but also supplements the omitted features of confidence learning, obtains a data set with low noise rate, and greatly improves the model learning effect.

[0049] 3. The present invention designs a semi-automatic algorithm in the manual identification process to organize chaotic data clearly, and there is no need for frequent window switching and long-term manual keyboard input, which greatly reduces the complexity of manual processing and improves work efficiency.

[0050] 4. The present invention performs validity testing on single-frame special object and special situation applications, thereby improving its reliability.

[0051] 5. The present invention sets the meta-time window according to the temporal habits of continuous facial emotional expression, effectively grasping the instantaneous features while improving the anti-noise ability of the emotion tracking process, reducing oscillations, and improving the stability and practicability of the application. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the present invention;

[0053] Figure 2 Semi-automatic creation of a manually processed control group. DETAILED DESCRIPTION

[0054] In order to help those skilled in the art better understand the technical solutions in this application, the technical solutions in the embodiments of this application are clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0055] like Figure 1 As shown, a method for facial expression recognition and emotion tracking based on a complex optimized dataset includes the following steps:

[0056] S1: Obtain the required functional data from the existing facial expression dataset and perform preprocessing and construction. The specific steps are as follows:

[0057] S11: Constructing a large sample training dataset of facial expressions:

[0058] 1. Obtain the original FER2013 facial expression image dataset provided by the Kaggle competition platform, and obtain the FER+ multi-label facial expression image dataset on the GitHub open source platform;

[0059] 2. Filter and process reference information. The filter information field tag set is:

[0060]

[0061] Normalize the confidence of each field, and the confidence of the jth field of the i-th sample is recorded as c ij , filter the data:

[0062]

[0063] Preliminary manual preprocessing of noise sample set:

[0064]

[0065] 3. Further randomly detect and discard x cases of expression class i whose deviation exceeds the threshold R:

[0066]

[0067] where N i is the sample size of expression class i. And design Figure 2 The semi-automatic algorithm was globally revised to create a manually processed control group and revise the sample set:

[0068]

[0069] S12: Build a test function dataset, redistribute various samples, and perform necessary creation and preprocessing:

[0070] Obtain the CK+ video sequence image expression dataset from the official website, extract some required samples based on the expression significance of the same category and each portrait sequence, and redistribute them. In addition, collect facial data of the elderly group among special objects. Preprocess the above raw data, grayscale, resize, detect and crop faces, label the data, and create a normalized standard test dataset. The special object data labels are fed back according to the specific emotional interaction situation requirements. Figure 2 Classification labeling.

[0071] S2: Use the confidence learning algorithm to perform complex analysis, and assist in artificial mixing to process the corresponding functional data sets separately to obtain the denoising rate-compliant data set and its control group. The specific steps are as follows:

[0072] S21: Establish a control group data set for confidence learning data processing and analyze the results of single confidence learning processing:

[0073] The meaning of the joint probability distribution of single-belief learning is:

[0074]

[0075] in Indicates the number of samples corresponding to the potential true label and the actual label. The noise label positioning this time is:

[0076]

[0077] Assume that the nth confidence learning noise label detection rate x e , detection accuracy x ec , non-detection rate y e , confidence effect threshold R n , then the confidence learning effect is:

[0078]

[0079] If r1(1+r2)≤1, there must be an effect; otherwise, the probability of effectiveness is:

[0080]

[0081] When E n ≥R n , continue with the compound analysis.

[0082] S22: Combining manual and single-step confidence learning processing, and conducting complex analysis after testing the training effect:

[0083] Combined with the positioning set ID1 obtained in S11, the transition noise label positioning is:

[0084]

[0085] Repeat S21 until E n <R n , and get the final noise label positioning:

[0086]

[0087] S3: Based on each data set, the CNN model is trained to obtain real-time recognition results of facial expression components. The specific steps are as follows:

[0088] We selected the lightweight model framework mini_Xception and performed a double network expansion on the basic model framework to improve network learning capabilities. Based on various datasets, we performed image enhancement through color transformation, displacement, flipping, and other methods to improve model robustness. We used the cross-entropy loss function to train the expression recognition model. The best model was selected and forward propagated, and the softmax function was used to obtain real-time recognition output of facial expression components. The softmax function formula is:

[0089]

[0090] where y i For the Class output score.

[0091] S4: Using the single-frame expression recognition results of the best model from the best dataset, design an algorithm to achieve effective emotion tracking. The specific steps are as follows:

[0092] S41: Verify the effectiveness of single-frame applications and set meta-time windows based on the temporal habits of continuous facial emotional expressions as a basis for instantaneous emotional intention feedback:

[0093] Considering the application context of emotional interaction, the reliability of single-frame emotional intent feedback is tested based on a special object supplementary dataset. Based on the temporal characteristics of continuous facial emotional expression, a meta-time window is set. The meta-time window emotion output is:

[0094]

[0095] Where t is the length of the meta-time window, x is the sampling frequency, P ij is the output result of the i-th category emotion at j sampling points, Output result for the lth meta-time window, c l Identify the sentiment for the lth meta-time window.

[0096] S42: Track changes in emotional components in real time over a dynamic timeframe as a basis for psychological analysis:

[0097] The time window positioning of various emotion meta-elements is obtained from S41:

[0098]

[0099] In the dynamic test time window, whose length is T, the sentiment tracking is:

[0100]

[0101] in For the The component proportions of each emotion class within a dynamically changing test time window of length T. As T increases, each emotion class is tracked synchronously in real time; when T stops, the cumulative tracking results up to the cutoff time are obtained.

[0102] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to be embodied in the widest possible manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for facial expression recognition and emotion tracking based on a complex optimized dataset, characterized in that: The specific steps include: S1: Divide and construct facial expression data according to functional classification; S2: Use the confidence learning algorithm to perform complex analysis and assist in artificial mixing to process the corresponding functional data sets respectively, and obtain a data set with low noise rate; S3: Based on each data set, the CNN model is trained to obtain real-time recognition results of facial expression components; S4: Based on the single-frame expression recognition results, an emotion tracking algorithm is designed to obtain the emotion tracking results; The specific steps of step S2 are as follows: S21: Establish a control group data set for confidence learning data processing and analyze the results of single confidence learning processing; S22: Combining manual and single-step confidence learning processing, and conducting complex analysis after testing the training effect; The specific steps of step S4 are as follows: S41: Verify the effectiveness of the application of single-frame special objects, and set the meta-time window according to the time habits of continuous facial emotional expression as the basis for instantaneous emotional intention feedback; S42: Track the changes in emotional components in real time within a dynamic timeframe as a basis for psychological analysis.

2. The method for facial expression recognition and emotion tracking based on a complex optimized data set according to claim 1, wherein: The specific steps of step S1 are as follows: S11: Construct a large-sample training dataset of facial expressions; S12: Build a test function dataset, redistribute various types of samples, and perform necessary creation and preprocessing.

3. The method for facial expression recognition and emotion tracking based on a complex optimized data set according to claim 2, wherein: In step S11, a large sample data set of 10,000 and its control data containing artificial information are obtained, the reference information is screened and processed, and a manual processing control group is created through a semi-automatic algorithm and further cleaning and global revision; the reference information is screened and processed, and the screening information field tag set is: Normalize the confidence of each field, and the confidence of the jth field of the i-th sample is recorded as c ij , filter the data: Preliminary manual preprocessing of noise sample set: Further randomly detect and discard x cases of expression class i whose deviation exceeds the threshold R: where N i The sample size of expression class i is set; the semi-automatic algorithm is globally revised to create a manual control group and revise the sample set:

4. The method for facial expression recognition and emotion tracking based on a complex optimized data set according to claim 2, wherein: One of the test function data sets described in step S12 is reallocated from the video sequence related data set and preprocessed in a unified format with the training data set. At the same time, the test function data set is supplemented and created for special objects.

5. The method for facial expression recognition and emotion tracking based on a complex optimized data set according to claim 1, wherein: The single confidence learning processing result described in step S21 is used to preliminarily analyze and evaluate the overall quality of the data set, and to determine whether to perform double processing based on the training effect, as a control group; assuming that the noise label detection rate x of the nth confidence learning is e , detection accuracy x ec , non-detection rate y e , confidence effect threshold R n , then the confidence learning effect is: If r1(1+r2)≤1, there must be an effect; otherwise, the probability of effectiveness is: When E n ≥R n , continue with the compound analysis.

6. The method for facial expression recognition and emotion tracking based on a complex optimized data set according to claim 1, wherein: The specific steps of step S3 are as follows: A lightweight model framework is selected, and on this basis, the model performance is weighed to perform necessary manual pruning to further improve the acceleration capability or network expansion to improve the network learning ability. Image enhancement is performed based on each data set to improve robustness. The expression recognition model is trained using the cross-entropy loss function. The best model is selected for forward propagation, and the real-time recognition output of facial expression components is obtained through the softmax function.

7. The method for facial expression recognition and emotion tracking based on a complex optimized data set according to claim 2, wherein: The single-frame special object application validity test described in step S41 is performed based on the special application scenario using the special object supplementary data set described in step S12; a meta-time window is set according to the temporal habits of continuous facial emotional expression; the meta-time window emotion output result is: Where t is the length of the meta-time window, x is the sampling frequency, P ij is the output result of the i-th category emotion at j sampling points, Output result for the lth meta-time window, c l Identify the sentiment for the lth meta-time window.

8. The method for facial expression recognition and emotion tracking based on a complex optimized data set according to claim 7, wherein: The real-time tracking of emotional components within the dynamic time in step S42 is completed based on the positioning of various emotional element time windows obtained in step S41; the positioning is: In the dynamic test time window, whose length is T, the sentiment tracking is: in is the component proportion of the i-th emotion in the dynamically changing test time window of length T.

Images