Facial expression recognition method and system based on time sequence and vectorization

Abstract

The invention discloses a facial expression recognition system and method based on a time sequence and vectorization, and the method comprises the steps: carrying out the processing of an inputted facial video sequence, extracting action units in each frame of image, and arranging the extracted action units into a time sequence according to a time sequence, the method comprises the following steps of: obtaining a plurality of action units, expressing all action units at each time point in a feature vector form, calculating the action units expressed in the feature vector form, generating a complete expression time sequence vector by using the calculated feature vectors, and identifying expressions by using a classification model based on the generated expression time sequence vector. According to the method, the time sequence and the vectorization technology are combined, the problems of sparsity and inaccurate recognition of action units are effectively solved, and the stability and the accuracy of expression recognition are remarkably improved through dynamic time sequence representation.

Description

Technical Field

[0001] The present invention relates to the technical field of facial expression recognition, and in particular to a facial expression recognition method and system based on time series and vectorization. Background Art

[0002] Expression is one of the important ways to convey emotions in interpersonal communication. Facial expression recognition refers to the use of computers to extract facial expression features from detected faces, so that computers can understand and process facial expressions according to human thinking, and respond to people's needs to establish a friendly and intelligent human-computer interaction environment. This research is a frontier hotspot in multidisciplinary cross-studies such as image processing, pattern recognition, psychology, emotional computing, and computer vision. When recognizing facial expressions, specific expressions are usually identified by analyzing facial key points or action units (AUs). It mainly consists of three parts: face detection, expression feature extraction, and expression feature classification and recognition.

[0003] However, existing expression recognition methods face problems such as sparse action units, recognition errors or omissions, which result in insufficient accuracy and robustness of expression recognition. In addition, expression is a dynamic process, and most existing methods recognize expressions based on single-frame images, ignoring the temporal dynamic characteristics of expressions, which affects the stability of recognition. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] Therefore, the purpose of the present invention is to provide a facial expression recognition method and system based on time series and vectorization, which effectively overcomes the sparsity problem of action units and improves the stability and accuracy of expression recognition.

[0006] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0007] A facial expression recognition method based on time series and vectorization, comprising:

[0008] S1, processing the input facial video sequence and extracting the action unit in each frame image;

[0009] S2. Arrange the extracted action units into a time series in chronological order, and represent all action units at each time point in the form of feature vectors. Since each frame of image can detect facial action units (AUs) through an algorithm similar to Openface, according to the facial action coding system updated by Paul Ekman et al. in 2002, there are 43 types of facial action units. Therefore, each frame of image can be represented by a feature vector with a dimension of 43. Arrange them in order of time points to obtain a time series.

[0010] S3, calculating the action unit represented by the feature vector;

[0011] S4, arranging the above-calculated feature vectors according to the original time sequence, and the generated time series vector is the expression time series vector;

[0012] S5. Based on the generated expression time series vector, the expression is recognized using a classification model.

[0013] As a preferred embodiment of the method for facial expression recognition based on time series and vectorization described in the present invention, in the step S1, the action unit in each frame image is extracted by the following method: facial feature points are extracted from the video sequence using a deep learning model, and different action units are further detected and encoded or a psychology-based action unit encoding system is used to extract facial muscle activity patterns through manual marking or automatic detection, or facial expression features are extracted through a deep convolutional neural network, and related action units are automatically identified based on these features.

[0014] As a preferred embodiment of the method for facial expression recognition based on time series and vectorization described in the present invention, in step S3, the specific steps of calculating the action unit represented by the feature vector form are as follows:

[0015] The moment with the largest expression intensity in the time series is selected as the climax moment, and its corresponding action unit vector is represented as c;

[0016] Before and after the climax moment c, select moments within a certain time range as surrounding moments, denoted as o1, o2, ..., o n ;

[0017] Assume that c and o i There is an inherent correlation between the two, and the calculation of the occurrence of o given c i The conditional probability p(o i c)

[0018]

[0019] Among them, c and oi are vector representations, · represents the vector inner product;

[0020] Maximize the sum of the conditional probabilities of all surrounding moments, that is:

[0021]

[0022] Gradually adjust c and o through optimization algorithms such as gradient descent i The vector representation of , so that the above objective function reaches the maximum value, thus obtaining the optimal c and o i The vector representation of .

[0023] As a preferred solution of the method for facial expression recognition based on time series and vectorization described in the present invention, in step S5, the classification model is a support vector machine SVM, a long short-term memory network LSTM or a convolutional neural network CNN.

[0024] As a preferred solution of the facial expression recognition method based on time series and vectorization described in the present invention, the input of the LSTM model is the expression time series vector, the output is the category of the expression, the model is trained by a machine learning method, and then the model is used to recognize the expression.

[0025] A facial expression recognition system based on time series and vectorization, comprising:

[0026] The action unit extraction module is configured to: process the input facial video sequence and extract the action unit in each frame image;

[0027] The time series representation module is configured to: arrange the extracted action units into a time series in chronological order, and represent all the action units at each time point in the form of feature vectors;

[0028] The feature vector calculation module is configured to: calculate the action unit represented in the form of feature vectors;

[0029] The expression time series vector generation module is configured to: generate a complete expression time series vector from the above-calculated feature vector;

[0030] The expression recognition module is configured to: recognize the expression using a classification model based on the generated expression time series vector.

[0031] Compared with the prior art, the present invention has the following beneficial effects: the present invention arranges the action units in the time series into vector form, so that each moment can be represented by a feature vector, thereby effectively overcoming the sparsity problem of the action units. At the same time, through the dynamic time series representation, the similarity between the action unit vector at the climax moment and the action unit vector at its surrounding moments is maximized, thereby improving the stability and accuracy of expression recognition. Compared with the existing methods, the present invention shows higher robustness in expression recognition in complex dynamic scenes and is suitable for various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below in combination with the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0033] Figure 1 A flowchart of a method for facial expression recognition based on time series and vectorization according to the present invention;

[0034] Figure 2 The present invention is a principle block diagram of a facial expression recognition system based on time series and vectorization. DETAILED DESCRIPTION

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0036] Secondly, the present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention in detail, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0037] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0038] The present invention provides a human facial expression recognition method and system based on time series and vectorization, which effectively overcomes the sparsity problem of action units and improves the stability and accuracy of expression recognition.

[0039] Example 1

[0040] Figure 1The flowchart of the method for facial expression recognition based on time series and vectorization of the present invention is shown in FIG. Figure 1 In this embodiment, a facial expression recognition method based on time series and vectorization is used. By arranging the action units in the time series into vector form, a feature vector can be used to represent the action unit at each moment, thereby effectively overcoming the sparsity problem of the action unit. At the same time, by using the idea of ​​the Skip-gram model, the stability and accuracy of expression recognition are improved by maximizing the similarity between the action unit vector at the climax moment and the action unit vector at the surrounding moments. The specific steps are as follows:

[0041] 1. Extraction of action units and time series representation:

[0042] The input facial video sequence is processed to extract the action units in each frame. The extracted action units are arranged into a time series in chronological order, and at each time point, all the action units at that moment are represented in the form of vectors. Commonly used action unit extraction methods include:

[0043] OpenFace: an open source toolkit that uses deep learning models to extract facial landmarks from video sequences and further detect and encode different action units.

[0044] Facial Action Coding System (FACS): A psychology-based action unit coding system that extracts facial muscle activity patterns through manual labeling or automatic detection;

[0045] DeepFace: Extracts features of facial expressions through a deep convolutional neural network (CNN) and automatically identifies related action units based on these features.

[0046] 2. Calculation of eigenvectors:

[0047] Inspired by the Skip-gram model, the calculation of feature vectors is divided into the following steps:

[0048] Define the climax moment c: Select the moment with the largest expression intensity in the time series as the climax moment, and its corresponding action unit vector is denoted as c;

[0049] Selection of surrounding moments O: Before and after the climax moment c, select moments within a certain time range as surrounding moments, denoted as o1, o2, ..., o n ;

[0050] Probability calculation: Assumptions c and o i There is an inherent correlation between the two, and the calculation of the occurrence of o given c i The conditional probability p(oi c)

[0051]

[0052] Among them, c and o i are vector representations respectively, and · represents the vector inner product.

[0053] Optimization goal: maximize the sum of the conditional probabilities of all surrounding moments, that is:

[0054]

[0055] Gradually adjust c and o through optimization algorithms such as gradient descent i The vector representation of , so that the above objective function reaches the maximum value, thus obtaining the optimal c and o i The vector representation of ;

[0056] 3. Generation of expression time series vector:

[0057] The feature vectors calculated above are used to generate a complete expression time series vector representation, which can effectively reduce the error caused by the sparsity of action units and improve the rationality of the similarity calculation between different expression vectors.

[0058] 4. Expression recognition:

[0059] Based on the generated expression time series vector, the expression is recognized using a classification model. The available models include:

[0060] Support Vector Machine (SVM): Uses linear or nonlinear kernel functions to classify vectorized expression features.

[0061] Long Short-Term Memory (LSTM): A recurrent neural network (RNN) that can capture the dynamic characteristics of time series and is suitable for processing continuous time series data.

[0062] Convolutional Neural Network (CNN): It recognizes facial expressions by extracting spatial patterns of time series features and is suitable for shorter time series.

[0063] These models can effectively identify the types of expressions in video sequences by learning feature vectors.

[0064] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A facial expression recognition method based on time series and vectorization, characterized in that: include: S1, processing the input facial video sequence and extracting the action unit in each frame image; S2, arranging the extracted action units into a time series in chronological order, and representing all the action units at each time point in the form of feature vectors; S3, calculating the action unit represented by the feature vector; S4, arranging the above-calculated feature vectors according to the original time sequence, and the generated time series vector is the expression time series vector; S5. Based on the generated expression time series vector, the expression is recognized using a classification model.

2. A facial expression recognition method based on time series and vectorization according to claim 1, characterized in that: In the step S1, the action units in each frame image are extracted specifically by the following method: facial feature points are extracted from the video sequence using a deep learning model, and different action units are further detected and encoded or a psychology-based action unit encoding system is used to extract facial muscle activity patterns through manual labeling or automatic detection or to extract features of facial expressions through a deep convolutional neural network, and related action units are automatically identified based on these features.

3. A facial expression recognition method based on time series and vectorization according to claim 1, characterized in that: In step S3, the specific steps of calculating the action unit represented by the feature vector are as follows: The moment with the largest expression intensity in the time series is selected as the climax moment, and its corresponding action unit vector is represented as c; Before and after the climax moment c, select the moments within 5s as the surrounding moments, denoted as o1, o2, ..., o n ; Assume that c and o i There is an inherent correlation between the two, and the calculation of the occurrence of o given c i The conditional probability p(o i c) Among them, c and o i are vector representations, · represents the vector inner product; Maximize the sum of the conditional probabilities of all surrounding moments, that is: Gradually adjust c and o through optimization algorithms such as gradient descent i The vector representation of , so that the above objective function reaches the maximum value, thus obtaining the optimal c and o i The vector representation of .

4. A facial expression recognition method based on time series and vectorization according to claim 1, characterized in that: In step S5, the classification model is a support vector machine (SVM), a long short-term memory network (LSTM), or a convolutional neural network (CNN).

5. A facial expression recognition method based on time series and vectorization according to claim 4, characterized in that: The input of the LSTM model is the expression time series vector, and the output is the category of the expression. The model is trained by a machine learning method, and then the model is used to recognize the expression.

6. A system for implementing a method for facial expression recognition based on time series and vectorization as described in any one of claims 1 to 4, characterized in that: include: The action unit extraction module is configured to: process the input facial video sequence and extract the action unit in each frame image; The time series representation module is configured to: arrange the extracted action units into a time series in chronological order, and represent all the action units at each time point in the form of feature vectors; The feature vector calculation module is configured to: calculate the action unit represented in the form of feature vectors; The expression time series vector generation module is configured to: generate a complete expression time series vector from the above-calculated feature vector; The expression recognition module is configured to: recognize the expression using a classification model based on the generated expression time series vector.

Images