Micro-expression recognition method, system and computer medium based on graph convolutional network

Abstract

The present invention belongs to the field of facial expression recognition technology and specifically discloses a micro-expression recognition method, system, and computer medium based on a graph convolutional network. This method extracts multi-scale local features of each AU based on the correspondence between each AU and a specific facial region. Using these features, an AU correlation matrix is constructed. Based on the AU correlation matrix, two two-layer graph convolutional networks are constructed, one of which is used to update the feature representation of each AU, and the other is used to update the AU classifier. The AU classifiers are trained using a multi-label classification loss function to obtain a final AU classifier for facial micro-expression recognition. This technical solution, based on a multi-scale feature extraction algorithm based on activation regions and two graph convolutional networks, simulates the correlation characteristics between AU features and the correlation characteristics between AU classifiers, ensuring the accuracy of AU recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of expression recognition, and relates to a micro-expression recognition method, system and computer medium based on a graph convolutional network. Background Art

[0002] Facial expressions, especially micro-expressions, imply a person's true inner feelings and play an important role in interpersonal communication, mental state diagnosis, and improving education quality.

[0003] There are two types of facial expression recognition methods: machine learning-based and deep learning-based. Machine learning-based methods are susceptible to poor feature extraction, while deep learning-based methods combine feature extraction and expression classification to overcome the shortcomings of machine learning-based methods. Among deep learning-based methods, convolutional neural networks are widely used to build expression classification models. Matsugu et al. (Matsugu M, Mori K, Mitari Y, et al. Subject independent facial expression recognition with robust face detection using a convolutional neural network [J]. Neural Networks, 2003, 16(5-6): 555-559) used a convolutional neural network to build a facial expression classifier for facial expression classification. Although this method achieved a high recognition rate, the classifier built using the convolutional neural network could only extract relevant information from an eight-neighborhood image. It had poor ability to extract relevant information between pixels that were far apart, which affected the accuracy of the classifier in facial expression recognition.

[0004] Because facial expressions are formed by the movement of facial muscles, existing technologies use facial action units (AUs) to more accurately describe subtle changes in facial expressions and reveal true inner feelings. A central point is designed for each AU, and local features are manually extracted based on this central point. However, the activation areas of different AUs vary, and using a feature representation at the same scale cannot effectively capture the scale differences of AUs. Alternatively, AU recognition algorithms combined with graph convolutional networks (GCNNs) are used to update the local features of AUs. This only considers the correlation between AU expression features and ignores the correlation between classifiers that ultimately identify AUs, reducing the effectiveness of AU recognition. Summary of the Invention

[0005] The purpose of the present invention is to provide a micro-expression recognition method, system and computer medium based on graph convolutional networks to improve the recognition effect of facial micro-expressions.

[0006] To achieve the above objectives, the present invention provides a method for recognizing facial micro-expressions based on a graph convolutional network, comprising the following steps:

[0007] Extract AU multi-scale local features based on the correspondence between each AU and a specific facial area;

[0008] Utilize AU multi-scale local features to construct AU correlation matrix;

[0009] Based on the AU correlation matrix, two two-layer graph convolutional networks are constructed, one of which is used to update the feature representation of each AU, and the other is used to update the AU classifier;

[0010] The AU classifier is trained using a multi-label classification loss function;

[0011] The final AU classifier is obtained for facial micro-expression recognition.

[0012] The working principles and beneficial effects of this basic solution are as follows: This solution designs a feature representation algorithm that effectively captures the multi-scale and local characteristics of AUs, improving the feature expression capabilities of AU recognition and laying the foundation for improved AU recognition. A graph neural network that can simulate the correlation between AUs incorporates prior knowledge of AU correlation into the feature representation, improving AU recognition efficiency. A classifier that simulates the correlation between AUs improves the final results of AU recognition.

[0013] Furthermore, the method for extracting AU multi-scale local features is as follows:

[0014] Using a multi-scale local area feature extraction algorithm, a multi-scale window is predefined according to the size of the AU activation area and mapped to the final feature layer extracted by the convolutional neural network to obtain the AU multi-scale local features;

[0015] Use RoIAlign to generate fixed-size features and unify the sizes of AU features in different regions;

[0016] The paired AU features of uniform size are fused through average pooling operation, and the 64-dimensional AU features are obtained through dimensionality reduction through fully connected mapping.

[0017] The facial area differences of each AU when activated are extracted in a targeted manner to facilitate subsequent recognition processing and improve accuracy.

[0018] Furthermore, the method of constructing the AU correlation matrix is:

[0019] The correlation matrix A between AUs is defined by the frequency of simultaneous occurrence between AUs and their total frequency of occurrence:

[0020] A i，j =Q i，j / N i ,

[0021] Among them, A i,j is the correlation matrix when the i-th AU and the j-th AU appear in a face image at the same time, Q i,j is the frequency of simultaneous occurrence of the i-th AU and the j-th AU, N i is the frequency of occurrence of the i-th AU.

[0022] To simulate the correlation between AUs using graph convolutional networks, it is first necessary to define the AU correlation matrix for subsequent operations.

[0023] Furthermore, the graph convolutional network layer used to update the AU feature representation is:

[0024]

[0025] Among them, H (l) is the node representation of the lth layer, H (l+1) is the node representation of the l+1th layer, D is the dimension reduction matrix of the association matrix A, defined as D = ∑ j (A+I) ij ; Since A is a square matrix, the dimensions of the node input and output remain unchanged, σ is the activation function leakyReLU, I is the identity matrix, and W is the parameter in the graph neural network.

[0026] The AU correlation propagation algorithm based on graph convolutional networks not only improves the AU feature expression ability, but also improves the ability of the AU classifier.

[0027] Furthermore, when the AU feature representation propagates AU correlation, the local multi-scale features extracted for each AU are input;

[0028] When propagating AU correlation in the AU classifier, the pre-trained classifier is input and then the graph convolutional network is used to iteratively update the AU classifier.

[0029] The AU classifier based on graph convolutional network transfers the correlation of AUs into the classifier, thereby directly improving the recognition effect of AUs.

[0030] Furthermore, the loss function is:

[0031]

[0032] in is the predicted value of the algorithm, and is the AU feature representation and classifier generated after the graph convolutional network; ω i is the weight, y i is the true value, is the predicted value.

[0033] The loss function is used to train the AU classifier to optimize the recognition accuracy of the AU classifier.

[0034] The present invention also provides a facial micro-expression recognition system based on a graph convolutional network, comprising a facial image acquisition unit and a processing unit. The facial image acquisition unit is used to acquire a facial image to be recognized. The output end of the facial image acquisition unit is connected to the processing unit. The processing unit executes the method of the present invention to perform facial micro-expression recognition.

[0035] Using this system, the facial micro-expression recognition algorithm based on graph convolutional networks can automatically detect facial motion units, facilitating the subsequent recognition and application of micro-expressions.

[0036] The present invention also provides a computer medium storing a program for executing the method of the present invention.

[0037] Computer media are easy to use and can be used to perform micro-expression recognition operations on a variety of corresponding processing devices, thereby expanding the scope of use. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the process of facial micro-expression recognition method based on graph convolutional network of the present invention;

[0039] Figure 2 Schematic diagram of the structure of extracting AU multi-scale local features in a preferred solution of the present invention;

[0040] Figure 3 Schematic diagram of the structure of the AU correlation matrix of a preferred embodiment of the present invention;

[0041] Figure 4 It is a schematic diagram of the process of corresponding updating of the graph convolutional network of the facial micro-expression recognition method based on the graph convolutional network of the present invention. DETAILED DESCRIPTION

[0042] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0043] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0044] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0045] Each AU corresponds to a specific facial region, thus possessing strong local features. AUs also exhibit correlations. For example, AU6 (cheek lift) and AU12 (mouth corner stretch) often appear in pairs, indicating a smile. Regionality and AU correlation are crucial for micro-expression recognition.

[0046] Based on this, the present invention discloses a facial micro-expression recognition method based on a graph convolutional network. This method targets the expression and inherent correlation characteristics of AUs. First, since each AU corresponds to a specific facial region and the activation regions of each AU vary significantly, this solution proposes a multi-scale feature extraction algorithm based on activation regions. This algorithm extracts appropriate local features for each AU, ensuring the accuracy of AU recognition. Second, to effectively capture the correlation characteristics between AUs, two graph convolutional networks are designed to simulate the correlation characteristics between AU features and the correlation characteristics between AU classifiers, respectively. This transfers the prior knowledge of the correlation between AUs into the feature representation, improving the efficiency of AU recognition.

[0047] like Figure 1 As shown, the facial micro-expression recognition method of this embodiment includes the following steps:

[0048] According to the correspondence between each AU and the specific area of the face, based on the concept of AU center point, an improved AU center point is proposed as shown in Table 1 to extract AU multi-scale local features;

[0049] Table 1 Improved AU center points

[0050]

[0051]

[0052] Utilize the multi-scale local features of AUs to construct the AU correlation matrix so that the correlation between AUs can be simulated using the graph convolutional network.

[0053] Based on the AU correlation matrix, two two-layer graph convolutional networks are constructed, where one graph convolutional network is used to update the feature representation of each AU, and the other graph convolutional network is used to update the AU classifier; Figure 4 As shown in the figure, based on MSUR feature extraction, when propagating AU correlation in AU feature representation, local multi-scale features extracted for each AU are input; when propagating AU correlation in AU classifier, pre-trained classifiers are input, and AU relationship modeling with GCN is used to pre-set scores for each AU. AU classifiers are then iteratively updated using a graph convolutional network. That is, by calculating the multi-label loss and updating using the backpropagation algorithm, this is a neural network training method. The purpose of the update is to learn a classifier that is more suitable for AUs.

[0054] The multi-label classification loss function is used to train the AU classifier and obtain the final AU classifier for facial micro-expression recognition.

[0055] In a preferred embodiment of the present invention, the method for extracting multi-scale local features of AU is as follows:

[0056] Based on the differences in facial regions when each AU is activated, a multi-scale local region feature extraction algorithm is used. Based on the size of the AU activation region, multi-scale windows (of different scales) are predefined, that is, activation templates are predefined. The activation templates of each AU vary greatly. When each AU is activated, the pre-defined activation template is used. The multi-scale windows are then mapped to the final feature layer extracted by the convolutional neural network ResNet-34 to obtain AU multi-scale local features. This is also a common algorithm in neural networks. Mapping the original image area to the feature layer only requires determining which areas in the feature map are associated with a given area in the original image. This can be determined by sensing the feature map.

[0057] Use RoIAlign (Region of Interest Alignment, RoIAlign is generally used as a proprietary name, from Kaiming He's Mask RCNN) to generate fixed-size features and unify the sizes of AU features in different regions;

[0058] Since AUs appear in pairs, the paired AU features of uniform size are fused through average pooling operation, and the 64-dimensional AU features are obtained through full-connection mapping and dimensionality reduction. Figure 2 As shown in the figure, taking AU2 as an example, the feature map of the last layer of ResNet-34 is 512x7x7. The pre-set window is mapped to the feature map, the fixed-length features are generated by the RoIAlign operator, and the paired features are fused through the average pooling operator (avg.pooling). The final features are reduced to 64-dimensional features through the fully connected mapping.

[0059] In a preferred embodiment of the present invention, the method for constructing the AU correlation matrix is:

[0060] The correlation matrix A between AUs is defined by the frequency of simultaneous occurrence between AUs and their total frequency of occurrence:

[0061] A i，j =Q i，j / N i ,

[0062] Among them, A i,j is the correlation matrix when the i-th AU and the j-th AU appear in a face image at the same time, Q i,j is the frequency of simultaneous occurrence of the i-th AU and the j-th AU, N i is the frequency of occurrence of the i-th AU, and the specific definition structure of the AU correlation matrix is as follows Figure 3 shown.

[0063] In a preferred embodiment of the present invention, the graph convolutional network layer update for updating the AU feature representation is defined as:

[0064]

[0065] Among them, H (l) is the node representation of the lth layer, H (l+1) is the node representation of the l+1th layer, D is the dimension reduction matrix of the association matrix A, defined as D = ∑ j (A+I) ij ; Since A is a square matrix, the dimensions of the node input and output remain unchanged, σ is the activation function leakyReLU, f(x)=x,if x>=0; f(x)=alpha x,x<0,alpha is 0.001; I is the unit matrix, and W is the parameter in the graph neural network.

[0066] The loss function is:

[0067]

[0068] Using the loss function, the feedforward propagation training and BP propagation training of the neural network are carried out, where is the predicted value of the algorithm, and is the AU feature representation and classifier generated after the graph convolutional network; ω i is the weight, y i is the true value, is the predicted value.

[0069] The present invention also provides a facial micro-expression recognition system based on a graph convolutional network, comprising a facial image acquisition unit and a processing unit. The facial image acquisition unit is used to acquire a facial image to be recognized. The output of the facial image acquisition unit is connected to the processing unit, which executes the method described in the present invention to recognize facial micro-expressions. Using this system, the facial micro-expression recognition algorithm based on a graph convolutional network can automatically detect facial motion units, facilitating the subsequent recognition and application of micro-expressions.

[0070] This solution proposes a multi-scale local AU feature extraction algorithm that effectively improves AU feature expression. It also incorporates prior knowledge in a predefined manner, significantly accelerating inference and training. An AU feature correlation learning algorithm based on a graph convolutional network effectively learns correlations between AUs and improves AU recognition efficiency through AU correlation analysis. An AU classifier based on a graph convolutional network transfers AU correlations to the classifier, directly improving AU recognition.

[0071] The present invention also provides a computer medium storing a program for executing the method of the present invention. The computer medium is easy to use and can be used to perform micro-expression recognition operations on a variety of corresponding processing devices, thereby expanding its scope of application.

[0072] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0073] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A facial micro-expression recognition method based on graph convolutional network, characterized in that: The steps include: Extract AU multi-scale local features based on the correspondence between each AU and a specific facial area; Utilize AU multi-scale local features to construct AU correlation matrix; Based on the AU correlation matrix, two two-layer graph convolutional networks are constructed, one of which is used to update the feature representation of each AU, and the other is used to update the AU classifier; The AU classifier is trained using a multi-label classification loss function; Get the final AU classifier for facial micro-expression recognition; The method for extracting AU multi-scale local features is as follows: Using a multi-scale local area feature extraction algorithm, a multi-scale window, that is, an activation template, is predefined according to the size of the AU activation area. The activation template of each AU is quite different. When each AU is activated, the pre-defined activation template is used and mapped to the final feature layer extracted by the convolutional neural network to obtain the AU multi-scale local features. Use RoIAlign to generate fixed-size features and unify the sizes of AU features in different regions; The paired AU features of uniform size are fused through average pooling operation, and the 64-dimensional AU features are obtained through dimensionality reduction through fully connected mapping.

2. The method for facial micro-expression recognition based on graph convolutional network according to claim 1, wherein: The method to construct the AU correlation matrix is: The correlation matrix A between AUs is defined by the frequency of simultaneous occurrence between AUs and their total frequency of occurrence: A i，j =Q i，j / N i , where A i,j is the correlation matrix when the i-th AU and the j-th AU appear in a face image at the same time, Q i,j is the frequency of simultaneous occurrence of the i-th AU and the j-th AU, N i is the frequency of occurrence of the i-th AU.

3. The method for facial micro-expression recognition based on graph convolutional network according to claim 1, wherein: The graph convolutional network layer used to update the AU feature representation is: Among them, H (l) is the node representation of the lth layer, H (l+1) is the node representation of the l+1th layer, D is the dimension reduction matrix of the association matrix A, defined as D = ∑ j (A+I) ij ; Since A is a square matrix, the dimensions of the node input and output remain unchanged, σ is the activation function leakyReLU, I is the identity matrix, and W is the parameter in the graph neural network.

4. The method for facial micro-expression recognition based on graph convolutional network according to claim 1, wherein: When AU feature representation propagates AU correlation, the local multi-scale features extracted for each AU are input; When propagating AU correlation in the AU classifier, the pre-trained classifier is input and then the graph convolutional network is used to iteratively update the AU classifier.

5. The method for facial micro-expression recognition based on graph convolutional network according to claim 1, wherein: The loss function is: in is the predicted value of the algorithm, and is the AU feature representation and classifier generated after the graph convolutional network; ω i is the weight, y i is the true value, is the predicted value.

6. A facial micro-expression recognition system based on graph convolutional network, characterized in that: The invention comprises a facial image acquisition unit and a processing unit, wherein the facial image acquisition unit is used to acquire a facial image to be identified, an output end of the facial image acquisition unit is connected to the processing unit, and the processing unit executes the method according to one of claims 1 to 5 to perform facial micro-expression recognition.

7. A computer medium, characterized in that The computer medium stores a program for executing the method according to any one of claims 1 to 5.

Images