A micro-expression recognition method and system based on dual feature fusion

Abstract

The present invention discloses a micro-expression recognition method and system based on dual feature fusion, including obtaining micro-expression data for training; using a micro-expression image feature extractor to obtain micro-expression single-frame features; calculating the gradient features between the expression occurrence frame features and the peak frame features; calculating the motion features between the expression occurrence frame and the peak frame; obtaining the gradient features and motion features after weighted fusion as complete micro-expression features; using a classifier to input the complete micro-expression features into the classifier to predict the category to which they belong. The present invention considers two complementary features, uses a gradient feature learning module to calculate micro-expression gradient features, and uses a motion feature learning module to calculate micro-expression motion features. A complete feature is obtained by combining a learnable fusion coefficient. Finally, the feature is input into the classifier for emotion recognition.

Description

Technical Field

[0001] The present invention belongs to the field of micro-expression recognition, and in particular relates to a micro-expression recognition method and system based on dual-feature fusion. Background Art

[0002] Micro-expression recognition plays an indispensable role in affective computing, which aims to recognize low-intensity, short-duration spontaneous expressions through computer technology. Despite decades of development, the difficulty of feature learning and data scarcity still limit its progress due to the natural characteristics of micro-expressions. Traditional hand-crafted features such as optical flow (Liong ST, See J, Wong KS, et al. Less is more: Micro-expression recognition from video using apex frame [J]. Signal Processing: Image Communication, 2018, 62: 82-92.) and local binary patterns (Pfister T, Li X, Zhao G, et al. Recognising spontaneous facial micro-expressions [C] / / 2011 international conference on computer vision. IEEE, 2011: 1449-1456.) are widely used in micro-expression recognition, but these general features are not specifically designed for micro-expressions, so they are easily disturbed by noise information and lack sufficient robustness to express the motion changes of micro-expressions. Recent studies have attempted to use deep learning methods to obtain motion representations of micro-expressions from key frames (occurrence frames and peak frames).

[0003] Due to the characteristics of micro-expressions, true emotions are extremely hidden and usually accompanied by small changes. Most studies emphasize the importance of local information, but excessive focus on significant changes may cause the model to ignore other important information. For example, the model may pay more attention to large movements of the corners of the mouth and ignore small changes in the eye area. In addition, due to the difficulty in collecting micro-expression data and insufficient labeled data, it is difficult to train a fully supervised deep model. Although self-supervised learning methods can utilize unlabeled data, current methods also focus on local areas and lack a complete representation of micro-expressions. Summary of the invention

[0004] In view of the above problems, the present invention proposes a micro-expression recognition method and system based on dual feature fusion, which extracts the occurrence frame and peak frame features through a micro-expression image feature extractor. The present invention considers two complementary features, uses a gradient feature learning module to calculate micro-expression gradient features, uses a motion feature learning module to calculate micro-expression motion features, combines a learnable fusion coefficient to obtain complete features, and finally inputs the features into a classifier for emotion recognition.

[0005] The present invention is achieved by at least one of the following technical solutions.

[0006] A micro-expression recognition method based on dual feature fusion comprises the following steps:

[0007] S1, obtain micro-expression data for training;

[0008] S2, using a micro-expression image feature extractor to obtain micro-expression single-frame features;

[0009] S3, obtaining the gradient feature between the expression occurrence frame feature and the peak frame feature;

[0010] S4, obtaining motion features between the expression occurrence frame and the peak frame;

[0011] S5, obtaining the weighted fused gradient features and motion features as complete micro-expression features, and defining a joint target loss function to enhance the complementarity of the gradient features and motion features;

[0012] S6. Input the complete micro-expression features into the micro-expression classifier to predict the category.

[0013] Furthermore, the micro-expression data includes expression occurrence frames and peak frames.

[0014] Furthermore, in step S1, the micro-expression data used for training includes a complete segment including n -1 frame of micro expression video , defined as , the valid micro-expression data is: ,in is the first frame of the video, is the frame where the expression occurs, is the expression peak frame, The last frame of the video.

[0015] Furthermore, in step S2, the micro-expression image feature extractor includes multiple single-layer residual convolution operations to convert the effective micro-expression data As the input of the feature extractor, single-frame features are obtained for subsequent feature learning; The frame where the expression occurs. is the expression peak frame;

[0016] The single-layer residual convolution operation includes a convolution layer and an aggregation layer combined by skip connections;

[0017] Use multi-layer residual convolution operations to learn single-frame features: , the calculation formula is ,in is a feature collection of effective micro-expression data, is the expression frame feature, is the expression peak frame feature.

[0018] Furthermore, in step S3, the gradient feature between the expression occurrence frame feature and the peak frame feature is obtained, specifically:

[0019] S31, respectively, using embedded functions , , Get the occurrence frame characteristics of the unified space , peak frame features And fusion frame features :

[0020]

[0021] in, is the expression frame feature, is the expression peak frame feature, To fuse frame features, three embedding functions , , Linear Rectification Unit As activation functions, we set , , is the learning parameter;

[0022] S32, convolution operation based on Sobel operator, obtains horizontal gradient features from fused frame features through convolution operation And the vertical gradient features :

[0023]

[0024] in Indicates The fused frame features of channels, N Indicates the number of channels of this feature, and learns horizontal and vertical gradients based on the fused frame features;

[0025] S33, subtract the phases and divide by the time difference to obtain the gradient feature in the time direction :

[0026] ,

[0027] in is the characteristic difference obtained by element-by-element subtraction, i is the time difference;

[0028] S34, channel stitching operation combines the gradient features in the horizontal, vertical and time directions to preserve the complete gradient information :

[0029] ;

[0030] S35, the fusion layer consists of a multi-head self-attention function, a multi-layer perceptron function, and a layer normalization function combined with a residual jump connection, using gradient information to learn gradient features :

[0031] ,

[0032] in, is the multi-head self-attention function, is the multi-layer perceptron function, is the layer normalization function;

[0033] Multi-head self-attention function in the fusion layer Contains multiple head attentions, each head attention is calculated about the query ,key Sum The self-attention function is obtained by normalizing the exponential function to obtain the query and key After the relationship with the value After splicing all the self-attention heads, further learning can obtain multi-head self-attention , the calculation formula is:

[0034] ,

[0035] ,

[0036] in, is the function of self-attention, is the normalized exponential function, is a learnable parameter, is the gradient feature, For the j Head attention, h is the number of head attentions, j ∈ h , d is the feature dimension of each head attention, It is a splicing operation. For multi-head self-attention.

[0037] Further, in step S4, the motion features between the expression occurrence frame and the peak frame are obtained, specifically:

[0038] S41, respectively, using embedded functions , , Get the occurrence frame characteristics of the unified space , peak frame features And fusion frame features , the calculation formula is:

[0039]

[0040] in, is the expression frame feature, is the expression peak frame feature, To fuse frame features, three embedding functions , , Both use the linear rectifier unit ReLU as the activation function and set , , is the learning parameter;

[0041] S42, bitwise subtraction operation Get time features , the calculation formula is:

[0042] ;

[0043] S43, motion layer includes multi-head cross-attention function , Multilayer Perceptron Function and layer normalization function , input spatial features and time characteristics Learning motion features , the calculation formula is:

[0044] ;

[0045] S44. Use temporal features as values ​​or keys and spatial features as queries.

[0046] Furthermore, in step S41, the original features of the image obtained by the feature extractor are used to supplement the overall features from different angles.

[0047] Further, in step S5, the learnable parameters are obtained The weighted fusion of gradient features and motion features is used as the complete micro-expression feature. The calculation formula is:

[0048] ,

[0049] in, For complete micro-expression features, is the gradient feature, For sports characteristics.

[0050] Furthermore, in step S5, the joint objective loss function includes the standard cross entropy loss of the gradient feature , standard cross entropy loss of motion features and orthogonality constraint loss , where the standard cross entropy loss of gradient features and the standard cross entropy loss of motion features correspond to the classification of two classifiers with the same structure but without sharing weights;

[0051] The standard cross entropy loss of the gradient feature and the standard cross entropy loss of motion features as follows:

[0052] ,

[0053] ,

[0054] in is the gradient feature, For sports characteristics, is the mathematical expectation, is the total number of categories, is the traversal value, For the category, represents the indicator function, when When is 1, is 0;

[0055] Orthogonality Constraint Loss Used to promote the orthogonality of gradient features and motion features:

[0056] ,

[0057] in, express Norm.

[0058] A system for implementing the micro-expression recognition method based on dual-feature fusion includes:

[0059] Micro-expression image feature extractor, used to obtain micro-expression single-frame features;

[0060] Gradient feature learning module, used to obtain the gradient features between the expression occurrence frame features and the peak frame features;

[0061] Motion feature learning module, used to obtain motion features between expression occurrence frame and peak frame;

[0062] The fusion coefficient learning module is used to obtain the weighted fusion of gradient features and motion features as complete micro-expression features;

[0063] The classifier is used to input the complete micro-expression features into the micro-expression classifier to predict the category to which it belongs.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. Using gradient features and motion features as complementary features can more comprehensively capture the dynamic information of micro-expressions. Gradient features emphasize local subtle changes, while motion features focus on motion changes between frames. The combination of the two can provide more accurate micro-expression representation. This method is not only suitable for accurate recognition of micro-expressions, but can also be widely used in multiple fields such as emotional computing, mental health monitoring, and emotional analysis under high-pressure environments, and has strong practical application value.

[0066] 2. It is proposed to construct a multi-layer residual convolution feature extraction network to extract single-frame features of micro-expressions frame by frame. The features extracted by the residual convolution operation of a single frame not only retain the local details of micro-expressions, but also significantly improve the accuracy and efficiency of feature extraction.

[0067] 3. A gradient feature learning module and a motion feature learning module are proposed to obtain complementary features. Two feature modules are designed to obtain two complementary features to jointly characterize micro-expressions. Gradient features can obtain areas containing changes and ignore irrelevant areas of the face. Motion features can identify areas with rich expression information. By combining gradient features with motion features, the model is more robust to micro-expression recognition in different scenarios and can maintain stable recognition performance under a variety of complex conditions.

[0068] 4. A fusion coefficient learning module is proposed to perform weighted fusion of gradient features and motion features, and the effect of feature combination is optimized using learnable coefficients, thereby improving the overall feature expression ability of micro-expressions and ensuring more accurate emotion classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A structural diagram of a micro-expression recognition system based on dual feature fusion according to an embodiment of the present invention;

[0070] Figure 2 Schematic diagram of the micro-expression image feature extractor of the embodiment;

[0071] Figure 3 This is a schematic diagram of the gradient feature learning module of the embodiment;

[0072] Figure 4 This is a diagram of the motion feature learning module of the embodiment. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0074] like Figure 1 As shown, a micro-expression recognition system based on dual feature fusion in this embodiment includes a micro-expression image feature extractor, a gradient feature learning module, a motion feature learning module, a fusion coefficient learning module and a feature classifier. The micro-expression image feature extractor is used to obtain the single-frame features of micro-expressions; the gradient feature learning module is used to obtain the gradient features between the expression occurrence frame features and the peak frame features; the motion feature learning module is used to obtain the motion features between the expression occurrence frame and the peak frame; the fusion coefficient learning module is used to obtain the gradient features and motion features after weighted fusion as complete micro-expression features; the classifier is used to input the complete micro-expression features into the classifier to predict the category to which it belongs. First, the occurrence frame and peak frame of the expression are obtained through data processing. The occurrence frame and peak frame features are extracted by the micro-expression image feature extractor, the gradient feature learning module is used to calculate the micro-expression gradient features, and the motion feature learning module is used to calculate the micro-expression motion features. The complete features are obtained by combining the learnable fusion coefficients. Finally, the features are input into the classifier for emotion recognition.

[0075] like Figure 1~Figure 4 As shown, a micro-expression recognition method based on dual feature fusion in this embodiment includes the following steps:

[0076] S1. Acquire micro-expression data for training; acquire facial micro-expression data for training through three spontaneous micro-expression datasets: CASMEII, SAMM, and SMIC, wherein the micro-expression data includes expression occurrence frames and peak frames.

[0077] The micro-expression data used for training includes a complete n -1 frame of micro expression video , defined as , the valid micro-expression data is: ,in is the first frame of the video, The frame where the expression occurs. is the expression peak frame, The last frame of the video. The emotion categories of the micro-expression data in this embodiment include positive, negative and surprised.

[0078] For unlabeled videos and open-world data, select As the occurrence frame, the middle frame is selected as the peak frame.

[0079] S2. Use the micro-expression image feature extractor to obtain the micro-expression single-frame features, learn the micro-expression data frame by frame, and use two-dimensional image convolution (such as the convolution kernel size is 3×3 or 5×5, and the step size is 1 or 2) to obtain the micro-expression single-frame features, such as Figure 2 As shown, specifically:

[0080] The feature extractor contains multiple single-layer residual convolution operations to transform the effective micro-expression data Obtain single-frame features as input to the feature extractor for subsequent feature learning;

[0081] The single-layer residual convolution operation includes a convolution layer and an aggregation layer combined by skip connections;

[0082] Use multi-layer residual convolution operations to learn single-frame features: , the calculation formula is ,in is a feature collection of effective micro-expression data, is the expression frame feature, is the expression peak frame feature.

[0083] The present invention constructs a multi-layer residual convolution feature extraction network to extract single-frame features of micro-expressions frame by frame. By extracting image features through a two-dimensional network, there is no need to use a three-dimensional network to learn multi-frame features, which effectively improves the learning efficiency of micro-expression features.

[0084] As an embodiment, the feature extractor can select different feature extraction networks according to different data sets, such as VGG network and AlexNet network.

[0085] S3, the gradient feature learning module obtains the gradient features between the expression occurrence frame features and the peak frame features, such as Figure 3 As shown, specifically:

[0086] S31, respectively, using embedded functions , , Get the occurrence frame characteristics of the unified space , peak frame features And fusion frame features :

[0087]

[0088] in, is the expression frame feature, is the expression peak frame feature, To fuse frame features, three embedding functions , , Linear Rectification Unit As activation functions, we set , , is the learning parameter;

[0089] S32, convolution operation based on Sobel operator, obtains horizontal and vertical gradient features from fused frame features through convolution operation:

[0090]

[0091] in Indicates Channel fusion frame features, N represents the number of channels of the feature. Based on the fusion frame learning horizontal and vertical gradients, the complete spatial features are more suitable for learning spatial gradients;

[0092] This operation is suitable for analyzing subtle movement changes in micro-expressions, and is particularly effective for detecting rapid changes in expression in scenarios such as psychological analysis or security monitoring.

[0093] S33, subtract the phases and divide by the time difference to obtain the gradient feature in the time direction:

[0094] ,

[0095] The numerator is the characteristic difference obtained by element-by-element subtraction, and the denominator i is the time difference;

[0096] S34, channel stitching operation combines the gradient features in the horizontal, vertical and time directions to preserve the complete gradient information:

[0097] ;

[0098] S35, the fusion layer consists of a multi-head self-attention function, a multi-layer perceptron function, and a layer normalization function combined with a residual jump connection, using gradient information to learn gradient features :

[0099] ,

[0100] in, is the multi-head self-attention function, is the multi-layer perceptron function, is the layer normalization function;

[0101] Multi-head self-attention function in the fusion layer Contains multiple head attentions, each head attention is calculated about the query ,key Sum The self-attention function is obtained by normalizing the exponential function to obtain the query and key After the relationship with the value After splicing all the self-attention heads, further learning can obtain multi-head self-attention , the calculation formula is:

[0102] ,

[0103] ,

[0104] in, is the function of self-attention, is the normalized exponential function, is a learnable parameter, is the gradient feature, For the j Head attention, h is the number of head attentions, j ∈ h , d is the feature dimension of each head attention, It is a splicing operation. For multi-head self-attention.

[0105] The default number of attention heads is 8, but it can be adjusted appropriately according to different application scenarios. For example, when processing high-resolution micro-expression videos, the number of attention heads can be increased to 16 to enhance feature extraction capabilities; in resource-constrained embedded devices or scenarios with high real-time requirements, the number of attention heads can be reduced to 4 to balance computational efficiency and accuracy.

[0106] S4, the motion feature learning module obtains the motion features between the expression occurrence frame and the peak frame, such as Figure 4 As shown, specifically:

[0107] S41, respectively, using embedded functions , , Get the occurrence frame characteristics of the unified space , peak frame features And fusion frame features , the calculation formula is:

[0108]

[0109] in, is the expression frame feature, is the expression peak frame feature, Three embedding functions for fusion frame features , , Both use the linear rectifier unit ReLU as the activation function and set , , To learn the parameters.

[0110] Instead of using gradient information, the original features of the image obtained by the feature extractor are used to supplement the overall features from different perspectives. For example, in medical image analysis, the extraction of original features such as texture or brightness can assist in detecting abnormal micro-expressions, while in public security monitoring, the supplementation of original features helps improve recognition accuracy under low light or complex background conditions. According to different application scenarios, the weights and processing methods of the original features can be flexibly adjusted, such as enhancing high-frequency or low-frequency features to meet specific task requirements.

[0111] S42, perform a bitwise subtraction operation to obtain time characteristics, and the calculation formula is:

[0112] ;

[0113] In order to obtain complete temporal information, unlike the gradient feature learning module, there is no division by the time difference.

[0114] S43, motion layer includes multi-head cross-attention function , Multilayer Perceptron Function and layer normalization function , input spatial features and time characteristics Learning motion features , the calculation formula is:

[0115] .

[0116] S44, the multi-head cross-attention function has a similar implementation to the multi-head self-attention function, except that it has a different query key-value triple. Since the core of learning motion features is temporal information, temporal features are used as values ​​or keys and spatial features as queries. This emphasizes the intrinsic correlation of spatiotemporal features to guide temporal features.

[0117] The gradient feature learning module and the motion feature learning module extract complementary features to jointly characterize micro-expressions. The gradient feature module can accurately capture the changing areas of facial expressions and effectively ignore irrelevant areas; while the motion feature module focuses on identifying dynamic areas containing rich information. By combining these two features, the model shows higher robustness in different scenarios and can stably and efficiently perform micro-expression recognition in complex environments.

[0118] S5, fusion coefficient learning module obtains the weighted fusion of gradient features and motion features as complete micro-expression features, fusion coefficient learning module, by introducing learnable coefficients Optimizing feature combinations effectively enhances the expressive power of the overall features of micro-expressions. This technical approach significantly improves the accuracy of emotion classification. The calculation formula is:

[0119] ,

[0120] in, For complete micro-expression features, is the gradient feature, For sports characteristics.

[0121] As another embodiment, the feature fusion method can be replaced by other methods. The feature fusion method can be implemented not only by a learnable weighted fusion method, but also by feature concatenation, feature alignment addition, model fusion, and other methods.

[0122] In order to ensure the complementarity of the two features, the standard cross entropy loss of the joint gradient feature is defined , standard cross entropy loss of motion features and orthogonality constraint loss The objective loss function is , where the two standard cross entropy losses correspond to the classification of the two classifiers:

[0123] ,

[0124] ,

[0125] in, is the mathematical expectation, is the total number of categories, is the traversal value, For the category, represents the indicator function, when When is 1, is 0.

[0126] The two classifiers are gradient feature classifier and motion feature classifier respectively; the two classifiers have the same structure but do not share weight parameters, and both are classifiers based on fully connected layers to improve the classification accuracy of the classifiers.

[0127] The orthogonality constraint loss Used to promote the orthogonality of gradient features and motion features:

[0128] ,

[0129] in, is the gradient feature, For sports characteristics, express norm; by properly adjusting the constraint parameters, better recognition effects can be achieved on different datasets (such as CASME II, SAMM, and SMIC).

[0130] S6. Input the complete micro-expression features into the micro-expression classifier to predict the category, which includes positive, negative and surprise.

[0131] The gradient feature classifier, motion feature classifier, and micro-expression classifier of this embodiment have the same structure, and are all classifiers based on a fully connected layer.

[0132] As shown in Table 1, the recognition accuracy of the method of the present invention was compared with other methods in Table 1 by testing in three public micro-expression datasets (Chinese Academy of Sciences micro-expression dataset CASME II, spontaneous micro-expression dataset SMIC, and spontaneous micro-facial motion dataset SAMM). In general, the recognition accuracy of the method of the present invention tested on the Chinese Academy of Sciences micro-expression dataset CASME II is the highest, which is 1.15% higher than the highest one among other methods (feature representation learning FRL-DGT based on adaptive displacement generation); the recognition accuracy of the method of the present invention tested on the spontaneous micro-expression dataset SMIC is the highest, which is 2.16% higher than the highest one among other methods (motion feature extraction MoExt method); the recognition accuracy of the method of the present invention tested on the spontaneous micro-facial motion dataset SAMM is also the highest, which is 3.79% higher than the highest one among other methods (motion feature extraction MoExt method); in the average recognition accuracy measured in the three public micro-expression datasets in Table 1, the method of the present invention is 2.76% higher than the highest average recognition accuracy among other methods (motion feature extraction MoExt).

[0133] Table 1 Comparison of recognition performance between the method of the present invention and other methods

[0134]

[0135] The method of the present invention applies the comprehensive analysis of gradient features and motion features to micro-expression recognition. Through this method, the subtle changes of micro-expressions can be captured more accurately, and the recognition results can be optimized by fusing features, thereby improving the accuracy and robustness of micro-expression classification.

[0136] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A micro-expression recognition method based on dual feature fusion, characterized in that: The following steps are involved: S1, obtain micro-expression data for training; S2, using a micro-expression image feature extractor to obtain micro-expression single-frame features; S3, obtaining the gradient feature between the expression occurrence frame feature and the peak frame feature; S4, obtaining motion features between the expression occurrence frame and the peak frame; S5, obtaining the weighted fused gradient features and motion features as complete micro-expression features, and defining a joint target loss function to enhance the complementarity of the gradient features and motion features; S6. Input the complete micro-expression features into the micro-expression classifier to predict the category.

2. A micro-expression recognition method based on dual feature fusion according to claim 1, characterized in that: The micro-expression data includes expression occurrence frames and peak frames.

3. The micro-expression recognition method based on dual feature fusion according to claim 1, characterized in that: In step S1, the micro-expression data used for training includes a complete segment including n -1 frame of micro expression video , defined as , the valid micro-expression data is: ,in is the first frame of the video, is the frame where the expression occurs, is the expression peak frame, The last frame of the video.

4. The micro-expression recognition method based on dual feature fusion according to claim 1, characterized in that: In step S2, the micro-expression image feature extractor includes multiple single-layer residual convolution operations to convert the effective micro-expression data Obtain single-frame features as input to the feature extractor for subsequent feature learning; in The frame where the expression occurs. is the expression peak frame; The single-layer residual convolution operation includes a convolution layer and an aggregation layer combined by skip connections; Use multi-layer residual convolution operations to learn single-frame features: , the calculation formula is ,in is a feature collection of effective micro-expression data, is the expression frame feature, is the expression peak frame feature.

5. The micro-expression recognition method based on dual feature fusion according to claim 1, characterized in that: In step S3, the gradient feature between the expression occurrence frame feature and the peak frame feature is obtained, specifically: S31, respectively, using embedded functions , , Get the occurrence frame characteristics of the unified space , peak frame features And fusion frame features : in, is the expression frame feature, is the expression peak frame feature, To fuse frame features, three embedding functions , , Linear Rectification Unit As activation functions, we set , , is the learning parameter; S32, convolution operation based on Sobel operator, obtains horizontal gradient features from fused frame features through convolution operation And the vertical gradient features : ， in Indicates The fused frame features of channels, N Indicates the number of channels of this feature, and learns horizontal and vertical gradients based on the fused frame features; S33, subtract the phases and divide by the time difference to obtain the gradient feature in the time direction : ， in is the characteristic difference obtained by element-by-element subtraction, i is the time difference; S34, channel stitching operation combines the gradient features in the horizontal, vertical and time directions to preserve the complete gradient information : ； S35, the fusion layer consists of a multi-head self-attention function, a multi-layer perceptron function, and a layer normalization function combined with a residual jump connection, using gradient information to learn gradient features : ， in, is the multi-head self-attention function, is the multi-layer perceptron function, is the layer normalization function; Multi-head self-attention function in the fusion layer Contains multiple head attentions, each head attention is calculated about the query ,key Sum The self-attention function is obtained by normalizing the exponential function to obtain the query and key After the relationship with the value After splicing all the self-attention heads, further learning can obtain multi-head self-attention , the calculation formula is: ， ， in, is the function of self-attention, is the normalized exponential function, is a learnable parameter, is the gradient feature, For the j Head attention, h is the number of head attentions, j ∈ h , d is the feature dimension of each head attention, It is a splicing operation. For multi-head self-attention.

6. The micro-expression recognition method based on dual feature fusion according to claim 1, characterized in that: In step S4, the motion features between the expression occurrence frame and the peak frame are obtained, specifically: S41, respectively, using embedded functions , , Get the occurrence frame characteristics of the unified space , peak frame features And fusion frame features , the calculation formula is: in, is the expression frame feature, is the expression peak frame feature, To fuse frame features, three embedding functions , , Both use the linear rectifier unit ReLU as the activation function and set , , is the learning parameter; S42, bitwise subtraction operation Get time features , the calculation formula is: ； S43, motion layer includes multi-head cross-attention function , Multilayer Perceptron Function and layer normalization function , input spatial features and time characteristics Learning motion features , the calculation formula is: ， in, is the multi-head cross-attention function, is the multilayer perceptron function, is the layer normalization function; S44. Use temporal features as values ​​or keys and spatial features as queries.

7. The micro-expression recognition method based on dual feature fusion according to claim 6, characterized in that: In step S41, the original features of the image obtained by the feature extractor are used to supplement the overall features from different angles.

8. The micro-expression recognition method based on dual feature fusion according to claim 1, characterized in that: In step S5, the learnable parameters are obtained The weighted fusion of gradient features and motion features is used as the complete micro-expression feature. The calculation formula is: ， in, For complete micro-expression features, is the gradient feature, For sports characteristics.

9. The micro-expression recognition method based on dual feature fusion according to claim 1, characterized in that: In step S5, the joint objective loss function includes the standard cross entropy loss of the gradient feature , standard cross entropy loss of motion features and orthogonality constraint loss , where the standard cross entropy loss of gradient features and the standard cross entropy loss of motion features correspond to the classification of two classifiers with the same structure but without sharing weights; The standard cross entropy loss of the gradient feature and the standard cross entropy loss of motion features as follows: ， ， in is the gradient feature, For sports characteristics, is the mathematical expectation, is the total number of categories, is the traversal value, For the category, represents the indicator function, when When is 1, is 0; Orthogonality Constraint Loss Used to promote the orthogonality of gradient features and motion features: ， in, express Norm.

10. A system for implementing a micro-expression recognition method based on dual feature fusion as described in any one of claims 1 to 9, characterized in that: include: Micro-expression image feature extractor, used to obtain micro-expression single-frame features; Gradient feature learning module, used to obtain the gradient features between the expression occurrence frame features and the peak frame features; Motion feature learning module, used to obtain motion features between expression occurrence frame and peak frame; The fusion coefficient learning module is used to obtain the weighted fusion of gradient features and motion features as complete micro-expression features; The classifier is used to input the complete micro-expression features into the micro-expression classifier to predict the category to which it belongs.

Images