A micro-expression recognition method based on transfer learning

Abstract

The present invention provides a micro-expression recognition method based on transfer learning, comprising preprocessing facial image data and the following steps: extracting the starting frame and apex frame of a micro-expression, and extracting half of the starting frame and apex frame of a macro-expression; performing facial detection on the extracted images, extracting optical flow features from the two frames as input; and extracting features using a ResNet network, dividing the extracted features into upper and lower parts. After average pooling, the extracted features are fed into a fully connected layer for recognition, and the extracted features are recognized and output. The present invention proposes a micro-expression recognition method based on transfer learning. The micro-expression recognition network structure adopts an improved ResNet18. In the sample preprocessing portion, the expression motion variation amplitudes of three micro-expression datasets, CASMEII, SAMM, and SMIC, are balanced to balance the sample distribution. The extracted optical flow features are aligned to reduce noise interference and balance the noise interference on the optical flow features.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a micro-expression recognition method based on transfer learning. Background Art

[0002] Facial expressions are a crucial form of communication. Transmitting information through facial expressions is a common occurrence in everyday life, and in certain situations, facial expressions can convey more information than language. Psychologically, when faced with unfavorable circumstances, people often manipulate facial expressions to deceive. Current psychological research suggests that when people attempt to suppress their emotions, they involuntarily display microexpressions. Microexpressions were first discovered in 1966. Three years later, Ekman et al. coined the term while analyzing a video of a patient interview. Microexpressions typically fluctuate uncontrollably between 1 / 25 and 1 / 2 seconds and occur infrequently. Untrained individuals are not very good at recognizing them, and the results reported by different researchers vary widely. Subsequently, Ekman and Friesen developed the Brief Affect Recognition Test (BART) in 1979. In subsequent experiments, they found a positive correlation between participants' ability to recognize microexpressions and their ability to detect lies. Ekman then conducted a brief facial expression recognition test, which also confirmed a positive correlation between participants' micro-expression recognition ability and lie detection ability, demonstrating that micro-expressions can effectively help people detect lies. When people express their inner states and psychological needs, they produce a large number of facial expressions. However, because micro-expressions are too brief to be easily detected by the human eye, computers can be used to address this problem.

[0003] Currently, algorithms used for micro-expression recognition fall into two main categories: traditional manual feature extraction for classification and deep learning-based feature extraction for classification. The key to feature extraction using deep learning methods lies in extracting effective features and preserving facial motion information with minimal loss. Current strategies include amplifying motion amplitude, using extracted optical flow features as input to preserve motion information, and using convolutional neural networks (CNNs) combined with long short-term memory (LSTM) networks to recognize video frames. While these strategies vary, their structures generally follow the same process: data preprocessing, feature extraction, and classification based on these features. However, imbalances in the training process can prevent the models from achieving optimal results. This imbalance arises from two aspects: an imbalance in the feature map hierarchy and an imbalance in the distribution of facial motion amplitude. These imbalances are caused by underutilization of the extracted features and a single amplification factor for facial amplitude. Furthermore, extracting optical flow features as network input can preserve motion information. However, these methods are sensitive to facial alignment and environmental changes, making them susceptible to interference. To effectively address the aforementioned issues, we propose an Adaptive Balanced Amplification Factor (ABAF) network based on a non-local attention mechanism. This is a simple yet effective micro-expression recognition model that explicitly addresses the three aforementioned issues. This model embodies three innovative aspects: We employ an adaptive balanced amplification factor (ABAF) to perform Eulerian video magnification (EVM) on micro-expressions. Based on the magnitude of facial changes, the amplification factor is adjusted to make the changes more similar and conform to a Gaussian distribution, addressing the imbalance in the distribution of facial movement changes. In the network, high-level features contain richer semantic information but coarser details, while low-level features contain less semantic information but more details.

[0004] On May 26, 2021, Xiao Ruixue's master's thesis at Shandong University, titled "Research on Micro-Expression Recognition Methods Based on Macro-Expression Transfer Learning," was published on the Doc88 website. The thesis proposes a micro-expression recognition model based on joint representation learning, which uses macro-expressions that share certain commonalities with micro-expression data to learn the common subspace where the difference between macro-expressions and micro-expressions is minimal. A micro-expression recognition model based on joint non-negative matrix factorization with dual graph regularization was also proposed. This model also incorporates information-rich macro-expression knowledge to identify common emotional factors within the macro-expression and micro-expression data themselves. The disadvantage of this method is that it fails to pay attention to the macro-expression motion amplitude after transferring the macro-expression, resulting in a large difference in motion amplitude between the source and target domains. Summary of the Invention

[0005] To address the aforementioned technical issues, this paper proposes a micro-expression recognition method based on transfer learning. The micro-expression recognition network architecture utilizes a modified ResNet-18. During sample preprocessing, the amplitude of facial expression movements across three micro-expression datasets, CASMEII, SAMM, and SMIC, is balanced to achieve a balanced sample distribution. The extracted optical flow features are aligned to reduce noise interference and balance the noise impact on the optical flow features. Feature fusion is performed on each layer to balance features, and a non-local attention mechanism module is added for feature enhancement. After feature extraction using ResNet-18, the feature maps of each layer are expanded to 512 channels using 1×1 convolutions. The feature maps of each layer are then pooled to 7×7 and flattened after fusion.

[0006] The present invention provides a micro-expression recognition method based on transfer learning, which includes preprocessing facial image data and further comprising the following steps:

[0007] Step 1: Extract the starting frame and vertex frame of the micro-expression respectively, and extract half of the frame between the starting frame and vertex frame of the macro-expression at the same time;

[0008] Step 2: Perform face detection on the extracted image and extract optical flow features from the two frames as input to the ResNet18 network.

[0009] Step 3: Use the ResNet network to extract features from the optical flow map, divide the extracted features into two parts, and send them to the fully connected layer for recognition after average pooling. The extracted features are recognized and output.

[0010] Preferably, the step of pre-processing the facial image data further includes amplifying the facial movement amplitude using an Euler video amplification algorithm so that the amplitude of micro-expression movement is increased.

[0011] In any of the above schemes, preferably, the preprocessing includes taking the maximum value of the optical flow between the starting frame and the vertex frame in the macro expression sample as the motion amplitude of the expression, and using a dichotomy method to find the motion amplitude close to the average value of the micro expression sample after amplification.

[0012] In any of the above solutions, preferably, step 2 includes using an adaptive amplification coefficient method to balance the expression variation range of different samples between data sets.

[0013] In any of the above solutions, preferably, the adaptive amplification factor method includes the following sub-steps:

[0014] Step 21: Use the optical flow method to extract the optical flow features of the sample start frame and vertex frame in the source data set;

[0015] Step 22: Calculate the maximum value of the optical flow as the motion amplitude of the sample in the source data set;

[0016] Step 23: Calculate the amplification factor based on the motion amplitude.

[0017] In any of the above schemes, it is preferred that when the sample motion amplitude is less than 1, a larger magnification factor is required; when the sample motion amplitude is greater than 8, no amplification is required; the magnification factor P of the i-th sample is i Defined as

[0018]

[0019] Among them, m i is the original facial motion amplitude, A, B, C and M are constant coefficients.

[0020] In any of the above solutions, preferably, step 3 includes adding a non-local module after the feature fusion of each layer of the ResNet network, and the non-local method is

[0021]

[0022] Among them, y i is the output signal at position i, x i is the input signal at position i, x j is the input signal at position j, f(x i ,x j ) represents the relationship between position i and all possible associated positions j, C(x) is the normalization parameter, For any other position in the feature map, g(x j ) is a linear transfer function.

[0023] In any of the above solutions, it is preferred that the relationship f(x i ,x j ) is calculated as

[0024]

[0025] Among them, [θ(x i ),φ(x j )] represents θ(x i ) and φ(x j ) cascade, θ(x i )=W θ x i ,φ(x j )=W φ x j , is the weight vector that projects the concatenated vector to a scalar, W θ is the weight parameter θ, W φ is the weight parameter φ.

[0026] In any of the above solutions, preferably, the linear conversion function g(x j ) is calculated as

[0027] g(x j )=W g x j

[0028] Among them, W g is the weight of the linear transfer function.

[0029] In any of the above schemes, it is preferred that y is made to i and x i The number of channels remains the same, and the sum is obtained

[0030] z i =W z y i +x i

[0031] Among them, z i is the weight sum calculated at point i, W z is the learnable weight matrix.

[0032] This paper proposes a micro-expression recognition method based on transfer learning. Transfer learning from macro-expressions can overcome the overfitting problem caused by insufficient micro-expression datasets. Existing macro-expression and micro-expression datasets, after using transfer learning, can meet experimental requirements.

[0033] The non-local module refers to the non-local enhancement module. Its idea is that the response at a certain pixel point is the sum of the feature weights at all other points, and each point is associated with all other points. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The figure is a flowchart of a preferred embodiment of a micro-expression recognition method based on transfer learning according to the present invention.

[0035] Figure 2 The flowchart of another preferred embodiment of the micro-expression recognition method based on transfer learning according to the present invention.

[0036] Figure 3 2 is a schematic diagram of an embodiment of a non-local network module of a micro-expression recognition method based on transfer learning according to the present invention.

[0037] Figure 4Schematic diagram of semantic feature fusion and enhancement using a local network module according to a preferred embodiment of the micro-expression recognition method based on transfer learning of the present invention.

[0038] Figure 5 Schematic diagram of the structure of a neural network according to an embodiment of the micro-expression recognition method based on transfer learning of the present invention.

[0039] Figure 6 Schematic diagram of an embodiment of micro-expression magnification and macro-expression reduction according to the micro-expression recognition method based on transfer learning of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0041] Example 1

[0042] like Figure 1 As shown, step 100 is executed to preprocess the facial image data and amplify the facial motion amplitude using the Euler video magnification algorithm to increase the motion amplitude of micro-expressions. The preprocessing includes using the maximum optical flow between the starting frame and the vertex frame in the macro-expression sample as the motion amplitude of the expression, and using a dichotomy method to find the motion amplitude close to the average value of the amplified micro-expression sample.

[0043] Step 110 is executed to extract the start frame and the vertex frame of the micro-expression respectively, and simultaneously extract the half frame between the start frame and the vertex frame of the macro-expression.

[0044] Execute step 120 to perform facial detection on the extracted image, extract optical flow features from the two frames, and use them as input to the ResNet18 network. An adaptive magnification coefficient method is used to balance the expression variation of different samples in the dataset. The adaptive magnification coefficient method includes the following sub-steps:

[0045] Execute step 121, extracting optical flow features of the sample start frame and vertex frame in the source data set using the optical flow method;

[0046] Execute step 122 and calculate the maximum value of the optical flow as the motion amplitude of the sample in the source data set;

[0047] Execute step 123 to calculate the amplification factor according to the motion amplitude.

[0048] When the sample motion amplitude is less than 1, a larger magnification factor is required; when the sample motion amplitude is greater than 8, no more amplification is required; the magnification factor P of the i-th sample i Defined as

[0049]

[0050] Among them, m i is the original facial motion amplitude, A, B, C and M are constant coefficients.

[0051] Execute step 130, use the ResNet network to extract features from the optical flow map, divide the extracted features into upper and lower parts, send them to the fully connected layer for recognition after average pooling, and recognize and output the extracted features. After the feature fusion of each layer of the ResNet network, add a non-local module. The non-local method is

[0052]

[0053] Among them, y i is the output signal at position i, x i is the input signal at position i, x j is the input signal at position j, f(x i ,x j ) represents the relationship between position i and all possible associated positions j, C(x) is the normalization parameter, For any other position in the feature map, g(x j ) is a linear transfer function.

[0054] The relationship f(x i ,x j ) is calculated as

[0055]

[0056] Among them, [θ(x i ),φ(x j )] represents θ(x i ) and φ(x j ) cascade, θ(x i )=W θ x i ,φ(x j )=W φ x j , is the weight vector that projects the concatenated vector to a scalar, W θ is the weight parameter θ, W φ is the weight parameter φ. The linear conversion function g(x j ) is calculated as

[0057] g(x j )=W g x j

[0058] Among them, W gis the weight of the linear transfer function.

[0059] Through 1×1 convolution, y i and x i The number of channels remains the same, and the sum is obtained

[0060] z i =W z y i +x i

[0061] Among them, z i is the weight sum calculated at point i, W z is the learnable weight matrix.

[0062] Example 2

[0063] This paper proposes a balanced micro-expression recognition network using a modified ResNet18 architecture. Sample preprocessing balances the amplitude of facial movements across three micro-expression datasets: CASMEII, SAMM, and SMIC, to achieve a balanced sample distribution. Extracted optical flow features are aligned to reduce noise interference and balance the impact of noise on optical flow features. Feature fusion is performed on each layer to balance features, and a non-local attention mechanism module is added for feature enhancement. After feature extraction using ResNet18, the feature maps of each layer are expanded to 512 channels using 1x1 convolutions. The feature maps of each layer are then pooled to 7x7 and averaged after fusion.

[0064] This paper proposes a method to use transfer learning method, integrate dense optical flow algorithm, introduce attention mechanism non_local, to recognize micro-expression of human face in video, and improve the accuracy of algorithm in detecting and recognizing micro-expression changes. The main research contents are as follows: Figure 2 As shown, the following steps are included:

[0065] Step 1: First, the facial image data is preprocessed and the facial movement amplitude is amplified using the Euler Video Magnification (EVM) algorithm to increase the amplitude of micro-expression movement;

[0066] Step 2, then extract the starting frame and vertex frame of the micro-expression respectively, and extract half of the frame between the starting frame and vertex frame of the macro-expression at the same time;

[0067] Step 3: Perform face detection on the extracted image. Perform optical flow feature extraction on the two frames as input.

[0068] Step 4: Use the ResNet network to extract features, divide the extracted features into upper and lower parts, and send them to the fully connected layer for recognition after average pooling. The extracted features are recognized and output. The main contribution of this invention is the design of a new micro-expression recognition method. The network parameters and process are as follows:

[0069] One of the difficulties of using micro-expression datasets compared to macro-expression datasets is the relatively small number of datasets and samples within each dataset. This does not meet the data requirements of deep neural networks and can easily lead to overfitting. Therefore, we employ a cross-domain learning approach to preprocess the SAMM, SMIC, and CASMEII datasets simultaneously. Since the total number of samples in these three datasets is only 442, sample augmentation is necessary. To achieve this, we introduce the macro-expression dataset CK+ and employ a domain adaptation method to reduce the data distribution gap between the source and target domains. The maximum optical flow value between the start and vertex frames of a sample is used as the motion amplitude of the expression. A computationally inefficient binary search method is used to find a motion amplitude close to the average value of the amplified micro-expression sample. This method ensures that the motion amplitude distribution of macro-expressions is similar to that of micro-expressions, resulting in better recognition performance. Furthermore, the small amplitude of micro-expressions, which is difficult to detect with the naked eye, also poses a challenge. During preprocessing of micro-expression datasets, the amplitude of change between adjacent frames is small, making it difficult to extract distinct optical flow features and resulting in data redundancy. To address this issue, our method employs a zoom-in / zoom-out strategy to extract optical flow features from the start and vertex frames, reducing data redundancy. We also use the motion amplification method (MAG), which is widely used in micro-expression recognition tasks, to amplify small movements. We also use the Euler video magnification method (EVM) to amplify micro-expressions. The magnitude of expression changes varies across different datasets, and the distribution of motion amplitudes is uneven.

[0070] Therefore, this method uses an adaptive magnification coefficient method to balance the expression change amplitude of different samples between data sets. First, the optical flow method is used to extract the optical flow features of the starting frame and vertex frame of the sample in the source data set, and the maximum value of the optical flow is calculated as the motion amplitude of the sample in the source data set. The magnification coefficient is calculated based on the motion amplitude. The larger the motion amplitude of the source sample, the smaller the magnification amplitude. When the sample motion amplitude is less than 1, a larger magnification coefficient is required. When the sample motion amplitude is greater than 8, it is no longer magnified. Therefore, the magnification coefficient P of the i-th sample is i Defined as:

[0071]

[0072] in A, B, C, and M are all constant coefficients, derived from prior knowledge.

[0073] The calculated magnification coefficient is adjusted according to different motion amplitudes. The sample distribution after amplification is more balanced, the difference in motion amplitude between data sets is smaller, and the recognition effect is better.

[0074] Non-local network module:

[0075] Non-local network module structure such as Figure 3 As shown in the figure. Due to the small number of samples for micro-expression recognition, a shallow network is needed to extract features to prevent overfitting. This paper uses ResNet18 as the network for feature extraction. When using deep neural networks for micro-expression recognition, the feature map of the last layer is usually used as the basis for classification, while the underlying features are lost, resulting in an imbalance in the feature map hierarchy. Figure 4 As shown in the figure, due to the imbalance of feature map levels in the utilization of high-level and low-level features, in order to utilize features of different resolutions, we adopt a feature fusion strategy and use a non-local attention mechanism to enhance the features of the fused feature map.

[0076] Since convolution is local, the integrated features are not optimal. Figure 5 As shown in Figure 2, in order to make the integrated features more discriminative, ResNet18 is improved by adding a non-local module after fusing the features of each layer of the network. Concatenation non-local attention is used here to improve the integrated features. The non-local method is shown in formula (2):

[0077]

[0078] Where x represents the input signal and y represents the output signal. i ,x j ) represents the relationship between position i and all possible associated positions j. The f value is proportional to the impact of position j on position i, as shown in formula (3):

[0079]

[0080] where [θ(x i ),φ(x j )] represents θ(x i ) and φ(x j ) cascade, θ(x i )=W θ x i ,φ(x j )=W φ x j , Is the weight vector that projects the concatenated vector to a scalar. The g function is a linear transformation as shown in formula (4), W z is a learnable weight matrix, implemented by 1×1 convolution:

[0081] g(x j )=W g x j (4)

[0082] C(x)=N is the normalization parameter, where non-local (2) is implemented using the non-local block.

[0083] In order to i Add and make y i and x i The number of channels remains the same, and the addition results in formula (10), where W z is the learnable weight matrix.

[0084] z i =W z y i +x i (5)

[0085] This residual structure can overcome the limitations of ordinary convolution, so that all feature points that produce positional associations can be utilized. By applying this method to the fused feature map, the enhanced features are more discriminative.

[0086] For the previously proposed balanced magnification coefficient method, after verification and trial, the values ​​were set to -0.0309, -0.0064, 2.1449, and 2.2, respectively. Due to differences in data distribution and environmental background, fine-tuning is required for each dataset, but the results are not significantly affected. Equation (1) can be adjusted to other functions as long as the magnification conditions are met. When the magnification coefficient is greater than 20, it must be limited to 20. Excessive magnification coefficients can easily cause image distortion and affect recognition performance. To reduce the amplitude of macro-expression changes, we use a binary search method to find frames with facial motion amplitudes between 8 and 11 in the video sequence as vertex frames. If there are no frames in the video that meet the conditions, the frame with the motion amplitude closest to 9.5 in the previously found frames is used as the vertex frame, thereby balancing the dataset distribution. For our proposed network, after using the ImageNet pre-trained weights, we use a decay weight of 0.000001, a batch size of 32, and an initial learning rate of 0.001, which is reduced by a factor of 2 every 10 cycles.

[0087] Compared with the existing technology, the present invention has the following advantages:

[0088] 1. This invention uses transfer learning from macroexpressions to compensate for the overfitting problem caused by insufficient microexpression datasets. Existing macroexpression and microexpression datasets, after transfer learning, can meet experimental requirements. Extensive experiments have been conducted on microexpression recognition, and algorithms have been refined.

[0089] 2. The balanced amplification coefficient of the present invention solves the problem that the network cannot efficiently learn the content of the features due to the large difference in data distribution. It balances the influence of interference in the image domain on the generation of optical flow maps. It uses feature fusion to fuse and average the features of each layer, and uses the non-local attention mechanism module to enhance the features, solving the problem of imbalance in the micro-expression feature map layer. The specific structure is as follows Figure 6 shown.

[0090] Example 3

[0091] This paper uses three micro-expression databases: CASMEII, SAMM, and SMIC. Table 1 summarizes the portions of these three databases we use. The SMIC dataset has only three categories: negative, positive, and surprise. To combine them into a single dataset and perform unified evaluation metrics, the emotion classification labels in all three datasets are appropriately mapped to a common subset consisting of negative, positive, and surprise.

[0092] This set includes 442 samples from 68 subjects.

[0093]

[0094] Table 1 Summary of three micro-expression databases

[0095] The CASMEII and SAMM databases provide labels for the start and apex frames of each video frame. For SMIC, the apex frames are not labeled; instead, the middle frame of each video frame is used as the apex frame. Furthermore, the Macro Expression Dataset CK+ was used as a pre-training dataset. This dataset contains 327 video clips with expression labels, which can also be re-labeled with the three labels mentioned above. After cropping all video frames to a 640*480 resolution, facial regions were detected and aligned using the DLib method for the start and apex frames. Optical flow was extracted and aligned, and the optical flow maps were further normalized to 224*224 pixels as network input. Furthermore, adversarial domain adaptation techniques were used for the macro and micro expression datasets to label whether an expression belongs to a macro expression, enabling the network to distinguish between the two domains. This enhanced distinction between the macro and micro expression domains was achieved. Furthermore, data augmentation was used, with color shifts of a probability of 0.2 and a maximum value of 20, and rotations of a maximum of 10 degrees.

[0096] For the previously proposed balanced amplification coefficient method, after verification and trial, A, B, C, and M were respectively -0.0309, -0.0064, 2.1449, and 2.2. Due to differences in data distribution and environmental background of different data sets, it is necessary to fine-tune each data set individually, but the effect on the results is not significant. In formula (1), Q(m i ) can be adjusted to other functions as long as the magnification conditions are met. When the magnification factor is greater than 20, it must be limited to 20. Excessively large magnification factors can easily cause image distortion and distortion, affecting recognition results. To reduce the amplitude of macro expression changes, we use a binary search method to find frames with facial motion amplitudes between 8 and 11 in the video sequence as vertex frames. If no such frames are found after the search is completed, the frame with the motion amplitude closest to 9.5 in the previously searched frames is used as the vertex frame, thereby balancing the distribution of the dataset.

[0097] For our proposed network, after adopting the ImageNet

[38] pre-trained weights, we use a decay weight of 0.000001, the batch size is set to 32, and the learning rate is initialized to 0.001 and reduced by a factor of 2 every 10 epochs.

[0098] During the fine-tuning and testing phases, the traditional leave-one-subject-out (LOSO) method was used. The leave-one-subject-out validation is suitable for ensuring subject-independent evaluation. Therefore, 68 training and testing procedures were performed. Due to the imbalanced label distribution, the unweighted F1 score (UF1) and the unweighted average recall (UAR) are considered as performance indicators here to avoid the proposed method from overfitting to a specific class. Assume that each true positive (TP) from c classes (total C classes) c ), false positive (FP c ) and false negative (FN c ) exceeds 68 times, UF1 can be calculated as:

[0099]

[0100] in:

[0101]

[0102] UAR can be expressed as:

[0103]

[0104] in:

[0105]

[0106] Table 2 shows the experimental results based on the LOSO evaluation method. According to Table 2, it can be seen that our proposed method significantly outperforms the baseline LBP-TOP and EMR

[27] after adding the non-local attention mechanism. The unweighted F1 score (UF1) and unweighted average recall (UAR) of the three datasets reach 0.8679 and 0.8628, respectively, while LBP-TOP only achieves 0.5882 and 0.5785, respectively. If we additionally add optical flow alignment and balanced data distribution methods, both UF1 and UAR can be improved by more than 4 percentage points compared to the previous ones. Therefore, the adaptive upscaling factor technique is crucial for the micro-expression recognition task with small datasets. The proposed method also outperforms the baseline method on each individual part of the combined dataset. More specifically, the methods in the last row of Table 2 (including all our proposed techniques) achieve more significant improvements in UF1 and UAR on the CASME II dataset. However, the balanced upscaling factor does not show the same effectiveness on the SMIC dataset and SAMM dataset. The reason for the inconsistency may be that the motion amplitude distribution of macro expressions is more consistent with the motion amplitude distribution of CASME II. Adjusting the A, B, C, and M parameters to make the distribution of macro-expression and micro-expression motion amplitude closer can also improve the recognition performance of other micro-expression datasets.

[0107]

[0108] Table 2 Comparison of different methods on different datasets

[0109] In order to better understand the present invention, the above is described in detail in conjunction with the specific embodiments of the present invention, but it is not intended to limit the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A micro-expression recognition method based on transfer learning, comprising preprocessing facial image data, characterized in that: The following steps are also included: Step 1: Extract the starting frame and vertex frame of the micro-expression respectively, and extract half of the frame between the starting frame and vertex frame of the macro-expression at the same time; Step 2: Perform facial detection on the extracted images and extract optical flow features from the two frames. The two frames are used as input to the ResNet18 network. An adaptive amplification coefficient method is used to balance the expression variation of different samples in the dataset. The adaptive amplification coefficient method includes the following sub-steps: Step 21: Use the optical flow method to extract the optical flow features of the sample start frame and vertex frame in the source data set; Step 22: Calculate the maximum value of the optical flow as the motion amplitude of the sample in the source data set; Step 23: Calculate the amplification factor according to the motion amplitude; When the sample motion amplitude is less than 1, a larger magnification factor is required; when the sample motion amplitude is greater than 8, no more amplification is required; the magnification factor P of the i-th sample i Defined as Among them, m i is the original facial motion amplitude, A, B, C and M are constant coefficients; Step 3: Use the ResNet network to extract features, divide the extracted features into two parts, and send them to the fully connected layer for recognition after average pooling. The extracted features are recognized and output.

2. The micro-expression recognition method based on transfer learning according to claim 1, wherein The step of pre-processing the facial image data also includes amplifying the facial movement amplitude using the Euler video amplification algorithm to increase the micro-expression movement amplitude.

3. The micro-expression recognition method based on transfer learning as claimed in claim 2, characterized in that The preprocessing includes taking the maximum value of the optical flow between the starting frame and the vertex frame in the macro expression sample as the motion amplitude of the expression, and using a dichotomy method to find the motion amplitude close to the average value of the micro expression sample after amplification.

4. The micro-expression recognition method based on transfer learning as claimed in claim 3, characterized in that The step 3 includes adding a non-local module after the feature fusion of each layer of the ResNet network. The non-local method is Among them, y i is the output signal at position i, x i is the input signal at position i, x j is the input signal at position j, f(x i ,x j ) represents the relationship between position i and all possible associated positions j, C(x) is the normalization parameter, For any other position in the feature map, g(x j ) is a linear transfer function.

5. The micro-expression recognition method based on transfer learning as claimed in claim 4, characterized in that The relationship f(x i ,x j ) is calculated as Among them, [θ(x i ),φ(x j )] represents θ(x i ) and φ(x j ) cascade, θ(x i )=W θ x i ,φ(x j )=W φ x i , is the weight vector that projects the concatenated vector to a scalar, W θ is the weight parameter θ, W φ is the weight parameter φ.

6. The micro-expression recognition method based on transfer learning according to claim 5, characterized in that The linear conversion function g(x j ) is calculated as g(x j )=W g x j Among them, W g is the weight of the linear transfer function.

7. The micro-expression recognition method based on transfer learning according to claim 6, characterized in that: Through 1×1 convolution, y i and x i The number of channels remains the same, and the sum is z i =W z y i +x i Among them, z i is the weight sum calculated at point i, W z is the learnable weight matrix.

Images