Micro-expression recognition method fusing macro-expression transfer learning and data dimension reduction
By employing dense optical flow algorithm, nonlinear amplification, unified manifold approximation and projection, vectorized max pooling layer and guidance mechanism, combined with convolutional neural network, the feature domain shift problem in macro-expression to micro-expression recognition is solved, achieving robust micro-expression recognition in complex environments.
Patent Information
- Application Number
- CN202511501379.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-21
- Publication Date
- 2026-01-13
AI Technical Summary
Existing technologies struggle to effectively utilize the rich data resources of macro-expressions, and when directly transferred to micro-expression tasks, feature domain shift issues arise. Furthermore, traditional methods struggle to achieve robust micro-expression recognition in complex environments, particularly under conditions of lighting changes, pose shifts, and occlusion, resulting in insufficient recognition accuracy.
Features are extracted using a dense optical flow algorithm, and dimensionality is reduced by nonlinear amplification, unified manifold approximation, and projection algorithm. Combined with vectorized max pooling layers and a guidance mechanism, macro-expression and micro-expression features are matched and fused, and learning is performed using a convolutional neural network.
It improves the robustness and accuracy of micro-expression recognition, achieves real-time performance and stability in complex environments, and enhances the ability of knowledge transfer and feature fusion across expression domains.
Smart Images
Figure CN121330744A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and relates to a micro-expression recognition method that integrates macro-expression transfer learning and data dimensionality reduction. Background Technology
[0002] In computer vision research, micro-expression recognition has received continuous attention in fields such as psychological assessment, interrogation analysis, and human-computer interaction because it can reflect the true emotional responses at the subconscious level. However, the physiological characteristics of micro-expressions themselves make them one of the most challenging areas in expression recognition. On the one hand, the changes in facial muscles during micro-expressions are extremely small (usually only 1 / 5 to 1 / 10 of those in macro-expressions), making their dynamic features easily masked by environmental noise or lighting interference; traditional motion feature extraction algorithms often struggle to extract stable representational information from them. On the other hand, micro-expression datasets are limited in size, with very few samples bearing precise labels. Currently used datasets (such as CASME and SMIC) generally have fewer than a thousand samples and suffer from problems such as imbalanced expression categories and significant differences in shooting conditions, making conventional models prone to overfitting during the training phase and exhibiting insufficient generalization ability.
[0003] Meanwhile, macro-expression recognition technology has become relatively mature. Macro-expressions have advantages such as large motion amplitude, significant features, and abundant sample size (e.g., the FER-2013 and CK+ datasets both have sample sizes exceeding tens of thousands). Deep learning-based macro-expression recognition models have achieved an accuracy exceeding 95% under controlled environments. However, macro-expressions and micro-expressions differ significantly in motion scale, duration (micro-expressions typically last 0.04-0.2 seconds, while macro-expressions last 0.5-4 seconds), and feature distribution. Directly transferring macro-expression recognition models to micro-expression tasks will result in a significant drop in recognition performance due to the "feature domain shift" problem, failing to fully utilize the rich data resources and mature model experience of macro-expressions.
[0004] At the feature processing level, traditional micro-expression recognition methods often employ linear dimensionality reduction techniques to process high-dimensional optical flow or texture features. However, the feature distribution of micro-expressions is highly nonlinear, and linear dimensionality reduction inevitably loses key nonlinear structural information, leading to a decrease in feature discriminative power after dimensionality reduction. At the same time, existing feature fusion methods often use simple weighting or concatenation strategies, failing to establish semantic connections between macro-expression and micro-expression features, and thus failing to achieve effective transfer of macro-expression knowledge to micro-expression tasks.
[0005] The complexity of real-world applications further amplifies these problems. In dynamic video monitoring or natural interaction environments, factors such as changes in lighting, pose shifts, and occlusion can weaken micro-expression feature signals. Existing algorithms often struggle to maintain both recognition accuracy and real-time performance while ensuring stability. Therefore, improving recognition robustness under limited sample conditions, establishing a correlation channel between macro-expressions and micro-expressions, and achieving cross-expression domain knowledge transfer and feature fusion have become key research directions for enhancing micro-expression recognition performance in complex scenarios. Summary of the Invention
[0006] In view of this, the purpose of this invention is to provide a micro-expression recognition method that integrates macro-expression transfer learning and data dimensionality reduction.
[0007] To achieve the above objectives, the present invention provides the following technical solution:
[0008] A micro-expression recognition method integrating macro-expression transfer learning and data dimensionality reduction includes the following steps:
[0009] Step 1: Extract macro-expression and micro-expression features using the dense optical flow algorithm;
[0010] Step 2: Non-linearly amplify the micro-expression features to enhance the intensity of micro-expression expression;
[0011] Step 3: Use uniform manifold approximation and projection to reduce the dimensionality of macro-expression and micro-expression features, and then perform macro-micro-expression matching;
[0012] Step 4: Use vectorized max pooling layers to preserve directional information in macro-expression and micro-expression features;
[0013] Step 5: Use a guided mechanism to fuse macro-expressions and micro-expressions, and then use a convolutional neural network for learning.
[0014] 2. The micro-expression recognition method integrating macro-expression transfer learning and data dimensionality reduction according to claim 1, characterized in that: step one specifically includes: performing face cropping and alignment processing on macro-expression and micro-expression samples. Next, the Farneback dense optical flow algorithm is used to construct the optical flow vector field of expression changes for the preprocessed macro-expression and micro-expression samples. A starting frame I is selected from the macro-expression or micro-expression video sequence. s (x,y) and peak frame I p (x, y), where (x, y) represents the image pixel coordinates. The starting frame corresponds to the initial state of the facial expression change, and the peak frame corresponds to the moment when the facial expression reaches its maximum intensity. The pixel displacement between the starting frame and the peak frame is calculated using the Farneback optical flow algorithm, as shown in the following formula:
[0015] V(x,y)=Farneback(Is ,I p )
[0016] Right now:
[0017] V(x,y)=[u(x,y),v(x,y)]
[0018] Where u(x,y) represents the horizontal displacement of the pixel, and v(x,y) represents the vertical displacement of the pixel.
[0019] 3. The micro-expression recognition method integrating macro-expression transfer learning and data dimensionality reduction according to claim 2, characterized in that: in step two, a nonlinear amplification method is used to enhance the expression intensity in the optical flow field of micro-expressions. To improve the distinction between salient and insignificant movements in micro-expressions, an exponential function is used as the nonlinear amplification function. Simultaneously, a piecewise function structure is adopted to accommodate the directional information in the optical flow field. The specific nonlinear amplification function is shown below:
[0020]
[0021] Where |·| represents absolute value operation, and λ is a manually adjustable parameter.
[0022] 4. The micro-expression recognition method integrating macro-expression transfer learning and data dimensionality reduction according to claim 3, characterized in that: in step three, the optical flow vector field is transformed from matrix form to vector form:
[0023] I(x,y)=[I x ,I y ]
[0024] I = Reshape(V(x,y))
[0025] Where Reshape(·) represents vector shape reshaping, I x and I y These represent the motion vectors in the horizontal and vertical directions, respectively.
[0026] Then, a unified manifold approximation and projection algorithm are used to perform nonlinear dimensionality reduction on the high-dimensional optical flow features, and the optical flow vector feature sets of the two types of samples are jointly constructed as follows:
[0027] I = {I1, I2, ..., I} N}
[0028] Among them, I i This includes optical flow vectors from macro-expression samples and micro-expression samples.
[0029] Subsequently, using the Unified Manifold Approximation and Projection (UMAP) algorithm, a low-dimensional mapping model is performed on the joint feature set to obtain a high-dimensional to low-dimensional nonlinear mapping function:
[0030]
[0031] Where m and n are the height and width of the image, respectively, and k ≤ 2mn. This mapping function is established by minimizing the cross-entropy loss between high- and low-dimensional neighborhood distributions. First, a weighted adjacency graph between samples in the high-dimensional space is constructed:
[0032]
[0033] Where d(·) is the Euclidean distance, ρ i σ represents the local radius. i The scale parameter is used to represent the next step, which involves finding a mapping point in the low-dimensional space and maintaining adjacency by minimizing the cross-entropy loss.
[0034]
[0035] Among them, w′ ij Representing the adjacency probability in low-dimensional space, we obtain the low-dimensional embedding vectors of macro and micro expression samples:
[0036]
[0037] in Macro-expressions and micro-expressions are paired based on the similarity of samples in the low-dimensional feature space to achieve sample alignment in transfer learning. Cosine similarity is used to measure the feature proximity between the two classes of samples, as shown in the following formula:
[0038] D(z Ma ,z Mi )=||z Ma -z Mi ||2
[0039] When D(z) Ma ,z Mi The macro-expression sample is considered to have successfully matched the micro-expression sample when the minimum value is reached.
[0040] 5. A micro-expression recognition method integrating macro-expression transfer learning and data dimensionality reduction according to claim 4, characterized in that: in step four, a vectorized max-pooling layer is used to separate the motion information contained in the features, positive information is separated by a sign function and a ReLU function, and negative information is separated by a negative sign function and a ReLU function.
[0041]
[0042] Where φ(·) represents the ReLU function and Sign(·) represents the sign function. Then, the intensity information in the features is separated using the Hadamard multiplier integral, and the salient information in the features is extracted using a max pooling layer, as shown in the following formula:
[0043]
[0044] in and These represent the positive and negative motion intensities after max pooling, respectively, and Maxpool(·) denotes a one-dimensional max pooling layer. This represents the Hadamard product. Then, the extracted motion intensity and orientation information are fused to obtain the vectorized max-pooled features:
[0045]
[0046] 6. A micro-expression recognition method integrating macro-expression transfer learning and data dimensionality reduction according to claim 5, characterized in that: in step five, a guidance mechanism is used during the learning process to transfer macro-expression features to micro-expressions to enhance the expression of micro-expressions. First, absolute value operation is used to filter the directional information in the macro-expression features, and then the tanh function is used to compress the elements in the features to the 0-1 interval to obtain the macro-expression weights:
[0047]
[0048] Then ω Ma Micro-expression features By fusing the data, we obtain micro-expression features guided by macro-expressions:
[0049]
[0050] in, This represents the Hadamard product. Finally, a one-dimensional convolutional neural network is used to learn macro-micro expression features.
[0051] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:
[0053] Figure 1This is a flowchart illustrating a micro-expression recognition method that integrates macro-expression transfer learning and data dimensionality reduction. Detailed Implementation
[0054] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0055] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0056] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.
[0057] See Figure 1 This invention provides a micro-expression recognition method that integrates macro-expression transfer learning and data dimensionality reduction. Figure 1 To illustrate the flowchart, the following steps, in conjunction with the accompanying diagram, are described:
[0058] Optionally, step one specifically includes: performing face cropping and alignment processing on the macro-expression and micro-expression samples. Next, the Farneback dense optical flow algorithm is used to construct the optical flow vector field of expression changes for the preprocessed macro-expression and micro-expression samples. A starting frame I is selected from the macro-expression or micro-expression video sequence. s (x,y) and peak frame I p(x, y), where (x, y) represents the image pixel coordinates. The starting frame corresponds to the initial state of the facial expression change, and the peak frame corresponds to the moment when the facial expression reaches its maximum intensity. The pixel displacement between the starting frame and the peak frame is calculated using the Farneback optical flow algorithm, as shown in the following formula:
[0059] V(x,y)=Farneback(I s ,I p )
[0060] Right now:
[0061] V(x,y)=[u(x,y),v(x,y)]
[0062] Where u(x,y) represents the horizontal displacement of the pixel, and v(x,y) represents the vertical displacement of the pixel.
[0063] Optionally, in step two, a nonlinear amplification method is used to enhance the expression intensity in the optical flow field of micro-expressions. To improve the distinction between significant and insignificant movements in micro-expressions, an exponential function is used as the nonlinear amplification function. Simultaneously, a piecewise function structure is adopted to accommodate the directional information in the optical flow field. The specific nonlinear amplification function is shown below:
[0064]
[0065] Where |·| represents absolute value operation, and λ is a manually adjustable parameter.
[0066] Optionally, in step three, the optical flow vector field is transformed from matrix form to vector form:
[0067] I(x,y)=[I x ,I y ]
[0068] I = Reshape(V(x,y))
[0069] Where Reshape(·) represents vector shape reshaping, I x and I y These represent the motion vectors in the horizontal and vertical directions, respectively.
[0070] Then, a unified manifold approximation and projection algorithm are used to perform nonlinear dimensionality reduction on the high-dimensional optical flow features, and the optical flow vector feature sets of the two types of samples are jointly constructed as follows:
[0071] I = {I1, I2, ..., I} N}
[0072] Among them, I i This includes optical flow vectors from macro-expression samples and micro-expression samples.
[0073] Subsequently, using the Unified Manifold Approximation and Projection (UMAP) algorithm, a low-dimensional mapping model is performed on the joint feature set to obtain a high-dimensional to low-dimensional nonlinear mapping function:
[0074]
[0075] Where m and n are the height and width of the image, respectively, and k ≤ 2mn. This mapping function is established by minimizing the cross-entropy loss between high- and low-dimensional neighborhood distributions. First, a weighted adjacency graph between samples in the high-dimensional space is constructed:
[0076]
[0077] Where d(·) is the Euclidean distance, ρ i σ represents the local radius. i The scale parameter is used to represent the next step, which involves finding a mapping point in the low-dimensional space and maintaining adjacency by minimizing the cross-entropy loss.
[0078]
[0079] Among them, w′ ij Representing the adjacency probability in low-dimensional space, we obtain the low-dimensional embedding vectors of macro and micro expression samples:
[0080]
[0081] in Macro-expressions and micro-expressions are paired based on the similarity of samples in the low-dimensional feature space to achieve sample alignment in transfer learning. Cosine similarity is used to measure the feature proximity between the two classes of samples, as shown in the following formula:
[0082] D(z Ma ,z Mi )=||z Ma -z Mi ||2
[0083] When D(z) Ma ,z Mi The macro-expression sample is considered to have successfully matched the micro-expression sample when the minimum value is reached.
[0084] Optionally, in step four, a vectorized max-pooling layer is used to separate the motion information contained in the features, positive information is separated using a sign function and a ReLU function, and negative information is separated using a negative sign function and a ReLU function.
[0085]
[0086] Where φ(·) represents the ReLU function and Sign(·) represents the sign function. Then, the intensity information in the features is separated using the Hadamard multiplier integral, and the salient information in the features is extracted using a max pooling layer, as shown in the following formula:
[0087]
[0088] in and These represent the positive and negative motion intensities after max pooling, respectively, and Maxpool(·) denotes a one-dimensional max pooling layer. This represents the Hadamard product. Then, the extracted motion intensity and orientation information are fused to obtain the vectorized max-pooled features:
[0089]
[0090] Optionally, in step five, a guidance mechanism is used during the learning process to transfer macro-expression features to micro-expressions to enhance their expressiveness. First, absolute value operations are used to filter the directional information in the macro-expression features. Then, the tanh function is used to compress the elements in the features to the 0-1 range to obtain the macro-expression weights.
[0091]
[0092] Then ω Ma Micro-expression features By fusing the data, we obtain micro-expression features guided by macro-expressions:
[0093]
[0094] in, This represents the Hadamard product. Finally, a one-dimensional convolutional neural network is used to learn macro-micro expression features.
[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A micro-expression recognition method integrating macro-expression transfer learning and data dimensionality reduction, characterized in that: The method includes the following steps: Step 1: Extract macro-expression and micro-expression features using the dense optical flow algorithm; Step 2: Non-linearly amplify the micro-expression features to enhance the intensity of micro-expression expression; Step 3: Use uniform manifold approximation and projection to reduce the dimensionality of macro-expression and micro-expression features, and then perform macro-micro-expression matching; Step 4: Use vectorized max pooling layers to preserve directional information in macro-expression and micro-expression features; Step 5: Use a guided mechanism to fuse macro-expressions and micro-expressions, and then use a convolutional neural network for learning.
2. The micro-expression recognition method according to claim 1, which integrates macro-expression transfer learning and data dimensionality reduction, is characterized in that: Step one specifically includes: performing face cropping and alignment processing on the macro-expression and micro-expression samples. Next, the Farneback dense optical flow algorithm is used to construct the optical flow vector field of expression changes for the preprocessed macro-expression and micro-expression samples. A starting frame I is selected from the macro-expression or micro-expression video sequence. s (x,y) and peak frame I p (x, y), where (x, y) represents the image pixel coordinates. The starting frame corresponds to the initial state of the facial expression change, and the peak frame corresponds to the moment when the facial expression reaches its maximum intensity. The pixel displacement between the starting frame and the peak frame is calculated using the Farneback optical flow algorithm, as shown in the following formula: V(x,y)=Farneback(I s ,I p ) Right now: V(x,y)=[u(x,y),v(x,y)] Where u(x,y) represents the horizontal displacement of the pixel, and v(x,y) represents the vertical displacement of the pixel.
3. The micro-expression recognition method according to claim 2, which integrates macro-expression transfer learning and data dimensionality reduction, is characterized in that: In step two, a nonlinear amplification method is used to enhance the expression intensity in the optical flow field of micro-expressions. To improve the distinction between salient and insignificant motions in micro-expressions, an exponential function is used as the nonlinear amplification function. Simultaneously, a piecewise function structure is employed to accommodate directional information in the optical flow field. The specific nonlinear amplification function is shown below: Where |·| represents absolute value operation, and λ is a manually adjustable parameter.
4. The micro-expression recognition method according to claim 3, which integrates macro-expression transfer learning and data dimensionality reduction, is characterized in that: In step three, the optical flow vector field is transformed from matrix form to vector form: I(x,y)=[I x ,I y ] I = Reshape(V(x,y)) Where Reshape(·) represents vector shape reshaping, I x and I y These represent the motion vectors in the horizontal and vertical directions, respectively. Then, a unified manifold approximation and projection algorithm are used to perform nonlinear dimensionality reduction on the high-dimensional optical flow features, and the optical flow vector feature sets of the two types of samples are jointly constructed as follows: I={I1,I2,...,I N } Among them, I i This includes optical flow vectors from macro-expression samples and micro-expression samples. Subsequently, using the Unified Manifold Approximation and Projection (UMAP) algorithm, a low-dimensional mapping model is performed on the joint feature set to obtain a high-dimensional to low-dimensional nonlinear mapping function: Where m and n are the height and width of the image, respectively, and k ≤ 2mn. This mapping function is established by minimizing the cross-entropy loss between high- and low-dimensional neighborhood distributions. First, a weighted adjacency graph between samples in the high-dimensional space is constructed: Where d(·) is the Euclidean distance, ρ i σ represents the local radius. i The scale parameter is used to represent the next step, which involves finding a mapping point in the low-dimensional space and maintaining adjacency by minimizing the cross-entropy loss. Among them, w′ ij Representing the adjacency probability in low-dimensional space, we obtain the low-dimensional embedding vectors of macro and micro expression samples: in Macro-expressions and micro-expressions are paired based on the similarity of samples in the low-dimensional feature space to achieve sample alignment in transfer learning. Cosine similarity is used to measure the feature proximity between the two classes of samples, as shown in the following formula: D(z Ma ,z Mi )=||z Ma -z Mi ||2 When D(z) Ma ,z Mi The macro-expression sample is considered to have successfully matched the micro-expression sample when the minimum value is reached.
5. The micro-expression recognition method according to claim 4, characterized in that: In step four, a vectorized max pooling layer is used to separate the motion information contained in the features. Positive information is separated using a sign function and a ReLU function, and negative information is separated using a negative sign function and a ReLU function. Where φ(·) represents the ReLU function and Sign(·) represents the sign function. Then, the intensity information in the features is separated using the Hadamard multiplier integral, and the salient information in the features is extracted using a max pooling layer, as shown in the following formula: in and These represent the positive and negative motion intensities after max pooling, respectively, and Maxpool(·) denotes a one-dimensional max pooling layer. This represents the Hadamard product. Then, the extracted motion intensity and orientation information are fused to obtain the vectorized max-pooled features:
6. The micro-expression recognition method according to claim 5, which integrates macro-expression transfer learning and data dimensionality reduction, is characterized in that: In step five, a guided mechanism is used during the learning process to transfer macro-expression features to micro-expressions, thereby enhancing the expression of micro-expressions. First, absolute value operations are used to filter the directional information in the macro-expression features. Then, the tanh function is used to compress the elements in the features to the 0-1 interval to obtain the macro-expression weights. Then ω Ma Micro-expression features By fusing the data, we obtain micro-expression features guided by macro-expressions: in, This represents the Hadamard product. Finally, a one-dimensional convolutional neural network is used to learn macro-micro expression features.