Micro-expression motion feature representation method and system

Abstract

The application discloses a micro-expression motion feature representation method and system, and belongs to the technical field of micro-expression recognition and computer vision. The method obtains a starting frame image and a target frame image in a micro-expression video sequence, performs CLAHE (Contrast Limited Adaptive Histogram Equalization) processing on the input image to obtain an enhanced continuous frame image; the motion displacement of each key point is calculated by using a key point correspondence relationship to construct a key point sparse optical flow feature; the key point motion amplitude, motion direction and local motion response are encoded to obtain a key point motion feature representation; the attention weight of each key point is adaptively calculated according to the key point motion response information, the motion feature is weighted and enhanced, the weighted key point motion feature is aggregated to form a micro-expression motion feature representation, and the micro-expression recognition network is input to complete the micro-expression classification and emotion recognition of learners. The application improves the representation ability and recognition robustness of weak micro-expression motion.

Description

Technical Field

[0001] This invention belongs to the field of micro-expression recognition technology, and relates to a method and system for representing micro-expression motion features. Background Technology

[0002] With the rapid development of online education platforms and smart teaching systems, the automatic analysis of learners' emotional states using computer vision technology has become an important research direction in educational intelligence. Micro-expressions, in particular, can reflect learners' genuine emotional changes during the learning process, and are of great significance for learning status assessment, personalized teaching feedback, and improving teaching quality. Therefore, learner emotion recognition based on micro-expressions has gradually attracted widespread attention.

[0003] Currently, mainstream micro-expression recognition methods mainly include approaches based on convolutional neural networks (CNNs), optical flow features, and Transformers. CNN-based methods classify emotions by learning facial texture features, effectively extracting local spatial information. Optical flow-based methods utilize motion changes between consecutive frames to describe facial muscle movements, reflecting the dynamic characteristics of micro-expressions better than static texture features. Given that micro-expressions primarily occur in local areas, existing methods often utilize key regions such as the eyes, eyebrows, and corners of the mouth for recognition to reduce background interference. Furthermore, image enhancement techniques such as CLAHE are widely used in expression recognition and other fields, improving the visibility of texture details in low-quality images by enhancing local contrast.

[0004] However, existing micro-expression recognition methods still have certain limitations in online education scenarios. Facial images acquired in online learning environments generally suffer from problems such as varying lighting, low resolution, and insufficient local contrast, making it difficult to effectively preserve texture details in key micro-expression regions, accurately extract weak motion information, and allowing key motion regions to be easily interfered with by the background. Therefore, how to enhance local texture information in low-quality images and achieve more accurate and robust micro-expression motion feature representation has become an important problem that urgently needs to be solved in the field of emotion analysis in online education. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for representing micro-expression motion features to solve the above-mentioned problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for characterizing micro-expression motion features, comprising the following steps: The starting frame image and the target frame image in the micro-expression video sequence are obtained. The input image is subjected to contrast-limited adaptive histogram equalization (CLAHE) to obtain the enhanced continuous frame image. The enhanced consecutive frame images are input into the face key point detection module to locate the face region, extract the coordinate information of the facial key points, and establish the correspondence between key points between consecutive frames. By utilizing the correspondence between key points in consecutive frames, the motion displacement of each key point in the time dimension is calculated to obtain the motion trajectory of the key points and construct the sparse optical flow features of the key points. The sparse optical flow features of the key points are input into the motion feature extraction module, and the motion amplitude, motion direction and local motion response of the key points are encoded to obtain the motion feature representation of the key points. An optical flow-guided keypoint attention module is constructed, which adaptively calculates the attention weights of each keypoint based on the motion response information of the keypoints, and weights and enhances the motion features. The weighted keypoint motion features are aggregated to form a micro-expression motion feature representation, which is then input into a micro-expression recognition network to complete learner micro-expression classification and emotion recognition.

[0007] Furthermore, the step of acquiring the start frame image and target frame image from the micro-expression video sequence, and performing contrast-limited adaptive histogram equalization (CLAHE) processing on the input image to obtain enhanced continuous frame images includes: The input image is converted to a color space that separates luminance and color, and only the luminance component is enhanced. The luminance image is divided into multiple local sub-regions, and the gray-level histogram of each sub-region is calculated. A contrast limit threshold is set, and the histograms of each local sub-region are cropped, with the excess parts redistributed evenly to each gray level. The gray-level mapping of each sub-region is completed according to the cumulative distribution function, and bilinear interpolation is used to smoothly fuse adjacent sub-regions. The enhanced luminance information and color information are then re-fused to obtain the enhanced image.

[0008] Furthermore, the step of inputting the enhanced consecutive frame images into the facial key point detection module to locate the facial region, extract the coordinate information of facial key points, and establish the correspondence between key points between consecutive frames includes: The face region in the input image is located using a face detection algorithm; the detected face region is geometrically normalized; face key points are extracted using a face detection model based on histogram of oriented gradients and a key point localization model based on ensemble regression tree; and the coordinates of the face key points corresponding to consecutive frames are obtained.

[0009] Furthermore, the step of calculating the motion displacement of each key point in the time dimension by utilizing the correspondence between key points in consecutive frames, obtaining the motion trajectory of the key points, and constructing sparse optical flow features of the key points includes: The motion matching relationship between corresponding keypoints in consecutive frames is established using the Lucas-Kanade sparse optical flow algorithm; the motion displacement of each keypoint in the horizontal and vertical directions is calculated; the motion amplitude and direction of the keypoints are calculated based on the horizontal and vertical displacements; the local texture response is calculated based on the grayscale changes of the enhanced consecutive frame images in the neighborhood of the keypoints; the spatial position, motion displacement, motion direction, motion amplitude, and local texture response of the keypoints are jointly constructed as the sparse optical flow features of the keypoints; and the pyramid optical flow strategy is used to estimate the motion of keypoints at different scales.

[0010] Furthermore, the motion feature extraction module includes: encoding the motion information of each key point using the sparse optical flow features of the key points as input; learning the motion response features and local texture change features of the key points respectively; fusing the motion response information and local texture information to obtain the key point motion feature representation describing the micro-expression motion state; and mapping the motion features of each key point to the same feature space.

[0011] Furthermore, the optical flow-guided keypoint attention module includes: calculating the motion saliency of each keypoint based on the keypoint motion response information; constructing a comprehensive keypoint response feature by combining local texture response information; calculating the attention weight corresponding to each keypoint using an attention mechanism to achieve adaptive allocation of keypoint importance; weighting the keypoint motion features according to the attention weights to enhance the main activation areas of micro-expressions such as eyebrows, eyes, and corners of the mouth, and reduce the feature response of low-motion areas such as the bridge of the nose and cheeks; and outputting the keypoint motion features enhanced by optical flow-guided attention. The attention weights are calculated based on the keypoint motion amplitude and local texture response, and are obtained using the Softmax function. The attention weights corresponding to each keypoint are determined according to the weight coefficient between motion information and texture information, and the value range of the weight coefficient is 0 to 1.

[0012] Furthermore, the micro-expression feature output includes: aggregating the weighted motion features of all key points to form an overall micro-expression motion representation; inputting the aggregated motion representation into a micro-expression recognition network for feature learning; outputting classification results corresponding to different micro-expression categories; and obtaining the learner's emotional state recognition result based on the classification results.

[0013] Secondly, the present invention provides a micro-expression motion feature representation system, comprising: The image enhancement module is used to acquire the start frame image and target frame image in the micro-expression video sequence, and perform contrast-limited adaptive histogram equalization (CLAHE) on the input image to obtain the enhanced continuous frame image. The key point detection module is used to input the enhanced continuous frame images into the face key point detection module, locate the face region, extract the coordinate information of the facial key points, and establish the correspondence between key points between continuous frames. The optical flow-guided keypoint attention module is used to calculate the motion displacement of each keypoint in the time dimension by utilizing the correspondence between keypoints between consecutive frames, obtain the motion trajectory of the keypoints, and construct the sparse optical flow features of the keypoints. The key point motion feature acquisition module is used to input the sparse optical flow features of the key points into the motion feature extraction module, encode the motion amplitude, motion direction and local motion response of the key points, and obtain the key point motion feature representation. The output module is used to construct an optical flow-guided keypoint attention module. It adaptively calculates the attention weights of each keypoint based on the keypoint motion response information and weights and enhances the motion features. The weighted keypoint motion features are aggregated to form a micro-expression motion feature representation, which is then input into the micro-expression recognition network to complete the learner's micro-expression classification and emotion recognition.

[0014] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the micro-expression motion feature representation method described above.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the micro-expression motion feature representation method.

[0016] Compared with the prior art, the present invention has the following technical effects: This invention enhances the discernibility of local texture details in low-quality facial images by utilizing CLAHE enhancement, providing clearer input for subtle motion analysis. By constructing keypoint sparse optical flow features that integrate motion displacement, motion amplitude, motion direction, and local texture response, it enables refined representation of micro-expression motion information. By introducing an optical flow-guided keypoint attention mechanism, attention weights are adaptively allocated based on keypoint motion response and texture changes. This effectively highlights sensitive areas of micro-expressions and suppresses background interference, thereby obtaining more discriminative micro-expression motion features and improving the recognition accuracy and robustness of the micro-expression recognition model. Attached Figure Description

[0017] Figure 1 This is a general framework diagram of the micro-expression motion feature representation method of the present invention.

[0018] Figure 2This is a flowchart illustrating the key point sparse optical flow motion feature extraction process of the present invention.

[0019] Figure 3 This is a structural diagram of the optical flow-guided key point attention module of the present invention.

[0020] Figure 4 This is a flowchart illustrating the optical flow-guided key point motion feature enhancement process of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] The present invention will now be described in further detail with reference to the accompanying drawings: Example 1, see Figure 1 In one embodiment of the present invention, a method for representing micro-expression motion features is provided. The implementation process of this method includes steps such as acquiring consecutive frame images, CLAHE local enhancement, keypoint detection, construction of sparse optical flow features of keypoints, and optical flow-guided keypoint attention calculation. Specifically, the method includes the following steps: S1: Obtain the starting frame image and target frame image from the micro-expression video sequence, perform contrast-limited adaptive histogram equalization (CLAHE) on the input image, and obtain the enhanced continuous frame image.

[0024] S2: Input the enhanced continuous frame images into the face key point detection module to locate the face region, extract the coordinate information of the facial key points, and establish the correspondence between key points between continuous frames.

[0025] S3: Calculate the motion displacement of each key point in the time dimension by utilizing the correspondence between key points in consecutive frames, obtain the motion trajectory of the key points, and construct the sparse optical flow features of the key points.

[0026] S4: Input the sparse optical flow features of the key points into the motion feature extraction module, encode the motion amplitude, motion direction and local motion response of the key points, and obtain the motion feature representation of the key points.

[0027] S5: Construct an optical flow-guided keypoint attention module, adaptively calculate the attention weight of each keypoint based on the keypoint motion response information, and enhance the motion features by weighting them; aggregate the weighted keypoint motion features to form a micro-expression motion feature representation, and input it into the micro-expression recognition network to complete the learner's micro-expression classification and emotion recognition.

[0028] In this invention, the micro-expression video sequences can originate from publicly available micro-expression datasets such as CASME II, SAMM, and SMIC, or from video data collected in actual online education scenarios. For each video sequence, the start frame and target frame are extracted based on the video annotation information, and are denoted as follows: and The starting frame image and the target frame image are input into the face preprocessing module to perform face detection, face cropping, and face alignment processing on the images, so that the two frames have a uniform face pose and spatial scale.

[0029] In step S1, the preprocessed consecutive frame images are input into the CLAHE enhancement module to perform local contrast enhancement. The CLAHE enhancement module converts the input image to a color space that separates luminance and color, enhancing only the luminance component. It divides the luminance image into multiple local sub-regions, calculates a gray-level histogram for each sub-region, and sets a contrast threshold to crop the local histogram. Excess values ​​are then redistributed evenly to the gray levels. The cumulative distribution function is used to complete the gray-level mapping of each sub-region. Finally, bilinear interpolation is used to smoothly fuse adjacent sub-regions, re-fusing the enhanced luminance and color information to obtain the enhanced image. The CLAHE enhancement process is represented as follows:

[0030]

[0031] in, This represents the enhanced starting frame image. This represents the enhanced target frame image. By limiting the local histogram contrast gain, the texture details of key micro-expression regions are enhanced while noise amplification is suppressed, improving the reliability of subsequent motion feature extraction.

[0032] In step S2, the enhanced consecutive frame images are input into the face keypoint detection module for keypoint localization. This invention employs a face detection method based on histogram of oriented gradients to localize the face region, and uses a keypoint localization model based on ensemble regression trees to extract face keypoints. Simultaneously, geometric normalization is performed on the detected face regions to reduce the impact of pose, scale, and positional variations, obtaining the corresponding face keypoint positions between consecutive frames, providing initial positions for subsequent keypoint motion estimation. Let the set of detected face keypoints be represented as... in This indicates the total number of key points.

[0033] In step S3, see Figure 2 The keypoints corresponding to consecutive frames are input into the Lucas-Kanade pyramid sparse optical flow algorithm to track corresponding keypoints between consecutive frames, establish motion matching relationships between corresponding keypoints in consecutive frames, and calculate the horizontal and vertical displacements of each keypoint between consecutive frames. Let the horizontal displacement of the i-th keypoint be ui and the vertical displacement be vi, then the motion amplitude of the corresponding keypoint is defined as...

[0034] Direction of motion is defined as

[0035] Motion amplitude is used to characterize the motion intensity of keypoints, and motion direction is used to characterize the motion direction of keypoints. To further describe the local texture changes within the neighborhood of a keypoint, a local neighborhood window is constructed centered on the i-th keypoint.

[0036] in This represents the coordinates of the i-th key point. This represents the radius of the neighborhood window.

[0037] Based on the enhanced consecutive frame images, the average gray-level change in the corresponding neighborhood is calculated, and the local texture response corresponding to the i-th keypoint is defined as...

[0038] in This indicates the number of pixels within the neighborhood window. This represents any pixel position within the neighborhood window. The horizontal and vertical displacements, motion amplitudes, motion directions, and local texture responses of the keypoints are combined to construct sparse optical flow features for the keypoints.

[0039] This feature vector integrates key point motion displacement, motion amplitude, motion direction, and local texture change information to achieve joint representation of motion and texture information, thereby improving the ability to describe subtle facial movements.

[0040] In step S4, the sparse optical flow features of each key point are input into the motion feature extraction module. The motion information of each key point is encoded using the sparse optical flow features of the key points as input. The motion response features and local texture change features of the key points are learned respectively. The motion response information and local texture information are fused to obtain the motion feature representation of the key points describing the micro-expression motion state. The motion features of each key point are uniformly mapped to the same feature space to provide input for subsequent attention calculation.

[0041] In step S5, see Figure 3 and Figure 4 An optical flow-guided keypoint attention module is constructed. Based on the motion response information of each keypoint, the motion saliency of each keypoint is calculated. Combined with local texture response information, a comprehensive keypoint response feature is constructed. An attention mechanism is used to calculate the attention weight corresponding to each keypoint, achieving adaptive allocation of keypoint importance. Let the motion amplitude of the i-th keypoint be mi, and the local texture response be Ti. Then, the comprehensive keypoint response is calculated based on both motion and texture information, and the attention weight corresponding to each keypoint is calculated using the Softmax function, expressed as follows:

[0042] in, Let represent the attention weight corresponding to the i-th keypoint, K represent the total number of keypoints, and λ represent the weighting coefficient between motion response and texture response, ranging from 0 to 1. When λ is larger, the attention weight focuses more on the motion response of the keypoint; when λ is smaller, the attention weight focuses more on the local texture response. After obtaining the attention weights of each keypoint, the sparse optical flow features corresponding to all keypoints are weighted and fused to obtain the optical flow-guided keypoint attention output features.

[0043] By weighting the motion features of key points according to attention weights, the main activation areas of micro-expressions such as eyebrows, eyes, and corners of the mouth are enhanced, while the feature responses of low-motion areas such as the bridge of the nose and cheeks are reduced, and the influence of background noise is suppressed, thus obtaining the motion features of key points enhanced by optical flow-guided attention.

[0044] After feature fusion, the weighted keypoint motion features are aggregated to form an overall micro-expression motion representation, which is then input into a subsequent micro-expression recognition network for feature learning. The network outputs classification results for different micro-expression categories and further obtains the learner's emotional state recognition result based on the classification results. The subsequent recognition network can employ convolutional neural networks, Transformer networks, or other neural network models capable of performing micro-expression classification tasks.

[0045] Example 2: This invention provides a method for representing micro-expression motion features based on CLAHE enhancement and optical flow-guided keypoint attention. The implementation process includes steps such as continuous frame image acquisition, CLAHE local enhancement, keypoint detection, construction of sparse optical flow features of keypoints, and optical flow-guided keypoint attention calculation.

[0046] In this invention, micro-expression video sequences are obtained as input data. These micro-expression video sequences can originate from publicly available micro-expression datasets such as CASME II, SAMM, and SMIC, or from video data collected in actual online education scenarios. For each video sequence, the onset frame and the target frame are extracted based on the video annotation information, and are denoted as follows: and .

[0047] The starting frame image and target frame image The input face preprocessing module performs face detection, face cropping, and face alignment on the image to ensure that the two frames have uniform face pose and spatial scale. In this invention, the processed face regions are uniformly scaled to 224×224 pixels to ensure consistency in subsequent motion feature calculations.

[0048] The preprocessed consecutive frame images are input into the CLAHE enhancement module for local contrast enhancement. The CLAHE enhancement module divides the input image into multiple local sub-regions, calculates a grayscale histogram for each sub-region, and crops the local histogram by setting a contrast threshold. Local histogram equalization is then performed on each sub-region, and finally, bilinear interpolation is used to smoothly blend adjacent sub-regions, resulting in the enhanced image.

[0049] The CLAHE enhancement process is represented as follows:

[0050]

[0051] in, This represents the enhanced starting frame image. This represents the enhanced target frame image.

[0052] After CLAHE enhancement, the enhanced consecutive frame images are input into the face landmark detection module for landmark localization. This invention employs a face detection method based on Histogram of Oriented Gradients (HOG) to localize the face region, and uses a landmark localization model based on ensemble regression trees to extract face landmarks. Let the set of detected face landmarks be represented as:

[0053] in, This indicates the total number of key points; in this invention, 68 key points are used.

[0054] The keypoints corresponding to consecutive frames are input into the Lucas-Kanade pyramid sparse optical flow algorithm to track the corresponding keypoints between consecutive frames and calculate the motion displacement of each keypoint between consecutive frames. Let the th frame be... The horizontal displacement of each key point is The vertical displacement is The motion amplitude of the corresponding key point is defined as The corresponding direction of motion is defined as Motion amplitude is used to characterize the intensity of keypoint motion, and motion direction is used to characterize the direction of keypoint motion.

[0055] To further describe the local texture changes within the neighborhood of keypoints, a local neighborhood window is constructed centered on the (i)th keypoint, defined as follows:

[0056] in, Indicates the first Coordinates of key points This represents the radius of the neighborhood window.

[0057] Based on the enhanced consecutive frame images, the average gray-level change in the corresponding neighborhood is calculated, and the local texture response corresponding to the (i)th keypoint is defined as:

[0058] in, This indicates the number of pixels within the neighborhood window. It represents any pixel position within the neighborhood window.

[0059] By combining the horizontal and vertical displacements, motion amplitudes, motion directions, and local texture responses of keypoints, a sparse optical flow feature for the keypoints is constructed, which is represented as follows:

[0060] in, Indicates the first The sparse optical flow motion feature vector corresponding to each key point integrates the key point's motion displacement, motion amplitude, motion direction, and local texture change information, providing input features for subsequent optical flow-guided key point attention calculation.

[0061] After constructing the sparse optical flow features of all key points, the feature vectors corresponding to each key point are input into the Flow-guided Landmark Attention (FLA) module for feature weighting.

[0062] Let the first The motion amplitude corresponding to each key point is Local texture response is Then, the keypoint comprehensive response is calculated based on both motion and texture information, and the attention weight corresponding to each keypoint is calculated using the Softmax function, which is expressed as follows:

[0063] in, Indicates the first Attention weights corresponding to each key point; Indicates the total number of key points; This represents the weighting coefficient between motion response and texture response. .

[0064] In this invention, The preferred value is 0.7. When When the value is large, the attention weight focuses more on the motion response of key points; when When the value is small, the attention weights focus more on local texture responses. Those skilled in the art can adjust the parameters according to different datasets or application scenarios. Adjustments may be made, but this invention does not limit the scope of the invention.

[0065] After obtaining the attention weights for each keypoint, the sparse optical flow features corresponding to all keypoints are weighted and fused to obtain the optical flow-guided keypoint attention output features, which are represented as follows:

[0066] Substituting the attention weights, it can be further expressed as:

[0067] in, This represents the micro-expression motion feature representation output by the optical flow-guided keypoint attention module. Indicates the first The sparse optical flow motion feature vectors corresponding to each key point.

[0068] After feature fusion, the resulting motion features are output as the micro-expression motion representation result and input into the subsequent micro-expression recognition network to complete classification. In this invention, the subsequent recognition network can adopt a convolutional neural network, a Transformer network, or other neural network models capable of performing micro-expression classification tasks; this invention does not limit this approach.

[0069] In this invention, publicly available micro-expression datasets CASME II, SAMM, and SMIC are used as experimental datasets. Starting and target frames are extracted based on the annotation information provided by each dataset and used as inputs to the method of this invention. Following the steps described above, CLAHE enhancement, keypoint detection, sparse optical flow estimation, keypoint sparse optical flow feature construction, and optical flow-guided keypoint attention calculation are sequentially performed to finally obtain a micro-expression motion feature representation.

[0070] During model training, the AdamW optimizer was used for parameter optimization, with an initial learning rate of 0.0001, a batch size of 64, and 100 training epochs. After model training, test set samples were input into the trained recognition network for classification prediction, and UF1 (Unweighted F1-score) and UAR (Unweighted Average Recall) were used as evaluation metrics to assess the model's recognition performance.

[0071] The micro-expression motion features obtained by the above implementation method can be used as input features for tasks such as micro-expression recognition, learner emotion analysis, and intelligent teaching feedback.

[0072] This invention first utilizes CLAHE enhancement to improve the discriminability of local texture details in low-quality facial images, providing clearer input for subtle motion analysis. Second, by constructing keypoint sparse optical flow features that fuse motion displacement, motion amplitude, motion direction, and local texture response, it achieves a refined representation of micro-expression motion information. Furthermore, it introduces an optical flow-guided keypoint attention mechanism, adaptively allocating attention weights based on keypoint motion response and texture changes, effectively highlighting sensitive areas of micro-expressions and suppressing background interference, thereby obtaining more discriminative micro-expression motion features. Experimental results show that this invention can effectively improve the recognition performance of micro-expression recognition models on public datasets such as CASME II, SAMM, and SMIC, demonstrating good robustness, generalization ability, and engineering application value.

[0073] Example 3, (1) Dataset Description This invention uses publicly available micro-expression datasets CASME II SAMM and SMIC for experimental verification.

[0074] The CASME II dataset, collected by high-speed cameras, contains 247 micro-expression samples, covering emotion categories such as happiness, disgust, surprise, and repression; the SAMM dataset, collected by 32 subjects, contains 159 micro-expression video samples; and the SMIC dataset contains 164 micro-expression samples, covering three emotion categories: positive, negative, and surprise.

[0075] For each micro-expression sample, the onset frame and the apex frame are first extracted based on the dataset annotations, and then used as input to the method of this invention. The onset frame represents the beginning state of the expression, and the apex frame represents the most intense state of the expression. The two frames contain complete micro-expression motion information.

[0076] To ensure the fairness of the experiment, all input images were uniformly scaled to 224×224 pixels and face alignment was performed.

[0077] (2) Comparison Model Explanation To objectively verify the effectiveness of the proposed micro-expression motion feature representation method based on CLAHE enhancement and optical flow-guided keypoint attention, the experiment selected representative micro-expression recognition methods published in recent years as comparison models. The selected methods cover the main technical routes in the current micro-expression recognition field, including graph structure modeling, spatiotemporal feature learning, self-supervised pre-training, Transformer modeling, and pre-trained language model transfer, such as AMAN, Graph-TCN, GEME, SMA-STN, MTMNet, MiMaNet, MAE, and μ-BERT. This comprehensive approach reflects the development level of existing technologies and verifies the performance advantages of the present invention compared to different types of methods.

[0078] Among these methods, AMAN and GEME primarily model micro-expression local motion information. Graph-TCN utilizes graph convolutional networks to learn the spatial topological relationships between facial key points. SMA-STN combines spatial motion attention with spatiotemporal feature learning to enhance the representation ability of key regions. MTMNet and MiMaNet employ multi-task learning and multi-scale feature fusion strategies, respectively, to improve model performance. MAE and μ-BERT introduce self-supervised pre-training and Transformer pre-training concepts to improve the model's ability to express weak motion features. These methods cover the main technical routes in the current field of micro-expression recognition, including graph structure modeling, spatiotemporal feature learning, multi-scale feature fusion, self-supervised learning, and pre-trained models, and can comprehensively reflect the current development level of technology.

[0079] The aforementioned comparative methods were validated on publicly available micro-expression datasets CASME II, SAMM, and SMIC, and performance comparisons were conducted using the same data partitioning method and evaluation metrics UF1 and UAR as this invention, ensuring the fairness and comparability of the experimental results. By comparing with the aforementioned representative existing methods, the effectiveness and advancement of the proposed CLAHE enhancement, Sparse Optical Flow (SLOF) motion modeling, and Optical Flow Guided Keypoint Attention (FLA) mechanisms in micro-expression motion feature representation can be comprehensively verified.

[0080] (3) Micro-expression movement characteristic characterization experiment Based on the aforementioned dataset, the starting frame and peak frame were input into the micro-expression motion feature representation framework proposed in this invention. The experimental environment was implemented using the PyTorch deep learning framework and trained using an NVIDIA A100 GPU. The model was trained using the AdamW optimizer, with an initial learning rate of 0.0001, a batch size of 64, and a training period of 100 epochs. UF1 and UAR were used as evaluation metrics. Tables 1, 2, and 3 present the experimental results of different methods on the publicly available micro-expression datasets CASME2, ​​SAMM, and SMIC, respectively. Table 1. Performance comparison results on the CASME2 dataset.

[0081] Table 2 shows the performance comparison results on the SAMM dataset.

[0082] Table 3 Performance comparison results on the SMIC dataset

[0083] As shown in Tables 1, 2, and 3, the micro-expression motion feature representation method based on CLAHE enhancement and optical flow-guided keypoint attention proposed in this invention achieves the best recognition performance on three publicly available micro-expression datasets: CASME II, SAMM, and SMIC. Specifically, on the CASME II dataset, the UF1 and UAR of the proposed method reach 84.6% and 83.9%, respectively, representing improvements of 3.4 and 3.5 percentage points compared to the best-performing comparison method, μ-BERT. On the SAMM dataset, UF1 and UAR reach 81.3% and 80.2%, respectively, representing improvements of 2.8 and 3.1 percentage points compared to μ-BERT. On the SMIC dataset, UF1 and UAR reach 84.2% and 83.6%, respectively, representing improvements of 2.7 and 2.8 percentage points compared to μ-BERT. Experimental results demonstrate that the proposed method maintains stable performance advantages on datasets of different sizes, acquisition environments, and classifications, exhibiting good generalization ability and robustness.

[0084] In another embodiment of the present invention, a micro-expression motion feature representation system is provided, which can be used to implement the above-mentioned micro-expression motion feature representation method. Specifically, the system includes: The image enhancement module is used to acquire the start frame image and target frame image in the micro-expression video sequence, and perform contrast-limited adaptive histogram equalization (CLAHE) on the input image to obtain the enhanced continuous frame image. The key point detection module is used to input the enhanced continuous frame images into the face key point detection module, locate the face region, extract the coordinate information of the facial key points, and establish the correspondence between key points between continuous frames. The optical flow-guided keypoint attention module is used to calculate the motion displacement of each keypoint in the time dimension by utilizing the correspondence between keypoints between consecutive frames, obtain the motion trajectory of the keypoints, and construct the sparse optical flow features of the keypoints. The key point motion feature acquisition module is used to input the sparse optical flow features of the key points into the motion feature extraction module, encode the motion amplitude, motion direction and local motion response of the key points, and obtain the key point motion feature representation. The output module is used to construct the optical flow-guided key point attention module. Based on the key point motion response information, it adaptively calculates the attention weight of each key point, weights and enhances the motion features, aggregates the weighted key point motion features to form a micro-expression motion feature representation, and inputs it into the micro-expression recognition network to complete the learner's micro-expression classification and emotion recognition.

[0085] All relevant content regarding the steps involved in the aforementioned embodiments of the micro-expression motion feature representation method can be referenced to the functional descriptions of the corresponding functional modules of the micro-expression motion feature representation system in the embodiments of the present invention, and will not be repeated here. The module division in the embodiments of the present invention is illustrative and is merely a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of the present invention can be integrated into a processor, exist as separate physical entities, or have two or more modules integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0086] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a micro-expression motion feature representation method.

[0087] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the micro-expression motion feature representation method in the above embodiments.

[0088] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for representing micro-expression motion features, characterized in that, Includes the following steps: The starting frame image and the target frame image in the micro-expression video sequence are obtained. The input image is subjected to contrast-limited adaptive histogram equalization (CLAHE) to obtain the enhanced continuous frame image. The enhanced consecutive frame images are input into the face key point detection module to locate the face region, extract the coordinate information of the facial key points, and establish the correspondence between key points between consecutive frames. By utilizing the correspondence between key points in consecutive frames, the motion displacement of each key point in the time dimension is calculated to obtain the motion trajectory of the key points and construct the sparse optical flow features of the key points. The sparse optical flow features of the key points are input into the motion feature extraction module to encode the motion amplitude, motion direction and local motion response of the key points to obtain the motion feature representation of the key points. An optical flow-guided keypoint attention module is constructed, and the attention weights of each keypoint are adaptively calculated based on the motion response information of the keypoints to enhance the motion features. The weighted motion features of key points are aggregated to form a micro-expression motion feature representation, which is then input into a micro-expression recognition network to complete the learner's micro-expression classification and emotion recognition.

2. The method for representing micro-expression motion features according to claim 1, characterized in that, The process of acquiring the start frame image and target frame image from the micro-expression video sequence, and performing contrast-limited adaptive histogram equalization (CLAHE) on the input image to obtain enhanced continuous frame images includes: The input image is converted to a color space that separates luminance and color, and only the luminance component is enhanced. The luminance image is divided into multiple local sub-regions, and the gray-level histogram of each sub-region is calculated. A contrast limit threshold is set, and the histograms of each local sub-region are cropped, with the excess parts redistributed evenly to each gray level. The gray-level mapping of each sub-region is completed according to the cumulative distribution function, and bilinear interpolation is used to smoothly fuse adjacent sub-regions. The enhanced luminance information and color information are then re-fused to obtain the enhanced image.

3. The method for representing micro-expression motion features according to claim 1, characterized in that, The step of inputting the enhanced consecutive frame images into the facial key point detection module to locate the face region, extract the coordinate information of facial key points, and establish the correspondence between key points between consecutive frames includes: The face region in the input image is located using a face detection algorithm; the detected face region is geometrically normalized; face key points are extracted using a face detection model based on histogram of oriented gradients and a key point localization model based on ensemble regression tree; and the coordinates of the face key points corresponding to consecutive frames are obtained.

4. The method for representing micro-expression motion features according to claim 1, characterized in that, The process of calculating the motion displacement of each key point in the time dimension by utilizing the correspondence between key points in consecutive frames, obtaining the motion trajectory of the key points, and constructing sparse optical flow features of the key points includes: The motion matching relationship between corresponding keypoints in consecutive frames is established using the Lucas-Kanade sparse optical flow algorithm; the motion displacement of each keypoint in the horizontal and vertical directions is calculated; the motion amplitude and direction of the keypoints are calculated based on the horizontal and vertical displacements; the local texture response is calculated based on the grayscale changes of the enhanced consecutive frame images in the neighborhood of the keypoints; the spatial position, motion displacement, motion direction, motion amplitude, and local texture response of the keypoints are jointly constructed as the sparse optical flow features of the keypoints; and the pyramid optical flow strategy is used to estimate the motion of keypoints at different scales.

5. The method for representing micro-expression motion features according to claim 1, characterized in that, The motion feature extraction module includes: encoding the motion information of each key point using the sparse optical flow features of key points as input; learning the motion response features and local texture change features of key points respectively; fusing the motion response information and local texture information to obtain the motion feature representation of key points describing the micro-expression motion state; and mapping the motion features of each key point to the same feature space.

6. The method for representing micro-expression motion features according to claim 1, characterized in that, The optical flow-guided keypoint attention module includes: calculating the motion saliency of each keypoint based on keypoint motion response information; constructing comprehensive keypoint response features by combining local texture response information; calculating the attention weights corresponding to each keypoint using an attention mechanism to achieve adaptive allocation of keypoint importance; weighting the keypoint motion features according to the attention weights to enhance the main activation areas of micro-expressions such as eyebrows, eyes, and corners of the mouth, and reduce the feature responses of low-motion areas such as the bridge of the nose and cheeks; and outputting the keypoint motion features enhanced by optical flow-guided attention. The attention weights are calculated based on the keypoint motion amplitude and local texture response, and are obtained using the Softmax function. The attention weights corresponding to each keypoint are determined according to the weighting coefficient between motion information and texture information, and the value range of the weighting coefficient is 0 to 1.

7. The method for representing micro-expression motion features according to claim 1, characterized in that, The micro-expression feature output includes: aggregating the weighted motion features of all key points to form an overall micro-expression motion representation; inputting the aggregated motion representation into a micro-expression recognition network for feature learning; outputting classification results corresponding to different micro-expression categories; and obtaining the learner's emotional state recognition result based on the classification results.

8. A micro-expression motion feature representation system, characterized in that, include: The image enhancement module is used to acquire the start frame image and target frame image in the micro-expression video sequence, and perform contrast-limited adaptive histogram equalization (CLAHE) on the input image to obtain the enhanced continuous frame image. The key point detection module is used to input the enhanced continuous frame images into the face key point detection module, locate the face region, extract the coordinate information of the facial key points, and establish the correspondence between key points between continuous frames. The optical flow-guided keypoint attention module is used to calculate the motion displacement of each keypoint in the time dimension by utilizing the correspondence between keypoints between consecutive frames, obtain the motion trajectory of the keypoints, and construct the sparse optical flow features of the keypoints. The key point motion feature acquisition module is used to input the sparse optical flow features of the key points into the motion feature extraction module, encode the motion amplitude, motion direction and local motion response of the key points, and obtain the key point motion feature representation. The output module is used to construct the optical flow-guided key point attention module, which adaptively calculates the attention weight of each key point based on the key point motion response information, and performs weighted enhancement of motion features. The weighted motion features of key points are aggregated to form a micro-expression motion feature representation, which is then input into a micro-expression recognition network to complete the learner's micro-expression classification and emotion recognition.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a micro-expression motion feature representation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the micro-expression motion feature representation method as described in any one of claims 1 to 7.

Images