Lightweight long video micro-expression recognition method and system

Abstract

The invention provides a lightweight long video micro-expression recognition method and system, and belongs to the field of computer vision, and the method comprises the steps: obtaining video data, and carrying out the first preprocessing to obtain first video data; a micro-expression interval determination model is adopted to extract deep spatial-temporal characteristics in the first video data, a confidence score of a micro-expression is determined for the deep spatial-temporal characteristics based on an intersection pseudo-label generation method, and a micro-expression interval is determined according to the confidence score; performing second preprocessing on the video data to obtain second video data; and based on a micro-expression recognition model, optimizing and recognizing a micro-expression occurrence position by adopting the micro-expression interval, and recognizing a micro-expression for the second video data. According to the method, the occurrence interval of the micro-expression can be accurately positioned, the subtle change of the micro-expression between different frames and the time sequence relevance are mined, the recognition precision is high, the model parameter quantity is remarkably reduced, and the reasoning speed is increased.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a lightweight long video micro-expression recognition method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art. Micro-expression annotation in long videos relies on professional psychologists, which is time-consuming and labor-intensive, and the annotation results may be unreliable due to disagreements among experts.

[0003] Existing micro-expression analysis methods often separate detection and recognition. Most are based on traditional algorithms designed for short videos, making them difficult to adapt to longer videos. Furthermore, the interweaving of micro-expressions and macro-expressions in long videos increases the difficulty of recognition. The small number of micro-expression samples in long videos, the large number of neutral frames and irrelevant head movements, and the inclusion of macro-expression samples lead to a lack of training data, which impacts model training effectiveness. Furthermore, micro-expressions are extremely short-lived and subtle, making it difficult for traditional image processing methods or deep neural networks to effectively capture these subtle facial changes, thus compromising recognition accuracy and robustness.

[0004] In addition, existing micro-expression recognition algorithms generally have problems with complex network structure and large number of parameters, which makes it difficult for these algorithms to run efficiently on platforms with less computing power. They are even more inefficient when used to process long videos and difficult to achieve real-time performance. Summary of the Invention

[0005] To overcome the shortcomings of the above-mentioned prior art, the present invention proposes a lightweight long video micro-expression recognition method and system, and proposes a lightweight and efficient long video micro-expression recognition model. The model designs a TDA-3D DenseNet module and constructs an ME-MobileNetV3 network. The model detects micro-expression intervals from long videos and performs category recognition, aiming to solve the problem of difficult to distinguish micro-expression and macro-expression interweaving and low recognition accuracy in long video micro-expression recognition.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In the first aspect, a lightweight long video micro-expression recognition method is disclosed, comprising: Acquire video data and perform first preprocessing to obtain first video data; extracting deep spatiotemporal features from the first video data using a micro-expression interval determination model, determining a micro-expression confidence score for the deep spatiotemporal features using an intersection-based pseudo-label generation method, and determining a micro-expression interval based on the confidence score; performing a second preprocessing on the video data to obtain second video data; The micro-expression occurrence location is optimized and identified using the micro-expression interval based on the micro-expression recognition model, and micro-expressions are recognized on the second video data.

[0007] Secondly, a lightweight long video micro-expression recognition system is disclosed, including: A data acquisition and preprocessing module is configured to: acquire video data and preprocess it to obtain first video data; a micro-expression interval detection module configured to: extract deep spatiotemporal features from the first video data using a micro-expression interval determination model, determine a micro-expression confidence score for the deep spatiotemporal features using an intersection-based pseudo-label generation method, and determine a micro-expression interval based on the confidence score; a feature extraction module, configured to: perform second preprocessing on the video data to obtain second video data; The micro-expression recognition module is configured to: use the micro-expression interval optimization to identify the location of micro-expressions based on the micro-expression recognition model, and recognize micro-expressions for the second video data.

[0008] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a TDA-3D DenseNet module, which effectively captures spatial and temporal information through a three-dimensional dense convolutional network (3D DenseNet) to perform deep spatiotemporal feature extraction. In combination with a new TDANet attention mechanism, the module automatically selects important temporal and spatial regions in the spatiotemporal feature graph, effectively extracts key features in the spatiotemporal dimensions, and enhances sensitivity to micro-expression intervals, which helps to accurately detect the occurrence intervals of micro-expressions.

[0009] In order to reduce the computational complexity and parameter count of the model and further improve recognition accuracy, this paper constructs a lightweight ME-MobileNetV3 network in the micro-expression recognition part, introduces the nonlinear activation function Mish, and proposes the MEBneck module, which enhances the network's ability to extract features of specific facial areas without adding additional parameters, while maintaining the model's ability in multi-scale spatial and temporal feature weighting.

[0010] This paper introduces the temporal consistency loss function into the long video micro-expression recognition network for the first time. By enhancing the temporal stability of micro-expression intervals, the accuracy of interval positioning is improved, thereby improving the overall recognition accuracy.

[0011] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0013] Figure 1 This is a flow chart of the lightweight long video micro-expression recognition method described in Example 1 of the present invention.

[0014] Figure 2 This is a schematic diagram of the DenseNet dense connection mechanism described in Example 1 of the present invention.

[0015] Figure 3 This is a structural diagram of TDANet described in Example 1 of the present invention.

[0016] Figure 4 This is a comparison diagram of the activation functions described in Example 1 of the present invention.

[0017] Figure 5 This is a structural diagram of the SimAM attention mechanism described in Example 1 of the present invention.

[0018] Figure 6 This is the confusion matrix of the lightweight long video micro-expression recognition method described in Example 1 of the present invention.

[0019] Figure 7 This is the confusion matrix of the MEAN method described in Example 1 of the present invention. DETAILED DESCRIPTION

[0020] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in this embodiment have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0021] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0022] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0023] Example 1 In one or more embodiments, a lightweight long video micro-expression recognition method is disclosed. The process of this method is divided into the steps of pre-processing, micro-expression interval detection, and micro-expression recognition. The specific process is as follows Figure 1As shown. In the preprocessing stage, face detection and cropping are first performed to obtain an image containing only the face, and 68 facial key points are detected. Then, the TV-L1 optical flow method is used to extract optical flow features. Then, the region of interest is selected on the feature map, and the image is resampled. The resampled image is subjected to feature splitting for subsequent detection and recognition processes. In the micro-expression interval detection stage, a TDA-3D DenseNet module is designed, which extracts deep spatiotemporal features from the video through a densely connected structure. To further improve the accuracy of detection, an intersection-based pseudo-labeling technique is adopted, combined with a sliding window to scan the video, calculate the confidence score of each window, and finally obtain a pseudo-label set for loss calculation, thereby accurately detecting the occurrence interval of micro-expressions. In the micro-expression recognition stage, a lightweight ME-MobileNetV3 network is constructed to improve the real-time and efficiency of the network. The Softmax function is used to calculate the probability of micro-expression classification, and the focal loss function is used to optimize the recognition results, thereby achieving accurate classification and recognition of micro-expressions. The lightweight long video micro-expression recognition method includes the following steps: Step S1: Acquire video data and pre-process to obtain first video data.

[0024] In this embodiment, the acquired video data is preprocessed, and the preprocessing process is divided into three main steps: facial key point detection and cropping, optical flow feature extraction and resampling to obtain first video data.

[0025] Step S1-1: Obtain long video data to be detected.

[0026] Step S1-2, performing facial key point detection and cropping on the video data; In this example, the Dlib toolkit is used to detect the face from the first frame of a video and further locate 68 facial landmarks, such as the eyes, nose, mouth, and facial contour. These landmarks are then used to extract the facial region of interest (ROI), typically focusing on the eyes, eyebrows, and mouth. This step ensures that subsequent processing focuses on the facial region, avoiding background or other distracting factors.

[0027] Facial landmark detection is the first step in micro-expression analysis, which aims to accurately locate facial regions for subsequent feature extraction.

[0028] After facial keypoint detection, a cropping operation is performed to crop the facial region in each frame of video to 128×128 pixels based on the detected keypoints. This reduces computational effort while allowing the model to focus on changes in the facial region. The cropped facial region serves as input for subsequent analysis, ensuring the model is not distracted by background and irrelevant areas, improving computational efficiency and accuracy.

[0029] Step S1-3, optical flow feature extraction; In this embodiment, the TV-L1 optical flow method is used to calculate the optical flow features, including the horizontal component ( u ) and the vertical component ( v Optical flow is a technique that describes the movement of pixels between consecutive frames and can capture subtle facial movements. Based on the optical flow components, optical strain is further calculated, a measure of the intensity of subtle facial movements. Optical strain is obtained by calculating the derivatives of the optical flow components and can reflect subtle changes in facial muscles.

[0030] For a micro-expression sequence ,make Indicates the image at pixel point at time t The brightness value at time . The moving distance within is , according to the principle of constant brightness, it can be considered that the brightness value of the pixel remains unchanged before and after the movement, that is, .make , , the optical flow guidance features are calculated using the starting frame and the vertex frame. The optical flow field calculated from these two frames can be expressed as a tuple: (1) Where X and Y represent the frame The width and height, and express The horizontal and vertical components of .

[0031] Since strain patterns are only related to facial deformation and are not easily affected by factors such as lighting conditions and facial occlusion, they perform well in micro-expression recognition tasks. Therefore, given an optical flow vector, optical strain can be derived to describe facial motion patterns. For sufficiently small facial pixel movements, optical strain can represent the deformation magnitude of facial muscle tissue. Optical strain is expressed as follows: (2) in, is the displacement vector, which represents the projection of the displacement caused by the deformation of the facial expression in the three-dimensional space onto the two-dimensional image; represents the derivative of u.

[0032] Expand formula (1) into matrix form: (3) Among them, the diagonal strain component is the normal strain component, is the shear strain component.

[0033] Specifically, normal strain measures the change in length perpendicular to a specific direction (such as the x or y direction), while shear strain measures the change in angular displacement within a plane due to shear force. Since muscle movement during micro-expression movements may involve multiple directions, the optical strain magnitude of each pixel can be calculated by the sum of the squares of the normal strain component and the shear strain component, using the formula: (4) For the detection network of micro-expression intervals, the selection value is adopted The first optical flow feature between every two frames is calculated using the _ _ method, where FPS is the number of frames per second of a micro-expression. Generally, a micro-expression can last for 0.2 seconds, so the number of frames within 0.1 seconds (half the duration of a micro-expression) is used to define the selection value.

[0034] Preferably, after optical flow feature extraction, this embodiment also undergoes feature splicing. Specifically, the horizontal component u , vertical component v and light strain They are respectively used as information of different channels and stacked in the channel dimension to form comprehensive features.

[0035] Step S1-4, resampling; To reduce computational complexity and preserve key information, this example processes the integrated feature image based on regions of interest (ROIs), adding 12-pixel edges to three key regions: ① left eye and left eyebrow; ② right eye and right eyebrow; and ③ mouth. Furthermore, to minimize computational cost, this example utilizes image resampling techniques, manually resizing regions ① and ② to 21×21 pixels, and region ③ to 21×42 pixels. This method allows the three regions to be combined into a single 42×42 pixel image. In subsequent processing, the optical flow features are applied across all channels after the split, ensuring efficient execution of these processing steps.

[0036] Preferably, after image resampling, this embodiment performs feature segmentation on the resampled feature image to obtain the first video data. Specifically, channel segmentation is performed according to the feature type. u , vertical component v and light strain They are extracted separately to form different feature channels.

[0037] These steps ensure that the model can focus on key areas and effectively extract deep spatiotemporal features, thereby improving the detection and recognition performance of micro-expressions.

[0038] Step S2: extracting deep spatiotemporal features from the first video data using a micro-expression interval determination model, determining a confidence score of a micro-expression for the deep spatiotemporal features using an intersection-based pseudo-label generation method, and determining a micro-expression interval based on the confidence score.

[0039] Step S2-1: Use a micro-expression interval determination model to identify and locate the occurrence time of micro-expressions from the video and extract deep spatiotemporal features.

[0040] Micro-expressions are typically short-lived and subtle, so accurately capturing their spatiotemporal characteristics is crucial. To efficiently detect micro-expression intervals, this embodiment proposes the use of the TDA-3D DenseNet module as a micro-expression interval determination model. This module uses a three-dimensional dense convolutional network (3D DenseNet) for deep spatiotemporal feature extraction and incorporates the novel TDANet attention mechanism. This effectively extracts key features across both spatiotemporal and temporal dimensions, enhancing sensitivity to micro-expression intervals.

[0041] The TDA-3D DenseNet module specifically includes a first 3D DenseNet module, a first TDANet attention mechanism, a second 3D DenseNet module, a second TDANet attention mechanism, a third 3D DenseNet, a third TDANet attention mechanism, and a fourth 3D DenseNet connected in sequence. The first feature is obtained through the above structure, and the first feature is obtained through a convolutional layer, a maximum pooling layer and a fully connected layer to obtain a deep spatiotemporal feature.

[0042] The input of micro-expression interval detection is the frame data of the video. The size of each frame is usually 42×42×3, and it contains pixel information of the facial expression area. Here, each input frame contains not only spatial information, but also the correlation of the time series. In the task of micro-expression interval detection, the continuity and temporal characteristics of video frames are very important. Based on this, this embodiment uses the 3D DenseNet module to extract deep spatiotemporal features. Compared with traditional 2D convolution, the 3D convolution in the 3D DenseNet module can capture spatial and temporal information at the same time, which is particularly suitable for processing temporal features in video data. The 3D convolution layer slides the convolution kernel in the spatial and temporal dimensions to extract the spatiotemporal features between each frame and its adjacent frames. Each layer of the DenseNet module receives the features of all previous layers. This dense connection method can effectively learn deep spatiotemporal features, thereby improving the accuracy of micro-expression detection. The dense connection mechanism of the DenseNet module, such as Figure 2 shown.

[0043] In this embodiment, based on the use of 3D DenseNet module to extract deep spatiotemporal features, the TDANet attention mechanism is introduced to perform weighted processing on the extracted features. The structure of the TDANet attention mechanism is as follows: Figure 3 As shown in Figure 1, it combines a channel attention module (CAM), a spatial attention module (SAM), and a temporal attention module (TAM) for feature extraction. Channel attention helps the model automatically learn which feature channels are most sensitive to micro-expressions and subsequently adjusts the channel weights of the feature map. Spatial attention allows the model to focus on subtle changes in facial regions, enhancing attention to areas where micro-expressions occur, particularly subtle changes in expression. Temporal attention weights each frame, helping the model focus on key moments where micro-expressions occur, thereby enhancing sensitivity to temporal features. After spatiotemporal feature extraction and attention weighting, the network further processes the features through a series of convolution and pooling operations, and outputs a confidence score through a fully connected layer. Ultimately, the model outputs a confidence score indicating whether each video clip contains a micro-expression and the time interval in which the micro-expression occurred. The combination of the spatiotemporal feature extraction capabilities of 3D convolution and the triple attention mechanism enhances the model's sensitivity to micro-expression intervals, more effectively capturing brief and subtle changes in expression within video sequences, thereby improving the accuracy and robustness of micro-expression detection.

[0044] Specifically: Channel Attention Model (CAM): Channel attention adjusts the weights of different channels to strengthen those feature channels with high discrimination for micro-expression recognition. For example, in facial expression feature extraction, texture information and edge features are often key to micro-expression recognition. Channel attention can help the model automatically identify and enhance these features, thereby improving the recognition ability of micro-expressions. The channel attention module is defined as: (5) in, is the input feature map, and The feature maps are Perform maximum pooling and average pooling operations. Maximum pooling is to take the maximum value in the pooling window, and average pooling is to calculate the average value within the pooling window. Through these two pooling operations, key information of the feature map can be obtained from different angles. It is a multi-layer perceptron, which is used here to perform nonlinear transformation on the pooled features, learn the correlation between different channels, map the pooled features to a low-dimensional space, and then map them back to the original channels. Sigmoid activation function maps the two results of the addition after multi-layer perceptron processing to the interval (0, 1) to obtain the channel attention weight , in order to adjust the weights of different channels and strengthen the feature channels with higher discrimination for micro-expression recognition.

[0045] Spatial Attention Model (SAM): Spatial attention emphasizes highly distinguishable regions of the image by weighting the spatial position of the input image. In the task of detecting micro-expression intervals, subtle changes in facial regions are particularly important. Spatial attention can guide the network to focus on details in the facial region, such as small movements of the corners of the mouth. Through this weighting mechanism, the model can more accurately identify changes in these key regions, thereby improving the accuracy of micro-expression detection. The spatial attention module is defined as: (6) in, is the input feature map, and The feature maps are Perform maximum pooling and average pooling operations, Express Perform convolution operation to extract spatial features. For the Sigmoid activation function, the convolution operation The output result is mapped to the interval (0, 1) to obtain the spatial attention weight , in order to weight the image space position.

[0046] Temporal Attention Model (TAM): Temporal attention dynamically adjusts the importance of different time frames by modeling the temporal dimension of the video. Since micro-expressions often occur in a very short period of time, temporal attention can help the network focus on the key frames or time steps where micro-expressions occur, avoiding the interference of irrelevant frames in the detection process, thereby enhancing the model's sensitivity and accuracy to short-term spatiotemporal changes. The temporal attention module is defined as: (7) in, is the input feature map, It is a multi-head attention mechanism that is used to further extract and assign weights to the features after the pooling operation, enhancing the ability to capture time dimension features so as to more accurately focus on the moment when micro-expressions occur. For the Sigmoid activation function, The output result is mapped to the interval (0, 1) to obtain the time attention weight , so that weighted operations can be performed on different time frames.

[0047] Assume that the input feature map is , Obtained through the CAM module , and Multiplying the elements gives , Obtained through the SAM module , and Multiplying the elements gives , Obtained through the TAM module , and Multiplying the elements gives , and finally 、 、 and the original input features Multiply the corresponding elements to get the output features The definition of TDANet is as follows: (8) Step S2-2: Determine the confidence score of the micro-expression based on the deep spatiotemporal features using a pseudo-label generation method based on intersection, and determine the micro-expression interval according to the confidence score.

[0048] Since all long video micro-expression datasets contain the start frame and end frame of each micro-expression sample, this embodiment adopts the intersection-based ( IoU ) to determine the confidence score of each micro-expression sample .

[0049] Specifically, this method generates pseudo labels through sliding window technology and calculates the intersection between each frame and the micro-expression interval (from the start frame to the end frame) ( IoU ) to assign a confidence score to each frame. This score reflects whether each frame belongs to the period when the micro-expression occurs. The specific implementation steps are as follows: By using a length of Sliding window W j Scan and mark each video W j Position frame F j , W j The length corresponds to the interval ,in is half of the average length of micro-expressions, and N represents the average length of each micro-expression dataset. Pseudo-labeling function g Using the Heaviside step function: (9) Among them, the calculation formula of IoU is: (10) in, Represents the true micro-expression interval, that is, the interval from the start frame to the end frame. W represents the interval corresponding to the sliding window. By calculating the confidence score s of each window, the pseudo label set is obtained. , this pseudo label set is used for subsequent loss calculation.

[0050] Use a linear function to transform the activation values z i The predicted micro-expression detection confidence score is obtained from (a continuous value between 0 and 1), the expression is: (11) in, w i is the weight, z i is the activation value of the input, b is the bias term.

[0051] This example effectively measures the overlap between the predicted and true labels by calculating the intersection of the predicted and true intervals, thereby assigning corresponding confidence values to the generated pseudo labels. This method helps improve label quality during training and, in turn, enhances the recognition accuracy of the model.

[0052] Step S3: performing a second preprocessing on the video data to obtain second video data; Perform facial key point detection on the video data and the cropped image in step S1-2, perform optical flow feature extraction, and obtain the horizontal component u and vertical component v of the optical flow and the optical strain component , define the value of the second optical flow feature , extract two frames of optical flow features for micro-expression recognition tasks, where, is the vertex frame, is the starting frame.

[0053] Then, a region of interest is selected on the feature map after feature splicing, and image resampling and feature splitting are performed on the region of interest to obtain the second video data.

[0054] Step S4: Based on the micro-expression recognition model, the micro-expression interval is optimized to identify the location where the micro-expression occurs, and micro-expressions are recognized for the second video data.

[0055] The ME-MobileNetV3 network is used as the micro-expression recognition model. Initially, a conventional 3×3 convolutional layer performs preliminary feature extraction on the second video data to capture essential information from the input image. The features are then passed through an improved attention mechanism module (ME Bneck) and two standard convolutional modules (Bneck) to extract key features highly correlated with micro-expressions. These modules help strengthen the network's response to changes in facial micro-expressions, thereby improving the model's recognition accuracy. Finally, the extracted features are compressed into a single-dimensional vector through a fully connected layer. A softmax layer then outputs the probability of each expression. The expression with the highest probability is selected as the predicted category for the input image.

[0056] Prediction categories include but are not limited to negative, positive, surprise or happiness, disgust, surprise, depression, and other micro-expression categories.

[0057] The specific architecture of the ME-MobileNetV3 network is: MobileNetV3 is a lightweight convolutional neural network designed specifically for mobile devices and resource-constrained environments. It effectively reduces the number of model parameters and computational overhead by using depthwise separable convolutions and a linear bottleneck architecture, resulting in efficient computation and low memory usage. While maintaining good performance, MobileNetV3 achieves outstanding performance in mobile image classification tasks through its unique network structure and optimization techniques. Despite its superior performance among similar models, MobileNetV3's original network structure is not fully suitable for micro-expression recognition. First, micro-expressions are extremely short-lived, with subtle and difficult-to-capture features. MobileNetV3 was optimized on the ImageNet dataset for large-scale image classification tasks and is designed to handle rich and diverse image content. However, micro-expression recognition requires accurate capture of subtle facial changes, which places higher demands on feature extraction. Second, micro-expression datasets are typically small and unevenly distributed, with significant differences from ImageNet. This can lead to wasted computational resources and the risk of overfitting in MobileNetV3, limiting the effective learning of micro-expression features. In addition, the Hardswish activation function used by MobileNetV3 has a high computational cost. In micro-expression recognition tasks with high real-time requirements, it may lead to excessive consumption of computing resources and affect the efficiency of the model.

[0058] In order to solve the above problems and further improve the adaptability of the network and its performance in micro-expression recognition, this embodiment constructs an ME-MobileNetV3 network, which specifically includes a sequentially connected convolutional layer, a first ME Bneck module, a first Bneck module, a second Bneck module, a second ME Bneck module, a third Bneck module, a fourth Bneck module, a third ME Bneck module, a fifth Bneck module, a sixth Bneck module, a fourth ME Bneck module, a seventh Bneck module, an eighth Bneck module, a ninth Bneck module, a fully connected layer, a full-scale maximum pooling layer, an activation function layer and an output layer. Among them, the first ME Bneck module, the first Bneck module, the second Bneck module, the third ME Bneck module, the fifth Bneck module, the sixth Bneck module, the fourth ME Bneck module, the seventh Bneck module and the eighth Bneck module are 3*3 convolution kernels and 3*3 convolution kernels, and the second ME Bneck module, the third Bneck module and the fourth Bneck module are 5*5 convolution kernels.

[0059] First, the number of convolution layers and channels was adjusted, and the network structure was optimized to make it more suitable for the characteristics of micro-expression recognition. The network structure of this embodiment has 13 Bnecks, which is 2 fewer Bnecks than the structure of MobileNetV3, reducing the overall number of network parameters. Secondly, the number of channels after deep convolution in Bneck was expanded, and the dimensionality increase operation after point convolution uniformly increased the number of channels to twice the number of corresponding Bneck output channels. By adjusting the number of convolution layers and channels, the network can better adapt to the characteristics of micro-expression datasets while reducing computational overhead. Finally, the Mish activation function is introduced to replace the Hardswish activation function in MobileNetV3, improving the defect of the high computational overhead of the Hardswish activation function. The formula and derivative of the Mish activation function are: (12) in, x It is the feature output after the convolution operation in the ME-MobileNetV3 network.

[0060] like Figure 4Figure 2 shows a comparison of the Hardswish activation function and the Mish activation function. Based on image analysis of the activation functions, the Mish activation function exhibits smoother characteristics and better preserves information during nonlinear activation. As network depth increases, the Mish activation function captures more detailed features, thereby improving the model's accuracy and generalization. The Mish activation function exhibits excellent smoothness, effectively alleviating the vanishing gradient problem, and its derivative is positive in most cases, enhancing the model's optimization and generalization performance. Using the Mish activation function makes the network more stable during training and improves the accuracy of micro-expression recognition. Through these improvements, ME-MobileNetV3 is better suited to micro-expression recognition tasks while maintaining its lightweight nature, ensuring efficient operation in applications with high real-time requirements.

[0061] Specifically, the improved ME Bneck module: In the original MobileNetV3 network architecture, the SE channel attention mechanism is embedded in the inverted residual structure, positioned after the depthwise convolutional layers and before the pointwise convolutional dimension reduction layers. The core purpose of this design is to enhance the model's feature extraction performance at the channel level. However, this mechanism has certain limitations, as it fails to fully consider spatial positional information. In the specific task of recognizing facial micro-expressions, different types of expressions exhibit significant spatial feature differences across the entire image region. Therefore, the dynamic changes in spatial information play a crucial role in this task. When extracting facial micro-expression features, greater attention should be paid to the information expressed in various facial regions to enhance the model's sensitivity to regional differences. Therefore, this embodiment proposes the ME Bneck module and introduces SimAM attention. Specifically, during the construction of the ME-MobileNetV3 network, the Bneck structure originally using the SE module in MobileNetV3 is replaced with an ME Bneck module that incorporates the SimAM attention mechanism. It can automatically adjust attention allocation according to individual differences and changes in expression, and infer weights from both spatial and channel dimensions without the need to manually adjust the network structure and introduce additional parameters, thereby reducing computational costs and improving model adaptability and generalization capabilities.

[0062] The SimAM attention mechanism belongs to the category of three-dimensional weighted attention mechanism, which organically integrates channel information and spatial information, and can simultaneously carry out weight learning in the channel dimension and the spatial dimension. Without adding additional parameters, the SimAM attention mechanism can prompt the model to pay more attention to the key information in the feature map, effectively capture the correlation between data, and thus improve the overall performance of the model. The structure of the SimAM attention mechanism is as follows Figure 5 shown.

[0063] For each neuron in a channel, SimAM defines an energy function To measure the linear separability between neurons. The specific formula is: (13) in, , , M is the number of neurons on a channel, and are the weights and biases during linear transformation, t and are neurons in a single channel of the input feature and other neurons, respectively, and y is the output value of the neuron (defined as 1 or -1). By deducing the energy function, we can get: (14) (15) in, is the regularization coefficient. is the mean value of the neuron, is the variance of the neuron, and the formula is as follows: (16) (17) By minimizing formula (14) and introducing the regularization term, we can obtain the minimum energy function: (18) The lower the energy value, the greater the difference between the neuron and the surrounding neurons, and the more important it is in attention allocation. Therefore, the importance of the neuron is evaluated by the inverse of the neuron, and the features are enhanced by sigmoid normalization. That is, the minimum energy value of all channels and spatial dimensions is By splicing and integrating, we can form E, the minimum energy set in all channels and spatial dimensions, and Apply the sigmoid function to map its value to the (0, 1) interval to obtain the attention coefficient.

[0064] (19) The generated attention coefficient is used to perform element-by-element multiplication with the original input feature map X to enhance the attention of the feature map, highlight important features, suppress unimportant features, and finally output the feature map after attention enhancement.

[0065] This embodiment optimizes feature extraction and accelerates network convergence. In the Bneck and ME Bneck modules, point-wise convolution is first used to increase feature dimensions, followed by further processing of feature maps using depthwise convolution. Batch normalization and the nonlinear activation function Mish are applied after each convolution operation to improve the model's stability and nonlinear expressiveness, thereby enhancing its robustness and generalization capabilities when processing micro-expressions.

[0066] More importantly, the loss function proposed in this embodiment is divided into interval detection loss and recognition loss. Among them, in the interval detection loss function, the time consistency loss is a new loss function introduced by this method.

[0067] Specifically: (1) Interval detection loss function The interval detection loss optimizes the accuracy of detecting micro-expression locations. By calculating the difference between the generated intervals and the true intervals, the model can pinpoint the exact moment a micro-expression occurs. Micro-expressions typically manifest as subtle changes over time, rather than as isolated expressions in a single frame.

[0068] The mean square error (MSE) loss function and temporal consistency loss are used to calculate the loss of micro-expression interval detection: When training the interval detection network, the pseudo-labeling technology based on IoU is used to determine the pseudo-label of each sample. The MSE loss function is used to measure the difference between the predicted confidence score and the true score, so that the network can learn accurate micro-expression interval features. The mean square error (MSE) loss function formula is: (20) in, is the true confidence score. is the predicted location confidence score. n is the number of samples.

[0069] Temporal consistency loss is used to ensure that the generated micro-expressions remain consistent across consecutive frames. By constraining the feature differences between adjacent frames, the micro-expressions generated by the model have smooth temporal transitions. The temporal consistency (TEM) loss formula is: (twenty one) in, and are the images generated by the i-th frame and the i+1-th frame respectively, m is the number of frames, Represents the square of the L2 norm.

[0070] The MSE loss function and the temporal consistency loss function are combined using a weighted sum. By adjusting the weights (α, β) of each loss term, the contribution of different tasks to the total loss is controlled to achieve optimal results and accurately locate the time interval when micro-expressions occur. The specific formula is as follows: (twenty two) This embodiment introduces a temporal consistency loss function during micro-expression interval detection to ensure that the model maintains continuous temporal consistency at the start and end of micro-expression detection. This helps avoid misjudgment or positioning errors of micro-expression intervals due to local fluctuations or transient errors.

[0071] (2) Identification loss function Recognition loss is used to optimize the classification task of micro-expressions and ensure that the generated micro-expressions can be correctly classified. It is optimized by measuring the difference between the true category label and the probability distribution predicted by the model. This embodiment uses the FocalLoss function to optimize the loss of the micro-expression recognition task. The Focal Loss function dynamically adjusts the sample weight by introducing an adjustable factor, which can reduce the weight of easy-to-classify samples and increase the weight of difficult-to-classify samples, so that the model pays more attention to those difficult-to-classify samples, thereby alleviating the problems caused by category imbalance, so that the model can have better classification effects when facing samples of various categories. The Focal Loss function is specifically defined as: (twenty three) Among them, the sample Micro-expression dataset A micro-expression sequence in The representation model will micro-expression dataset A specific sample in Predicted as class The probability value of Focal Loss is calculated independently for each sample in the dataset. It is a preset weight parameter used to balance the weights of different categories, and the weight is set smaller for negative samples. It is an adjustment parameter, which can be set by The weight of difficult-to-classify samples can be increased, thereby increasing the model's attention to difficult-to-classify samples.

[0072] This example verifies the effectiveness of the proposed lightweight long video micro-expression recognition model through experiments: 1. Dataset In this paper, we select two long video datasets, CAS (ME)² and SAMMLV, to evaluate the performance of the proposed method.

[0073] (1) The SAMMLV dataset contains 133 micro-expressions and 343 macro-expressions from 28 subjects. The SAMMLV dataset provides the start frame, apex frame, and end frame labels of micro-expressions. The SAMMLV dataset includes seven categories of facial expressions, namely disgust, anger, sadness, fear, happiness, surprise, and others.

[0074] (2) The Chinese Academy of Sciences Micro-Expression Dataset CAS(ME)2 consists of 87 long video expression samples and 357 expression samples, including 300 macro expression samples and 57 micro expression samples. CAS(ME) 2 The dataset has four categories of expression labels: positive, negative, surprised, and others.

[0075] In this experiment, the CAS(ME)² and SAMMLV datasets were repartitioned to unify the sample labels. Specifically, "disgust," "anger," "sadness," and "fear" samples were classified as "negative" samples; "happy" samples were classified as "positive" samples; "surprise" samples remained unchanged; and "other" samples could not be classified and were therefore not used.

[0076] 2. Experimental Environment and Setup The machine configuration and development environment used in this experiment are: Ubuntu 20.04; GeForce RTX3090; CUDA 11.0; PyTorch 1.7.1+cu110; Pycharm 2019.3.5; Python 3.7. This paper adopts a two-step learning process to train the architecture. When training the localization network and the shared network together, the Adam optimizer is used with a learning rate of 5×10 -4 , trained for a maximum of 200 epochs. To address the dataset imbalance, the non-ME and ME classes are sampled at an 8:1 ratio. The recognition network training settings are similar, but due to the smaller number of training samples, a maximum of 800 epochs are trained.

[0077] Considering the small sample size of the micro-expression dataset and the large differences among the subjects, this experiment adopts the leave-one-out cross-validation method, that is, each time all samples in the dataset except one group of samples are used as the training set, and then the group of samples is used as the test set; this operation needs to be performed once for each subject, and the experimental results of each subject are averaged to obtain the final experimental results.

[0078] 3. Experimental Evaluation Criteria The evaluation criteria used in this experiment include Recall, Accuracy, F1-score, Precision, Unweighted Average Recall (UAR), Unweighted F1-score (UF1), and the overall evaluation score (Spot-Then-Recognize, STRS).

[0079] For the micro-expression interval detection task, the Recall, Accuracy, and F1-scores are used for evaluation. The True Positive (TP), False Positive (FP), and False Negative (FN) are determined by calculating the Intersection over Union (IoU) between the predicted interval and the true interval. This method uses the evaluation method adopted by MEGC2020 to calculate the detected interval ( ) and the true interval ( ) for comparison, when hour( The number of micro-expression intervals that exist but are not detected in the entire dataset is FN, which is then used to calculate the recall rate, precision rate, and F1 score. The formula is as follows: (twenty four) (25) (26) (27) Here, M is the number of true intervals in the dataset, and N is the number of intervals detected by the method.

[0080] For the micro-expression recognition task, Recall, F1-scorer, Precision, UAR, and UF1 indicators are used for evaluation. Accuracy is the ratio of the number of correctly predicted emotion categories to the total number of predictions, and the formula is: (28) (29) (30) (31) Where C is the total number of categories. UAR and UF1 indicators are used to deal with the problem of category imbalance. The formula is as follows: (32) (33) Where nc is the number of samples in the cth category.

[0081] The performance of the entire network is evaluated using the Spot-Then-Recognize Score (STRS). STRS measures the model's ability to detect micro-expression intervals and correctly classify emotions. It is the product of the F1 score for micro-expression interval detection and the F1 score for micro-expression recognition. The formula is: (34) 4. Experimental Results and Analysis (1) Experimental results of the overall network structure performance The proposed method demonstrates good interval detection and recognition performance on long video micro-expression datasets, as shown in Table 1. On long video datasets such as CAS(ME)² and SAMMLV, the proposed method achieves excellent precision, recall, and F1 scores, achieving a recognition rate of 68.15% on the SAMMLV dataset. Although the number of TPs recognized in long videos is not high, this is because long videos typically contain multiple macro-expressions and only a small number of micro-expressions, and macro-expressions and micro-expressions are intricately intertwined. However, the proposed method still has strong adaptability and is more accurate than most current long video micro-expression recognition methods. Overall, the method proposed in this embodiment is highly robust and effective in long video micro-expression analysis.

[0082]

[0083] (2) Comparative experiments on long video micro-expression datasets Under the same experimental conditions, the proposed method is compared with the current mainstream long video micro-expression detection and recognition methods. Table 2 shows the performance comparison results, the evaluation indicator is F1-score s .

[0084]

[0085] As can be seen from Table 2, the F1-scores of the proposed method in this paper reached 0.1383 on the CAS(ME)2 dataset, which is better than other methods in the table. The F1-score result on the SAMMLV dataset is 0.1423, which is second only to the result of SOFTNet, proving the effectiveness of the proposed method in this paper.

[0086] Table 3 shows the performance comparison between the proposed method and the MEAN method in micro-expression recognition in long videos, with the evaluation index being F1-score. r .

[0087]

[0088] It can be seen from Table 3 that the method proposed in this paper has the advantages of CAS(ME) 2 The F1-scorer on the dataset and SAMMLV dataset reached 0.6921 and 0.7083 respectively, which are better than the MEAN method. Figure 6 CAS(ME) 2 The confusion matrix of the dataset and SAMMLV dataset in this paper’s method, Figure 7 CAS(ME) 2 The confusion matrix of the dataset and SAMMLV dataset on the MEAN method. Figure 6-Figure 7 As shown in the figure, it can be proved that the method proposed in this paper is superior in the task of micro-expression recognition.

[0089] 5. ME-MobileNetV3 Network Performance Evaluation To evaluate the parameter count and runtime efficiency of the proposed ME-MobileNetV3 network in the micro-expression recognition task, we designed performance comparison experiments using different network structures. The results are shown in Table 4. We used the leave-one-out cross-validation (LOSO) method to evaluate runtime efficiency. As shown in the table, compared to the unmodified network, the ME-MobileNetV3 network reduces the number of parameters and shortens runtime while ensuring reliable results.

[0090]

[0091] Experimental results show that the proposed model significantly reduces the number of model parameters and improves the inference speed while maintaining high recognition accuracy, providing an efficient and accurate solution for the detection and recognition of micro-expression intervals in long videos, fully verifying the innovation and effectiveness of the model.

[0092] Example 2 In one or more embodiments, a lightweight long video micro-expression recognition system is disclosed, specifically including: A data acquisition and preprocessing module is configured to: acquire video data and preprocess it to obtain first video data; a micro-expression interval detection module configured to: extract deep spatiotemporal features from the first video data using a micro-expression interval determination model, determine a micro-expression confidence score for the deep spatiotemporal features using an intersection-based pseudo-label generation method, and determine a micro-expression interval based on the confidence score; a feature extraction module, configured to: perform second preprocessing on the video data to obtain second video data; The micro-expression recognition module is configured to: use the micro-expression interval optimization to identify the location of micro-expressions based on the micro-expression recognition model, and recognize micro-expressions for the second video data.

[0093] The descriptions of the various embodiments in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0094] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A lightweight long video micro-expression recognition method, characterized in that: include: Acquire video data and perform first preprocessing to obtain first video data; extracting deep spatiotemporal features from the first video data using a micro-expression interval determination model, determining a micro-expression confidence score for the deep spatiotemporal features using an intersection-based pseudo-label generation method, and determining a micro-expression interval based on the confidence score; performing a second preprocessing on the video data to obtain second video data; The micro-expression occurrence location is optimized and identified using the micro-expression interval based on the micro-expression recognition model, and micro-expressions are recognized on the second video data.

2. A lightweight long video micro-expression recognition method according to claim 1, characterized in that: The first preprocessing includes detecting and cropping facial key points on the video data; Extracting optical flow features from the cropped video data, where the optical flow features include a horizontal component, a vertical component, and optical strain; The following formula is used to calculate the first optical flow feature between every two video frames: Among them, k is the first optical flow feature, FPS is the number of frames per second maintained by the micro-expression; After feature splicing is performed on the first optical flow features, the features are resampled based on the region of interest, and the resampled image is feature-splitting to obtain the first video data.

3. A lightweight long video micro-expression recognition method as claimed in claim 1, characterized in that The micro-expression interval determination model adopts the TDA-3D DenseNet module, which specifically includes a first 3D DenseNet module, a first TDANet attention mechanism, a second 3D DenseNet module, a second TDANet attention mechanism, a third 3D DenseNet, a third TDANet attention mechanism, and a fourth 3D DenseNet connected in sequence. The first feature is obtained through the above structure, and the first feature is obtained through a convolutional layer, a maximum pooling layer and a fully connected layer to obtain a deep spatiotemporal feature.

4. A lightweight long video micro-expression recognition method as claimed in claim 3, characterized in that: The TDANet attention mechanism integrates the channel attention module, spatial attention module and temporal attention module for feature extraction. The TDANet attention mechanism expression is: in, is the input feature map, is the output of the channel attention module, for and The result of element-wise multiplication, is the output of the spatial attention module, for and The result of element-wise multiplication, is the output of the temporal attention module, for and The result of element-wise multiplication, Output of TDANet attention mechanism; The channel attention module is defined as: in, and Perform maximum pooling and average pooling operations on the feature map respectively. is a multi-layer perceptron, is the Sigmoid activation function; The spatial attention module is defined as: in, Express Perform convolution operation; The temporal attention module is defined as: in, It is a multi-head attention mechanism.

5. A lightweight long video micro-expression recognition method as claimed in claim 1, characterized in that The pseudo-label generation method based on intersection determines the confidence score of micro-expressions based on deep spatiotemporal features, and determines the micro-expression interval based on the confidence score as follows: Generate pseudo labels using a sliding window technique and assign a confidence score to each frame by calculating the intersection between each frame and the micro-expression interval; The confidence score is: in, w i is the weight, z i is the activation value of the input, b is the bias term.

6. A lightweight long video micro-expression recognition method as claimed in claim 1, characterized in that The second preprocessing includes detecting and cropping facial key points on the video data; Extracting optical flow features from the cropped video data, where the optical flow features include a horizontal component, a vertical component, and optical strain; The second optical flow feature between every two video frames is calculated using the following formula: Among them, k is the second optical flow feature, is the vertex frame, is the starting frame; After feature splicing of the second optical flow features, the second optical flow features are resampled based on the region of interest, and the resampled image is feature-splitting to obtain the second video data.

7. A lightweight long video micro-expression recognition method as claimed in claim 1, characterized in that: The micro-expression recognition model adopts the ME-MobileNetV3 network, which specifically includes a convolutional layer, a first ME Bneck module, a first Bneck module, a second Bneck module, a second ME Bneck module, a third Bneck module, a fourth Bneck module, a third MEBneck module, a fifth Bneck module, a sixth Bneck module, a fourth ME Bneck module, a seventh Bneck module, an eighth Bneck module, a ninth Bneck module, a fully connected layer, a full-scale maximum pooling layer, an activation function layer and an output layer connected in sequence.

8. A lightweight long video micro-expression recognition method according to claim 1, characterized in that: SimAM attention is introduced in the ME Bneck module, and the SimAM attention measures the linear separability between neurons by defining an energy function; the energy function is minimized and a regularization term is introduced to obtain the minimum energy function, the minimum energy values of all channels and spatial dimensions are integrated to obtain the minimum energy set, and the activation function is applied to the inverse of the minimum energy set to obtain the SimAM attention coefficient.

9. A lightweight long video micro-expression recognition method as claimed in claim 1, characterized in that: The micro-expression interval optimization is used to identify the location of micro-expressions, and the interval detection loss function is introduced in the micro-expression interval detection stage: Among them, α, β are weights; L MSE is the mean square error loss function: Where, is the true confidence score, is the predicted location confidence score, n is the number of samples; L temporal is the temporal consistency loss function: Where, and are the images generated by the i-th frame and the i+1-th frame respectively, m is the number of frames, Represents the square of the L2 norm.

10. A lightweight long video micro-expression recognition system, characterized by: include: A data acquisition and preprocessing module is configured to: acquire video data and preprocess it to obtain first video data; a micro-expression interval detection module configured to: extract deep spatiotemporal features from the first video data using a micro-expression interval determination model, determine a micro-expression confidence score for the deep spatiotemporal features using an intersection-based pseudo-label generation method, and determine a micro-expression interval based on the confidence score; a feature extraction module, configured to: perform second preprocessing on the video data to obtain second video data; The micro-expression recognition module is configured to: use the micro-expression interval optimization to identify the location of micro-expressions based on the micro-expression recognition model, and recognize micro-expressions for the second video data.

Images