Micro-expression recognition method based on double-flow feature fusion

Abstract

The invention relates to a micro-expression recognition method based on double-flow feature fusion, and belongs to the field of computer vision. The method comprises the steps of obtaining N frames of images of a micro-expression video from a start frame to a vertex frame, and calculating an optical flow field between adjacent frames by adopting an optical flow algorithm so as to effectively extract optical flow features of micro-expressions; the method comprises the following steps: acquiring N frames of images of a micro-expression video from a start frame to a vertex frame, converting the N frames of RGB face images into a CIE Lab color space, calculating a pixel difference between two frames of converted Lab images, and extracting pixel stream features of micro-expressions; and inputting the optical flow features and the pixel flow features into a constructed double-flow three-dimensional convolutional network for feature extraction and fusion, and classifying micro expressions. According to the method, the dynamic and subtle changes of the micro-expression are effectively captured by combining the optical flow and the pixel difference characteristics. By integrating spatial and temporal information, richer feature representations are provided. The improved attention mechanism further focuses on fine facial changes, and the accuracy of micro-expression classification is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision, and particularly relates to a micro-expression recognition method based on double-flow feature fusion. BACKGROUND

[0002] Human facial expression, as a kind of non-verbal signal formed by facial muscle contraction and activity, is a key way to show internal emotion and psychological activity. It plays an irreplaceable role in emotion transmission, interpersonal communication and social relationship coordination. Our common traditional facial expression is mostly an explicit expression that can be easily detected by the naked eye. Unlike the traditional facial expression, micro-expression is a facial muscle change with extremely short duration and uncontrollable, which is unconsciously revealed when people try to hide or suppress their true emotions. Numerous research results show that micro-expression contains a large amount of dense emotional information, which can reflect basic emotional states such as anger, fear, sadness and surprise, and has a high value in emotion recognition. Due to the instantaneous and difficult-to-detect characteristics of micro-expression, it has broad application prospects in psychological assessment, law enforcement interrogation, risk behavior recognition, emotional intelligent system and human-computer interaction. Therefore, creating an efficient and accurate micro-expression recognition technology has become an important research task in the field of artificial intelligence and pattern recognition.

[0003] Although deep learning models show strong ability in automatic feature extraction, greatly reducing the workload of manual feature design, in the field of micro-expression recognition which is extremely challenging, the mainstream methods still have obvious shortcomings. One of the core problems is the single dependence on information modalities. Optical flow features can effectively capture the inter-frame motion patterns and show the instantaneous dynamics of muscle contraction, but the calculation process is relatively complex, and it is sensitive to light conditions, and important appearance information is also missing. While the original pixel features retain rich texture, wrinkles, skin stretching and other appearance details, it is difficult to accurately distinguish the pixel changes caused by real subtle expression movements from irrelevant disturbances such as head movement, imaging noise and light fluctuations, and it also lacks accuracy in describing the directionality and timing patterns of movement. To break through the bottleneck of existing micro-expression recognition technology, the key is to design a recognition framework that can effectively and cooperatively fuse motion features and appearance texture features, so as to more comprehensively and stably present the subtle dynamics and appearance changes of micro-expression. At the same time, the framework needs to find a clever balance between computational efficiency and model complexity, to ensure that it can still be efficiently trained under limited data, and has strong generalization ability, and thus realizes more accurate and reliable automatic recognition of micro-expression. SUMMARY

[0004] The key to overcoming the bottlenecks of existing micro-expression recognition technology lies in designing a recognition framework that effectively integrates motion features and surface texture features, thereby more comprehensively and robustly representing the subtle dynamics and appearance changes of micro-expressions. Simultaneously, this framework needs to achieve a delicate balance between computational efficiency and model complexity, ensuring efficient training and strong generalization ability even with limited data, ultimately achieving more accurate and reliable automated recognition of micro-expressions.

[0005] This invention proposes a dual-stream 3D convolutional recognition network that integrates optical flow information and pixel difference information. This network incorporates multiple lightweight structural modules, including standard 3D convolutional blocks, depthwise separable convolutional blocks, SandGlass residual blocks, densely connected modules, and globally separable convolutions. It introduces a dual-channel flow feature modeling mechanism to capture subtle facial muscle movement trends and extract minute changes. Pixel flow and optical flow each form two temporal image data streams, providing rich semantic support for subsequent network learning. The SandGlass3D block combines two depthwise convolution operations with pointwise convolutions and introduces an improved dynamically configurable attention mechanism (DPCBAM), thereby enhancing the dual-stream 3D convolutional network's ability to focus on locally important regions while incorporating nonlinear features. This invention can effectively improve the accuracy and computational efficiency of micro-expression recognition, possessing good generalization ability and promising practical application prospects.

[0006] The technical solution adopted in this invention is: a micro-expression recognition method based on dual-stream feature fusion, comprising the following specific steps:

[0007] Step 1: Obtain the micro-expression dataset by selecting N frames from the start frame to the top frame of the micro-expression video and preprocessing the data.

[0008] Step 2: In the N frames of images from Step 1, the optical flow algorithm is used to calculate the optical flow field between adjacent frames in order to effectively extract the optical flow features of micro-expressions and form an N-1 frame optical flow image sequence.

[0009] Step 3: Convert the N frames of RGB face images from Step 1 to the CIE Lab color space, calculate the pixel difference between the two converted Lab images, extract the pixel stream features of micro-expressions, and form an N-1 frame pixel stream image sequence.

[0010] Step 4: Extract the optical flow features and pixel flow features between the two frames and input them into the constructed dual-stream 3D convolutional network for analysis and recognition to obtain micro-expression classification and recognition. Evaluate the constructed dual-stream 3D convolutional network based on various indicators.

[0011] Specifically, in Step 1, a micro-expression dataset is first selected, and then a new target folder system is constructed to store images for subsequent processing. Each subject, corresponding video segment, and its frame image in the selected dataset are traversed sequentially. Each frame image is read using an image processing tool, and the face region is extracted using a face detection algorithm. The detected face images are then cropped and normalized to standardize the image size to 128×128 pixels. Finally, the processed image files are saved to a preset path according to the classification structure of the original data.

[0012] Specifically, in Step 2, adjacent frames are selected as input pairs for optical flow calculation. The TV-L1 algorithm is used to calculate the optical flow field between each pair of images, thereby obtaining pixel-level motion vectors. The optical strain formula of the TV-L1 algorithm is shown below:

[0013]

[0014] Where, ε xx ε represents the horizontal optical flow strain between two image frames. yy ε represents the vertical optical flow strain between two images. xy The shear optical flow strain between two frames is denoted as .

[0015] Specifically, in Step 3, a linear transformation matrix is ​​used to map the original RGB image to the CIE XYZ color system. The transformation matrix is ​​shown below:

[0016]

[0017] Where X, Y, Z are the three primary color values ​​in the CIE XYZ color space, representing the coordinates of the color in the XYZ space; R, G, B are the normalized RGB values, representing the intensity of the red, green, and blue channels of the input image.

[0018] To ensure the accuracy of subsequent color space conversion and the consistency of normalization processing, a reference white point corresponding to the D65 light source was selected as the normalization benchmark, and the color information of the input image was normalized. The tristimulus value of this white point is: X n =0.95047, Y n =1.00000, Z n = 1.08883. The normalization formula is shown below:

[0019]

[0020] The XYZ values ​​are converted to Lab values ​​using a nonlinear mapping f(t), and L, a, and b are calculated. The formulas are shown below:

[0021]

[0022] The definition of the piecewise function f(t) for the nonlinear transformation is as follows:

[0023]

[0024] Finally, the pixel difference between the two converted Lab images is calculated, and pixel stream features of micro-expressions are extracted. The pixel stream feature calculation process is as follows: Figure 2 As shown, image a is the Lab image of the previous frame, image b is the Lab image of the next frame, and image c is the pixel stream feature image after pixel difference calculation.

[0025] Specifically, in Step 4, the constructed dual-stream 3D convolutional network consists of Conv3D blocks, DWConv3D blocks, SandGlass3D blocks, Dense3D blocks, and GDConv3D blocks. The structure of the constructed dual-stream 3D convolutional network is as follows: Figure 3 As shown.

[0026] The SandGlass3D block design uses two depthwise convolution operations, placed outside of two 1×1×1 pointwise convolutions. Both the first depthwise convolutional layer and the last pointwise convolutional layer employ the ReLU activation function to introduce a non-linear transformation. The specific design is as follows: Figure 4 As shown.

[0027] After the first deep convolutional layer, the serial connection between channel attention and spatial attention in the CBAM module is redesigned into a parallel structure. With this structure, attention calculations in the channel and spatial dimensions are performed independently, applying weighted processing to the input feature maps separately to enhance the expressive power of features in their respective dimensions. Finally, the two different attention features are fused together through addition, thereby extracting the ability of the two-stream 3D convolutional network to capture fine-grained differences in micro-expressions. This invention names this module DPCBAM, and the specific design of the module is as follows: Figure 5 As shown.

[0028] Finally, a Softmax classifier is used to identify and classify the types of micro-expressions. The formula for the Softmax loss is as follows:

[0029]

[0030] Where n represents the total number of samples in the dataset; c represents the total number of categories; y i f represents the true class of the i-th sample; i This represents the feature vector of the i-th sample; Represents the true category y i The weight vector is the basis used by the two-stream 3D convolutional network to determine whether a sample belongs to this category; Represents the true class weight vector With eigenvector f i The angle between them.

[0031] Specifically, the evaluation metrics for the constructed two-stream 3D convolutional network include: accuracy, recall, precision, and F1 score, as shown in the following formulas.

[0032]

[0033]

[0034] In the formula, TP represents the number of true micro-expression categories correctly predicted by the two-stream 3D convolutional network; TN represents the number of other categories correctly predicted by the two-stream 3D convolutional network; FP represents the number of other categories incorrectly predicted as true micro-expression categories by the two-stream 3D convolutional network; and FN represents the number of true micro-expression categories incorrectly predicted as other categories by the two-stream 3D convolutional network.

[0035] The beneficial effects of this invention are as follows: It constructs an efficient and accurate micro-expression recognition system. The dual-stream feature fusion integrates subtle changes in both motion and color dimensions, effectively enhancing the ability of the dual-stream 3D convolutional network to capture fine-grained micro-expression features. An improved attention mechanism, DPCBAM, is proposed, optimizing the channel and spatial attention in CBAM from a traditional serial structure to a parallel structure. This allows the dual-stream 3D convolutional network to independently learn and strengthen salient features in different dimensions, ultimately enhancing the overall feature representation capability through fusion. Combined with the depthwise separable convolution and pointwise convolution modules introduced in the SandGlass3D structure, it effectively reduces the number of parameters and computational complexity of the dual-stream 3D convolutional network while maintaining high recognition performance. It is suitable for deployment on resource-constrained edge devices and has good practicality and scalability. A complete data processing and classification system is constructed. The recognition results are evaluated in multiple dimensions using a standard evaluation index system, verifying the superior performance of this method in multi-category micro-expression recognition tasks and demonstrating good generalization ability and practical application prospects. Attached Figure Description

[0036] Figure 1 This is a flowchart of a micro-expression recognition method based on dual-stream feature fusion according to the present invention;

[0037] Figure 2 A flowchart for calculating pixel stream features;

[0038] Figure 3 The structure diagram of the constructed two-stream 3D convolutional network;

[0039] Figure 4 This is a structural design drawing of a SandGlass 3D block.

[0040] Figure 5 This is a module design diagram for the DPCBAM attention mechanism. Detailed Implementation

[0041] The present invention will be further described below with reference to embodiments and accompanying drawings, but the scope of protection of the present invention is not limited to the scope described.

[0042] Example 1: As Figure 1 As shown, a micro-expression recognition method based on dual-stream feature fusion includes the following specific steps:

[0043] Step 1: Obtain the micro-expression dataset. Select 9 frames from the micro-expression video, from the start frame to the top frame, and preprocess the data. The specific process is as follows:

[0044] Step 1.1: Obtain the micro-expression dataset. Obtain 9 frames of micro-expression video from the start frame to the top frame, ensuring that it has a clear hierarchical directory structure.

[0045] Step 1.2: Construct a new target folder system for storing images to be processed later, which will facilitate data classification and retrieval.

[0046] Step 1.3: Iterate through each subject, corresponding video segment, and frame image in the selected dataset. Use OpenCV image processing tool to read each frame image and use MTCNN face detection algorithm to extract face regions. Crop and normalize the detected face images to standardize the image size to 128×128 pixels in order to unify the input specifications of the dual-stream 3D convolutional network.

[0047] Step 1.4: Save the processed image files to the preset path according to the classification structure of the original data to achieve efficient data organization and management.

[0048] Step 2 uses an optical flow algorithm to calculate the optical flow field between adjacent frames in the 9 frames from Step 1, in order to effectively extract the optical flow features of micro-expressions, forming an 8-frame optical flow image sequence. The specific process is as follows:

[0049] Step 2.1: Select adjacent frames as input pairs for optical flow calculation, and use the TV-L1 algorithm to calculate the optical flow field between each pair of images. The optical strain formula of the TV-L1 algorithm is shown below:

[0050]

[0051] Where, ε xx ε represents the horizontal optical flow strain between two image frames.yy ε represents the vertical optical flow strain between two images. xy The shear optical flow strain between two frames is denoted as .

[0052] Step 2.2: Extract the optical flow features of micro-expressions and form an 8-frame optical flow image sequence.

[0053] Step 3 converts the 9 RGB face images from Step 1 to the CIE Lab color space, calculates the pixel difference between two converted Lab images, extracts the pixel stream features of micro-expressions, and forms an 8-frame pixel stream image sequence. The specific process is as follows:

[0054] Step 3.1: Map the original RGB image to the CIE XYZ color system using a linear transformation matrix. The transformation matrix is ​​shown below:

[0055]

[0056] Where X, Y, Z are the three primary color values ​​in the CIE XYZ color space, representing the coordinates of the color in the XYZ space; R, G, B are the normalized RGB values, representing the intensity of the red, green, and blue channels of the input image.

[0057] Step 3.2: To ensure the accuracy of subsequent color space conversion and consistency of standardization processing, a reference white point corresponding to the D65 light source is selected as the normalization benchmark, and the color information of the input image is normalized. The tristimulus value of this white point is: X n =0.95047, Y n =1.00000, Z n = 1.08883. The normalization formula is shown below:

[0058]

[0059] Step 3.3: Use the nonlinear mapping f(t) to convert the XYZ values ​​to Lab values, and calculate L, a, and b. The formula is shown below:

[0060] L=116·f(Y')-16

[0061] a = 500 * (f(X') - f(Y'))

[0062] b = 200 * (f(Y') - f(Z'))

[0063] The definition of the piecewise function f(t) for the nonlinear transformation is as follows:

[0064]

[0065] Step 3.4: Finally, calculate the pixel difference between the two converted Lab images and extract the pixel flow features of micro-expressions. The pixel flow feature calculation process is as follows: Figure 2 As shown, image a is the Lab image of the previous frame, image b is the Lab image of the next frame, and image c is the pixel stream feature image after pixel difference calculation.

[0066] Step 4: Extract the optical flow features and pixel flow features between the two images and input them into the constructed two-stream 3D convolutional network for analysis and recognition to obtain micro-expression classification and recognition. Evaluate the two-stream 3D convolutional network based on various indicators. The specific process is as follows:

[0067] The two-stream 3D convolutional network constructed in Step 4.1 consists of Conv3D blocks, DWConv3D blocks, SandGlass3D blocks, Dense3D blocks, and GDConv3D blocks. The structure of the constructed two-stream 3D convolutional network is as follows: Figure 3 As shown.

[0068] Step 4.2, the design of the SandGlass3D block uses two depthwise convolution operations, placed outside of two 1×1×1 pointwise convolutions. Both the first depthwise convolutional layer and the last pointwise convolutional layer employ the ReLU activation function to introduce non-linear transformations. The specific design is as follows: Figure 4 As shown.

[0069] Step 4.3: After the first deep convolutional layer, the serial connection of channel attention and spatial attention in the CBAM module is redesigned into a parallel structure. With this structure, attention calculations in the channel and spatial dimensions are performed independently, applying weighted processing to the input feature maps separately to enhance the expressive power of features in their respective dimensions. Finally, the two different attention features are fused together through addition, thereby extracting the ability of the two-stream 3D convolutional network to capture fine-grained differences in micro-expressions. This is named DPCBAM, and the specific design of the module is as follows... Figure 5 As shown.

[0070] Step 4.4 Finally, the Softmax classifier is used to identify and classify the types of micro-expressions. The formula for the Softmax loss is as follows:

[0071]

[0072] Where n represents the total number of samples in the dataset; c represents the total number of categories; y i f represents the true class of the i-th sample; i This represents the feature vector of the i-th sample; Represents the true category y iThe weight vector is the basis used by the two-stream 3D convolutional network to determine whether a sample belongs to this category; Represents the true class weight vector With eigenvector f i The angle between them.

[0073] Step 4.5: Evaluate the constructed two-stream 3D convolutional network using metrics including accuracy, recall, precision, and F1 score. The specific formulas are shown below.

[0074]

[0075] In the formula, TP represents the number of true micro-expression categories correctly predicted by the two-stream 3D convolutional network; TN represents the number of other categories correctly predicted by the two-stream 3D convolutional network; FP represents the number of other categories incorrectly predicted as true micro-expression categories by the two-stream 3D convolutional network; and FN represents the number of true micro-expression categories incorrectly predicted as other categories by the two-stream 3D convolutional network.

[0076] Table 1. Indicators on the two public datasets

[0077]

[0078] As shown in Table 1, the present invention achieves an accuracy of 93.00%, a recall of 88.55%, an unweighted F1 score (UF1) of 89.21%, an unweighted mean recall (UAR) of 88.55%, a precision of 90.34%, and an F1 score of 89.44% on the public dataset SAMM (5-class classification). Furthermore, on the public dataset CASME2 (5-class classification), the unweighted F1 score (UF1) reaches 89.39%, and the unweighted mean recall (UAR) reaches 88.99%, significantly outperforming other methods in the field, demonstrating the effectiveness and generalization of the present invention.

[0079] This invention enhances image representation capabilities by constructing a well-structured data preprocessing workflow, fusing optical strain and pixel flow features, and integrating the DPCBAM attention mechanism into a lightweight 3D convolutional framework. This effectively improves the ability of dual-stream 3D convolutional networks to model fine-grained features of micro-expressions. The proposed method not only balances the accuracy of feature representation with computational efficiency but also possesses good generalization performance.

[0080] This invention is illustrated through specific implementation processes. Various modifications and equivalent substitutions can be made to this invention without departing from its scope. Therefore, this invention is not limited to the specific implementation processes disclosed, but should include all embodiments falling within the scope of the claims of this invention.

Claims

1. A micro-expression recognition method based on dual-stream feature fusion, characterized in that: The specific steps include the following: Step 1: Obtain the micro-expression dataset by selecting N frames from the start frame to the top frame of the micro-expression video and preprocessing the data. Step 2: Calculate the optical flow field between adjacent frames in the N frames of images from Step 1 using the optical flow algorithm, extract the optical flow features of micro-expressions, and form an N-1 frame optical flow image sequence. Step 3: Convert the N frames of RGB face images from Step 1 to the CIE Lab color space, calculate the pixel difference between the two converted Lab images, extract the pixel stream features of micro-expressions, and form an N-1 frame pixel stream image sequence. Step 4: Extract the optical flow features and pixel flow features between the two images and input them into the constructed two-stream 3D convolutional network for analysis and recognition to obtain micro-expression classification and recognition. Evaluate the constructed two-stream 3D convolutional network based on various indicators.

2. The micro-expression recognition method based on dual-stream feature fusion according to claim 1, characterized in that: Step 1 is as follows: First, select the micro-expression dataset, then construct a new target folder system for storing images for subsequent processing. Iterate through each subject, corresponding video clip and its frame image in the selected dataset. Use image processing tools to read each frame image, and use face detection algorithms to extract face regions. Crop and normalize the detected face images, standardize the image size to 128×128 pixels, and finally save the processed image files to the preset path according to the classification structure of the original data.

3. The micro-expression recognition method based on dual-stream feature fusion according to claim 1, characterized in that: Step 2 specifically involves selecting adjacent frames as input pairs for optical flow calculation, and using the TV-L1 algorithm to calculate the optical flow field between each pair of images, thereby obtaining pixel-level motion vectors. The optical strain formula of the TV-L1 algorithm is shown below: Where, ε xx ε represents the horizontal optical flow strain between two image frames. yy ε represents the vertical optical flow strain between two images. xy The shear optical flow strain between two frames is denoted as .

4. The micro-expression recognition method based on dual-stream feature fusion according to claim 1, characterized in that: Step 3 involves using a linear transformation matrix to map the original RGB image to the CIE XYZ color system, where the transformation matrix is ​​shown below: Where X, Y, Z are the three primary color values ​​in the CIE XYZ color space, representing the coordinates of the color in the XYZ space; R, G, B are the normalized RGB values, representing the intensity of the red, green, and blue channels of the input image. The reference white point corresponding to the D65 light source is selected as the normalization benchmark. The color information of the input image is normalized, and the tristimulus value of this white point is: X n =0.95047, Y n =1.00000, Z n =1.08883, the normalization calculation formula is as follows: The XYZ values ​​are converted to Lab values ​​using a nonlinear mapping f(t), and L, a, and b are calculated as shown in the following formula: L=116·f(Y')-16 a = 500 * (f(X') - f(Y')) b = 200 * (f(Y') - f(Z')) The definition of the piecewise function f(t) for the nonlinear transformation is as follows: Finally, the pixel difference between the two converted Lab images is calculated to extract the pixel flow features of micro-expressions.

5. The micro-expression recognition method based on dual-stream feature fusion according to claim 1, characterized in that: The dual-stream 3D convolutional network constructed in Step 4 consists of Conv3D blocks, DWConv3D blocks, SandGlass3D blocks, Dense3D blocks, and GDConv3D blocks.

6. The micro-expression recognition method based on dual-stream feature fusion according to claim 5, characterized in that: The SandGlass3D block design employs two depthwise convolution operations, placed outside of two pointwise convolutions. Both the first and last pointwise convolutional layers utilize the ReLU activation function to introduce non-linear transformations. An improved attention mechanism, DPCBAM, is added after the first depthwise convolutional layer. Finally, a Softmax classifier is used to identify and classify micro-expressions. The formula for the Softmax loss is shown below: In the formula, n represents the total number of samples in the dataset; c represents the total number of categories; y i f represents the true class of the i-th sample; i This represents the feature vector of the i-th sample; Represents the true category y i The weight vector is the basis used by the two-stream 3D convolutional network to determine whether a sample belongs to this category; Represents the true class weight vector With eigenvector f i The angle between them.

7. The micro-expression recognition method based on dual-stream feature fusion according to claim 1, characterized in that: The evaluation metrics for the constructed two-stream 3D convolutional network in Step 4 include: Accuracy, Recall, Precision, and F1 Score, as shown in the following formulas: In the formula, TP represents the number of true micro-expression categories correctly predicted by the two-stream 3D convolutional network; TN represents the number of other categories correctly predicted by the two-stream 3D convolutional network; FP represents the number of other categories incorrectly predicted as true micro-expression categories by the two-stream 3D convolutional network; and FN represents the number of true micro-expression categories incorrectly predicted as other categories by the two-stream 3D convolutional network.

Images