Micro-expression recognition method and system based on improved feature pyramid network

Abstract

The application discloses a micro-expression recognition method and system based on an improved feature pyramid network, and specific steps are as follows: step one: data preprocessing, for a given micro-expression video sequence, a frame picture is randomly extracted from several frames around a marked starting frame as a starting frame for model training, and a frame is randomly extracted from frames around a marked vertex frame as a vertex frame for model training; the random frame extraction method can reduce the model overfitting to a certain extent; after the frame extraction, a Dlib tool is used to detect the mark points on the face, and the image is cropped according to the mark points; all the cropped images are adjusted to 224x224 pixel size; the micro-expression recognition method based on the improved feature pyramid network aims to perform multi-scale feature learning on the features of the facial micro-expression through the feature pyramid, so that the features are enhanced, and the model is more general.

Description

Technical Field

[0001] The present invention belongs to the technical fields of deep learning, computer vision and expression recognition, and particularly relates to a micro-expression recognition method and system based on an improved feature pyramid network. Background Art

[0002] In the fast-paced world of social life, people may face pressure from others and societal expectations to display specific emotions or behaviors during social interactions. This pressure can lead to emotional suppression, but emotions can still be expressed through micro-expressions—brief facial cues resulting from a failure in emotional control. Therefore, compared to easily visible macro-expressions, these brief and subtle micro-expressions can better reflect a person's true emotions at the moment. Faced with the severe challenge of a surge in psychological problems, the use of micro-expression analysis technology will assist doctors in identifying mental health issues early on, enabling timely intervention and treatment.

[0003] Microexpressions, due to their minuteness and instantaneous nature, present challenges in recognition. Existing microexpression recognition methods fall into two main categories: traditional machine learning-based methods and deep learning-based methods. Traditional machine learning methods often rely on cumbersome manual feature engineering, whether for local binary patterns or optical flow feature extraction. These features are then manually extracted and fed into the model for training. Consequently, with the advancement of deep learning technology, more and more researchers are adopting deep learning-based microexpression recognition methods.

[0004] There are many patents for micro-expression recognition systems and methods based on deep learning that have been disclosed. Application number CN202310808285.0 is "Micro-expression recognition method based on three-dimensional residual convolutional neural network and optical flow method" and application number CN202410691565.2 is "A micro-expression recognition method based on convolutional neural network and optical flow features". Both methods use the optical flow method for feature extraction. The difference is that the first method only uses the three-channel image sequence obtained by optical flow feature extraction as the input of the network model, while the second method first extracts four facial regions of interest (i.e., nose, mouth, left eye and eyebrow, right eye and eyebrow) based on the information of facial key points, and then extracts optical flow features. The way of extracting features using the optical flow method not only has problems such as sensitivity to noise and changes in illumination, but the optical flow method mainly extracts pixel-level motion information, and the extracted features are not high-level enough. While the "Micro-Expression Recognition System Based on a Large Model," application number CN202410577624.3, can directly connect video input to a data processing unit to recognize micro-expressions, it still requires the participation of a pre-trained large model. While the inclusion of a pre-trained large model can reduce the workload and complexity of preparing training data, it also increases the model's data dependency to a certain extent, posing the risk of overfitting. Summary of the Invention

[0005] The present invention aims to provide a micro-expression recognition method and system based on an improved feature pyramid network. After preprocessing collected facial video data, the improved feature pyramid network is used to perform multi-scale global feature enhancement of facial micro-expressions. A symmetrical facial segmentation strategy is then used to enhance the local features of the data. Finally, a three-branch convolutional neural network is used to capture and analyze subtle and transient signals in facial expressions, determining the presence and type of facial micro-expressions. The system is then applied to address the issues raised in the aforementioned background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a micro-expression recognition method based on an improved feature pyramid network, the specific steps of which are as follows:

[0007] Step 1: Data preprocessing: For a given micro-expression video sequence, randomly extract a frame from several frames surrounding the marked starting frame as the starting frame for model training. Similarly, randomly extract a frame from the frames surrounding the marked vertex frame as the vertex frame for model training. This random frame extraction method can reduce model overfitting to a certain extent. After the frames are extracted, the Dlib tool is used to detect facial landmarks on the images and crop the images based on these landmarks. All cropped images are resized to 224×224 pixels.

[0008] Step 2: After data preprocessing, all sequence images are further enhanced with multi-scale global features using an improved feature pyramid network;

[0009] Step 3: Local feature enhancement method using symmetrical face segmentation strategy;

[0010] Step 4: Use a three-branch convolutional neural network to perform micro-expression recognition on the data after feature enhancement.

[0011] As a preferred technical solution in the present invention, the improved feature pyramid network in step 2 includes three parts: a top-down path, a bottom-up path, and a lateral connection.

[0012] As a preferred technical solution of the present invention, in step three, before using the symmetry facial division strategy (SFD), face head posture detection is performed first. If the head rotation angle is within a certain range, it can continue to be used.

[0013] As a preferred technical solution of the present invention, in step 4, the three-branch convolutional neural network includes UM 3 Module and S 2 A 2 M module, where

[0014] UM 3 The module is used for macro-facial expression recognition and performs information mixing by traversing all possible combinations of channel subsets;

[0015] S 2 A 2 The M module is modified from the MA-Net model; the original MA-Net model processes channel attention and spatial attention at the same time, which can easily confuse statistical data with significant information. 2 A 2 The M module processes channel attention and spatial attention separately and puts the attention fusion mechanism at the end, so that S 2 A 2 The M module extracts the channel and spatial attention of saliency and statistics in two sub-branches, and makes S 2 A 2 The M module avoids the information confusion caused by the early channel attention fusion in the convolutional block attention module, thereby improving local feature extraction and making local feature extraction more accurate.

[0016] As a preferred technical solution in the present invention, the input RGB image sequence is processed by the feature pyramid network and SFD.

[0017] The complete face image data without cropping is used as global information and put into the global branch of the three-branch convolutional neural network; the global branch consists of several UM 3 The module consists of a convolutional layer and a UM 3 After the module traverses all possible combinations of channel subsets to mix information, it passes through a convolutional layer to obtain the output Y global , and calculate the loss value L through a fully connected layer global ;

[0018] The left face data and right face data after SFD cropping are respectively sent to the left branch and right branch of the three-branch convolutional neural network for training, where the left branch and the right branch are composed of several S 2 A 2 M module and a convolutional layer, through S 2 A 2 After M extracts the channel and spatial attention of the significance and statistics in the branch, it passes through a convolution layer and merges with other branches to obtain the output; the output of the left branch and the global branch is combined and calculated through a fully connected layer L left ; The output of the right branch and the global branch are combined and L is calculated through a fully connected layer right ; Finally, the outputs of the left branch and the right branch are combined to get the output Y local And calculate L through a fully connected layer local .

[0019] As a preferred technical solution in the present invention, the Y global and Y local Each is a one-dimensional tensor, representing the prediction results of different branches of the model, where Y global is the output result after the data passes through the global branch of the network, and Y local It is the output result obtained by adding the data after passing through the left branch and the right branch of the network and then dividing it by 2. The final prediction result Y output by the model is as follows:

[0020] Y=αY local +(1-α)Y global (1)

[0021] L global , L left , L right and L local Will form the corresponding loss value, L global Represents the loss value calculated by the global branch of the model; L left Represents the loss value calculated by combining the global branch of the model with the left branch; L right Represents the loss value calculated by combining the global branch of the model with the right branch; L localRepresents the loss value calculated by combining the left branch and the right branch of the model. The final loss function formula is as follows:

[0022]

[0023] α and β in the above formula represent trade-off parameters.

[0024] As a preferred technical solution in the present invention, the value of α is 0.7, and the value of β is 0.6.

[0025] The present invention also discloses a micro-expression recognition system based on an improved feature pyramid network, including a micro-expression recognition method. The system includes a data acquisition module, a micro-expression analysis module, and a front-end interaction module, wherein

[0026] The data acquisition module obtains the required data through the video files uploaded by the user. The system supports a variety of common video formats, such as MP4, AVI, etc. Through the system interface, users can conveniently upload the target video. The data acquisition module is also equipped with upload progress monitoring and format detection functions to ensure that the video files can be correctly parsed and processed by the system;

[0027] The micro-expression analysis module is implemented based on PyTorch. The system loads a pre-trained deep learning model that has been trained and verified on a large-scale micro-expression dataset. Through the model loading interface provided by PyTorch, the system can accurately classify and locate micro-expressions on uploaded video frames.

[0028] The front-end interaction module includes displaying data on the interface through a display device and providing user interaction and data call.

[0029] As a preferred technical solution in the present invention, a preprocessing module is also provided on the data acquisition module, which is used to perform video decoding, frame extraction and noise reduction on the input video, and use the prepared video frames as input for micro-expression analysis to ensure the flexibility and user-friendliness of the system, so that users can easily upload the target video to the system for subsequent micro-expression analysis.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. The micro-expression recognition method based on the improved feature pyramid network aims to perform multi-scale feature learning on the features of facial micro-expressions through the feature pyramid to achieve the purpose of feature enhancement and make the model more generalizable.

[0032] 2. The method proposed in the present invention not only uses the improved feature pyramid network for multi-scale feature learning, but also makes full use of the symmetry of facial expressions to enhance the local features of the data, thereby reducing the impact of the subtlety of micro-expressions on its recognition.

[0033] 3. By using this system to analyze the interviewer's facial micro-expressions, it is possible to identify the interviewer's instantaneous true emotions and discern more authentic emotions, thereby avoiding missing micro-expressions that the human eye cannot directly and quickly perceive, making it simpler and more efficient.

[0034] 4. The system is easy to operate and can be used quickly without complex training. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a network structure diagram of the feature pyramid of the present invention;

[0036] Figure 2 This is a schematic diagram of the symmetrical face segmentation strategy adopted in the present invention;

[0037] Figure 3 This is a system diagram of a three-branch convolutional neural network of the present invention;

[0038] Figure 4 This is a diagram of the overall model of the present invention;

[0039] Figure 5 It is a system diagram of the present invention. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] See also Figures 1 to 5 The present invention provides a technical solution: a micro-expression recognition method based on an improved feature pyramid network, the specific steps are as follows:

[0042] Step 1: Data preprocessing: For a given micro-expression video sequence, randomly extract a frame from several frames surrounding the marked starting frame as the starting frame for model training. Similarly, randomly extract a frame from the frames surrounding the marked vertex frame as the vertex frame for model training. This random frame extraction method can reduce model overfitting to a certain extent. After the frames are extracted, the Dlib tool is used to detect facial landmarks on the images and crop the images based on these landmarks. All cropped images are resized to 224×224 pixels.

[0043] Step 2: After data preprocessing, all sequence images are further enhanced with multi-scale global features using a feature pyramid network;

[0044] Step 3: Local feature enhancement method using symmetrical face segmentation strategy;

[0045] Step 4: Use a three-branch convolutional neural network to perform micro-expression recognition on the data after feature enhancement.

[0046] In this embodiment, the feature pyramid network in step 2 mainly includes three parts: a top-down path, a bottom-up path, and a horizontal connection; Figure 1 , the F in i represents the feature layer of the i-th layer from bottom to top; and P i Represents the feature layer of the i-th layer from top to bottom; and whether it is the top-up F i to F i+1 , or top-down P i to P i-1 , the present invention adopts bilinear interpolation (BI) for optimization; bilinear interpolation is a simple and fast interpolation method that can relatively well preserve image details and textures without causing excessive smoothing, and is also suitable for real-time applications;

[0047] And for F i to P i In order to solve the horizontal connection of the feature pyramid network, the present invention introduces an attention module (AM); the introduction of the attention mechanism in the horizontal connection enables the network to automatically learn which features are more important, thereby improving the utilization of detailed information; at the same time, the introduction of the attention mechanism will also help the network suppress unimportant features, thereby reducing noise interference and making the network pay more attention to information that is beneficial to the task; after the data passes through the feature pyramid network, a feature map containing multi-scale information will be obtained.

[0048] In this embodiment, in step 3, before using the symmetrical face segmentation strategy, the face head posture detection is performed first. If the head rotation angle is within a certain range, it can continue to be used; specifically, the rotation angle of the head of the face in the image is first obtained by the head posture estimation provided by OpenFace2.0, and the core area of ​​the face is cropped according to a specific formula; if the head rotation angle exceeds the given range, the symmetrical face segmentation strategy cannot be used well. For details, refer to Figure 2In the figure, (a) represents the data after face target detection, and the blue box represents the position of the face area delineated after face target detection, in preparation for the subsequent symmetrical face segmentation; (b) is the symmetrical face quartering method, which first crops according to the position of the blue box; after cropping the face part, the face area is symmetrically divided according to the red straight line, and the final data is shown in (c); the divided data will be sent to the subsequent three-branch convolutional neural network for training, the upper left part of the face area and the lower left part of the face area will be sent to the left branch neural network for training, and the upper right part of the face area and the lower right part of the face area will be sent to the right branch neural network for training.

[0049] In this embodiment, in step 4, the three-branch convolutional neural network includes SFD, and three types are formed on the output end of SFD, including UM 3 Module and S 2 A 2 M module, where

[0050] UM 3 The module is used for macro-facial expression recognition and performs information mixing by traversing all possible combinations of channel subsets;

[0051] S 2 A 2 The M module is modified from the MA-Net model; the original MA-Net model processes channel attention and spatial attention at the same time, which can easily confuse statistical data with significant information. 2 A 2 The M module processes channel attention and spatial attention separately and puts the attention fusion mechanism at the end, so that S 2 A 2 The M module extracts the channel and spatial attention of saliency and statistics in two sub-branches, and makes S 2 A 2 The M module avoids the information confusion caused by the early channel attention fusion in the convolutional block attention module, thereby improving local feature extraction and making local feature extraction more accurate.

[0052] In this embodiment, the input RGB image sequence is processed by the feature pyramid network and SFD.

[0053] The complete face image data without cropping is used as global information and put into the global branch of the three-branch convolutional neural network; the global branch consists of several UM 3 The module consists of a convolutional layer and a UM 3 After the module traverses all possible combinations of channel subsets to mix information, it passes through a convolutional layer to obtain the output Y global , and calculate the loss value L through a fully connected layerglobal ;

[0054] The left face data and right face data after SFD cropping are respectively sent to the left branch and right branch of the three-branch convolutional neural network for training, where the left branch and the right branch are composed of several S 2 A 2 M module and a convolutional layer, through S 2 A 2 After M extracts the channel and spatial attention of the significance and statistics in the branch, it passes through a convolution layer and merges with other branches to obtain the output; the output of the left branch and the global branch is combined and calculated through a fully connected layer L left ; The output of the right branch and the global branch are combined and L is calculated through a fully connected layer right ; Finally, the outputs of the left branch and the right branch are combined to get the output Y local And calculate L through a fully connected layer local .

[0055] In this embodiment, Y global and Y local Each is a one-dimensional tensor, representing the prediction results of different branches of the model, where Y global is the output result after the data passes through the global branch of the network, and Y local It is the output result obtained by adding the data after passing through the left branch and the right branch of the network and then dividing it by 2. The final prediction result Y output by the model is as follows:

[0056] Y=αY local +(1-α)Y global (1)

[0057] L global , L left , L right and L local Will form the corresponding loss value, L global Represents the loss value calculated by the global branch of the model; L left Represents the loss value calculated by combining the global branch of the model with the left branch; L right Represents the loss value calculated by combining the global branch of the model with the right branch; L local Represents the loss value calculated by combining the left branch and the right branch of the model. The final loss function formula is as follows:

[0058]

[0059] α and β in the above formula represent trade-off parameters.

[0060] In this embodiment, the value of α is 0.7, and the value of β is 0.6.

[0061] The present invention also discloses a micro-expression recognition system based on an improved feature pyramid network, including a micro-expression recognition method. The system includes a data acquisition module, a micro-expression analysis module, and a front-end interaction module, wherein

[0062] The data acquisition module obtains the required data through the video files uploaded by the user. The system supports a variety of common video formats, such as MP4, AVI, etc. Through the system interface, users can easily upload the target video. The data acquisition module is also equipped with upload progress monitoring and format detection functions to ensure that the video files can be correctly parsed and processed by the system;

[0063] The micro-expression analysis module is implemented using PyTorch. The system loads a pre-trained deep learning model that has been trained and validated on a large-scale micro-expression dataset. Through the model loading interface provided by PyTorch, the system can accurately classify and locate micro-expressions in uploaded video frames, utilizing PyTorch's tensor operations and automatic differentiation capabilities for forward reasoning and backpropagation. This ensures accurate recognition and feedback of micro-expressions. The system also leverages PyTorch's parallel computing capabilities, accelerating micro-expression analysis through GPU acceleration, enabling real-time or near-real-time micro-expression detection.

[0064] The front-end interaction module includes displaying data on the interface through a display device and providing user interaction and data call.

[0065] In this embodiment, a preprocessing module is also provided on the data acquisition module. The preprocessing module is used to perform video decoding, frame extraction and noise reduction on the input video, and use the prepared video frames as input for micro-expression analysis to ensure the flexibility and user-friendliness of the system, so that users can easily upload the target video to the system for subsequent micro-expression analysis.

[0066] Although embodiments of the present invention have been shown and described (see the detailed description above for details), it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, and the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A micro-expression recognition method based on an improved feature pyramid network, characterized by: The specific steps are as follows: Step 1: Data preprocessing: For a given micro-expression video sequence, randomly extract a frame from several frames surrounding the marked starting frame as the starting frame for model training. Similarly, randomly extract a frame from the frames surrounding the marked vertex frame as the vertex frame for model training. This random frame extraction method can reduce model overfitting to a certain extent. After the frames are extracted, the Dlib tool is used to detect facial landmarks on the images and crop the images based on these landmarks. All cropped images are resized to 224×224 pixels. Step 2: After data preprocessing, all sequence images are further enhanced with multi-scale global features using a feature pyramid network; Step 3: Local feature enhancement method using symmetrical face segmentation strategy; Step 4: Use a three-branch convolutional neural network to perform micro-expression recognition on the data after feature enhancement; In step 4, the three-branch convolutional neural network is composed of UM 3 Module and S 2 A 2 M modules, among which UM 3 The module is used for macro-facial expression recognition and performs information mixing by traversing all possible combinations of channel subsets; S 2 A 2 The M module processes channel attention and spatial attention separately and puts the attention fusion mechanism at the end, so that S 2 A 2 The M module extracts the channel and spatial attention of saliency and statistics in two sub-branches, and makes S 2 A 2 The M module avoids information confusion caused by early channel attention fusion in the convolutional block attention module; After the input RGB image sequence passes through the feature pyramid network and SFD, The complete face image data without cropping is used as global information and put into the global branch of the three-branch convolutional neural network; the global branch consists of several UM 3 The module consists of a convolutional layer and a UM 3 After the module traverses all possible combinations of channel subsets to mix information, it passes through a convolutional layer to obtain the output Y global , and calculate the loss value L through a fully connected layer global ; The final prediction result Y of the model is as follows: (1)； The left face data and right face data after SFD cropping are respectively sent to the left branch and right branch of the three-branch convolutional neural network for training, where the left branch and the right branch are composed of several S 2 A 2 M module and a convolutional layer, through S 2 A 2 After M extracts the channel and spatial attention of the significance and statistics in the branch, it passes through a convolution layer and merges with other branches to obtain the output; the output of the left branch and the global branch is combined and calculated through a fully connected layer L left ; The output of the right branch and the global branch are combined and L is calculated through a fully connected layer right ; Finally, the outputs of the left branch and the right branch are combined to get the output Y local And calculate L through a fully connected layer local ; The final loss function formula is as follows: ， (2)； In the above formula, α and β represent the trade-off parameters; global is the output result after the data passes through the global branch of the network, and Y local It is the output result obtained by adding the data after passing through the left branch and the right branch of the network and then dividing it by 2. global Represents the loss value calculated by the global branch of the model; L left Represents the loss value calculated by combining the global branch of the model with the left branch; L right Represents the loss value calculated by combining the global branch of the model with the right branch; L local Represents the loss value calculated by combining the left branch and the right branch of the model.

2. The micro-expression recognition method based on the improved feature pyramid network according to claim 1, wherein: The improved feature pyramid network in step 2 includes three parts: a top-down path, a bottom-up path, and a lateral connection.

3. The micro-expression recognition method based on the improved feature pyramid network according to claim 1, wherein: In step 3, before using the Symmetry Facial Division (SFD) strategy, the face head posture detection is performed first. If the head rotation angle is within a certain range, it can continue to be used.

4. The micro-expression recognition method based on the improved feature pyramid network according to claim 2, wherein: The value of α is 0.7, and the value of β is 0.

6.

5. A micro-expression recognition system based on an improved feature pyramid network, comprising the micro-expression recognition method according to any one of claims 1 to 4, characterized in that: The system includes a data acquisition module, a micro-expression analysis module, and a front-end interaction module. The data acquisition module acquires the required data through the video files uploaded by the user. The data acquisition module is also provided with upload progress monitoring and format detection functions to ensure that the video files can be correctly parsed and processed by the system; The micro-expression analysis module is implemented based on PyTorch. The system loads a pre-trained deep learning model that has been trained and verified on a large-scale micro-expression dataset. Through the model loading interface provided by PyTorch, the system can accurately classify and locate micro-expressions on uploaded video frames. The front-end interaction module includes displaying data on the interface through a display device and providing user interaction and data call.

6. The micro-expression recognition system based on the improved feature pyramid network according to claim 5, characterized in that: The data acquisition module is also provided with a pre-processing module, which is used for performing video decoding, frame extraction and noise reduction processing on the input video.

Images