Micro-expression recognition method and system based on improved feature pyramid network

Abstract

The invention discloses a micro-expression recognition method and system based on an improved feature pyramid network, and the method comprises the following specific steps: 1, data preprocessing: for a given micro-expression video sequence, randomly extracting a frame picture from several frames around a marked starting frame as a starting frame for model training, and carrying out the data preprocessing; in the same way, randomly extracting a frame from frames around the marked vertex frame as a vertex frame for model training; a random frame extraction method can reduce the condition of model overfitting to a certain extent; detecting mark points on the face of the picture after frame extraction by using a Dlib tool, and cutting the image according to the mark points; the pixel size of all the cut images is adjusted to be 224 * 224; the micro-expression recognition method based on the improved feature pyramid network aims at conducting multi-scale feature learning on the features of the face micro-expression through a feature pyramid, the purpose of feature enhancement is achieved, and the model is more generalized.

Description

Technical Field

[0001] The present invention belongs to the technical fields of deep learning, computer vision and expression recognition, and in particular relates to a micro-expression recognition method and system based on an improved feature pyramid network. Background Art

[0002] In the fast-paced social life, people may face pressure from others or social expectations, and need to show specific emotions or behaviors in social interactions. This pressure may lead to emotional suppression, but emotions can still be expressed in micro-expressions, which are emotional clues that appear briefly on the face due to failure of emotional control. Therefore, compared with the macro-expressions that are easily visible on the face, the short and subtle micro-expressions can better reflect a person's instantaneous true emotions. Under the severe challenge of the surge in psychological problems, the use of micro-expression analysis technology will assist doctors in detecting mental health problems at an early stage and provide timely intervention and treatment.

[0003] Micro-expressions are challenging to identify due to their smallness and instantaneousness. Existing micro-expression recognition methods are mainly divided into two categories: one is the traditional micro-expression recognition method based on machine learning; the other is the micro-expression recognition method based on deep learning. When using traditional machine learning methods to recognize micro-expressions, it is usually necessary to rely on cumbersome manual feature engineering. Whether it is local binary pattern or optical flow feature extraction, feature extraction is performed manually first, and then the extracted features are sent to the model for training. Therefore, with the development of deep learning technology, more and more researchers have begun to adopt micro-expression recognition methods based on deep learning.

[0004] There are many patents for micro-expression recognition systems and methods based on deep learning that have been disclosed. The "Micro-expression Recognition Method Based on Three-Dimensional Residual Convolutional Neural Network and Optical Flow Method" with application number CN202310808285.0 and the "Micro-expression Recognition Method Based on Convolutional Neural Network and Optical Flow Features" with application number CN202410691565.2 both use the optical flow method for feature extraction. The difference is that the first method only uses the three-channel image sequence obtained by optical flow feature extraction as the input of the network model, while the second method first extracts four facial regions of interest (i.e., nose, mouth, left eye and eyebrows, right eye and eyebrows) based on the information of facial key points, and then extracts optical flow features. The way of extracting features using the optical flow method not only has problems such as sensitivity to noise and changes in illumination, but also mainly extracts pixel-level motion information, and the extracted features are not high-level enough. Although the "Micro-expression recognition system based on a large model" with application number CN202410577624.3 can directly connect the video input to the data processing unit to recognize micro-expressions, it still requires the participation of a pre-trained large model. Although the addition of a pre-trained large model can reduce the workload and complexity of preparing training data in advance, it also increases the data dependence of the model to a certain extent, and there is a risk of overfitting. Summary of the invention

[0005] The purpose of the present invention is to provide a micro-expression recognition method and system based on an improved feature pyramid network. After preprocessing the collected face video data, the features of the micro-expressions of the face are enhanced by multi-scale global features based on the improved feature pyramid network, and the local features of the data are enhanced by a symmetrical face segmentation strategy. Finally, a three-branch convolutional neural network is used to capture and analyze the tiny and instantaneous signals in the facial expressions, determine whether there are facial micro-expressions and the type of micro-expressions they belong to, and implement system applications to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a micro-expression recognition method based on an improved feature pyramid network, the specific steps are as follows:

[0007] Step 1: Data preprocessing. For a given micro-expression video sequence, randomly extract a frame from the frames around the marked start frame as the start frame of the model training. Similarly, randomly extract a frame from the frames around the marked vertex frame as the vertex frame of the model training. The random frame extraction method can reduce the overfitting of the model to a certain extent. After the frame extraction, the Dlib tool is used to detect the landmarks on the face of the image, and the image is cropped according to these landmarks. All cropped images will be resized to 224×224 pixels.

[0008] Step 2: After data preprocessing, all sequence images are further enhanced with multi-scale global features using an improved feature pyramid network to reduce the impact of micro-expressions on model recognition;

[0009] Step 3: Use the local feature enhancement method of symmetrical facial segmentation strategy to reduce the interference of redundant background on the model;

[0010] Step 4: Use a three-branch convolutional neural network to perform micro-expression recognition on the data with enhanced features.

[0011] As a preferred technical solution in the present invention, the improved feature pyramid network in step 2 mainly includes three parts: a top-down path, a bottom-up path and a lateral connection.

[0012] As a preferred technical solution in the present invention, in the step three, before using the symmetry facial division strategy (SFD), it is necessary to first perform face head posture detection. If the head rotation angle is within a certain range, it can continue to be used.

[0013] As a preferred technical solution of the present invention, in step 4, the three-branch convolutional neural network includes UM 3 Module and S 2 A 2 M module, where

[0014] UM 3 The module is used in the field of macro facial expression recognition. It mixes information by traversing all possible combinations of channel subsets to solve the problem of insufficient information mixing of various subsets in the existing multi-scale module based on Res2Net.

[0015] S 2 A 2 The M module is modified from the MA-Net model; the original MA-Net model processes channel attention and spatial attention at the same time, which can easily confuse statistical data with saliency information. 2 A 2 The M module processes channel attention and spatial attention separately and puts the attention fusion mechanism at the end, so that S 2 A 2 The M module can extract the channel and spatial attention of saliency and statistics in two sub-branches and make S 2 A 2 The M module avoids the information confusion caused by the early channel attention fusion in the convolutional block attention module, thereby improving local feature extraction and making local feature extraction more accurate.

[0016] As a preferred technical solution in the present invention, after the input RGB image sequence passes through the feature pyramid network and SFD, the data will be divided into three types, which include the following:

[0017] The first type is the uncut complete face image data, which is used as global information and put into the global branch of the three-branch convolutional neural network; the global branch consists of several UM 3 The module consists of a convolutional layer and a 3 After the module traverses all possible combinations of channel subsets to mix information, it passes through a convolutional layer to obtain the output Y global , and calculate the loss value L through a fully connected layer global ;

[0018] The left and right face data after SFD cropping are sent to the second and third branches of the three-branch convolutional neural network, namely the left branch and the right branch for training, respectively. The left branch and the right branch are composed of several S 2 A 2 M module and a convolutional layer, through S 2 A 2 After extracting the significant and statistical channel and spatial attention in the branch, M passes through a convolution layer and merges with other branches to get the output; the output of the left branch and the global branch is combined and calculated through a fully connected layer L left ; The output of the right branch and the global branch are combined and L is calculated through a fully connected layer right ; Finally, the outputs of the left branch and the right branch are combined to get the output Y local And calculate L through a fully connected layer local .

[0019] As a preferred technical solution in the present invention, the Y global and Y local are all one-dimensional tensors, representing the prediction results of different branches of the model, where Y global is the output result after the data passes through the global branch of the network, and Y local It is the output result obtained by adding the data after passing through the left branch and the right branch of the network and then dividing it by 2. The final prediction result Y formula output by the model is as follows:

[0020] Y=αY local +(1-α)Y global (1)

[0021] L global , L left , L right and L local Will form the corresponding loss value, L globalrepresents the loss value calculated by the global branch of the model; L left Represents the loss value calculated by combining the global branch of the model with the left branch; L right Represents the loss value calculated by combining the global branch of the model with the right branch; L local Represents the loss value calculated by combining the left branch and the right branch of the model. The final loss function formula is as follows:

[0022]

[0023] α and β in the above equation represent trade-off parameters.

[0024] As a preferred technical solution in the present invention, the value of α is 0.7, and the value of β is 0.6.

[0025] The present invention also discloses a micro-expression recognition system based on an improved feature pyramid network, including a micro-expression recognition method. The system includes a data acquisition module, a micro-expression analysis module, and a front-end interaction module.

[0026] The data acquisition module obtains the required data through the video files uploaded by the user. The system supports a variety of common video formats, such as MP4, AVI, etc. Through the system interface, users can conveniently upload target videos. The data acquisition module is also equipped with upload progress monitoring and format detection functions to ensure that the video files can be correctly parsed and processed by the system;

[0027] The micro-expression analysis module is implemented based on PyTorch. The system loads a pre-trained deep learning model that has been trained and verified on a large-scale micro-expression dataset. Through the model loading interface provided by PyTorch, the system can accurately classify and locate micro-expressions on the uploaded video frames.

[0028] The front-end interaction module includes displaying data on an interface through a display device and providing user interaction and data call.

[0029] As a preferred technical solution in the present invention, a preprocessing module is also provided on the data acquisition module, which is used for performing video decoding, frame extraction and noise reduction on the input video, and taking the prepared video frames as the input of micro-expression analysis, so as to ensure the flexibility and user-friendliness of the system, so that the user can easily upload the target video to the system for subsequent micro-expression analysis.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. The micro-expression recognition method based on the improved feature pyramid network aims to perform multi-scale feature learning on the features of facial micro-expressions through the feature pyramid to achieve the purpose of feature enhancement and make the model more generalizable.

[0032] 2. The method proposed in the present invention not only utilizes the improved feature pyramid network for multi-scale feature learning, but also makes full use of the symmetry of facial expressions to enhance the local features of the data, thereby reducing the impact of the slightness of micro-expressions on their recognition.

[0033] 3. By using this system to analyze the interviewer's facial micro-expressions, it is possible to identify the interviewer's instantaneous true emotions and discern more authentic emotions, thereby avoiding missing micro-expressions that the human eye cannot directly and quickly perceive, making it simpler and more efficient.

[0034] 4. The system is easy to operate and can be used quickly without complicated training. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a network structure diagram of the feature pyramid of the present invention;

[0036] Figure 2 This is a schematic diagram of the symmetrical face segmentation strategy adopted by the present invention;

[0037] Figure 3 It is a system diagram of a three-branch convolutional neural network of the present invention;

[0038] Figure 4 It is the architecture diagram of the overall model of the present invention;

[0039] Figure 5 It is a system diagram of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] See also Figures 1 to 5 The present invention provides a technical solution: a micro-expression recognition method based on an improved feature pyramid network, the specific steps are as follows:

[0042] Step 1: Data preprocessing. For a given micro-expression video sequence, randomly extract a frame from the frames around the marked start frame as the start frame of the model training. Similarly, randomly extract a frame from the frames around the marked vertex frame as the vertex frame of the model training. The random frame extraction method can reduce the overfitting of the model to a certain extent. After the frame extraction, the Dlib tool is used to detect the landmarks on the face of the image, and the image is cropped according to these landmarks. All cropped images will be resized to 224×224 pixels.

[0043] Step 2: After data preprocessing, the feature pyramid network is used to further perform multi-scale global feature enhancement on all sequence images to reduce the impact of micro-expressions on model recognition;

[0044] Step 3: Use the local feature enhancement method of symmetrical facial segmentation strategy to reduce the interference of redundant background on the model;

[0045] Step 4: Use a three-branch convolutional neural network to perform micro-expression recognition on the data with enhanced features.

[0046] In this embodiment, the feature pyramid network in step 2 mainly includes three parts: a top-down path, a bottom-up path, and a lateral connection; Figure 1 , F in i represents the feature layer of the i-th layer from bottom to top; and P i represents the feature layer of the i-th layer from top to bottom; and whether it is the top-up F i to F i+1 , or top-down P i To P i-1 , the present invention adopts bilinear interpolation (BI) for optimization; bilinear interpolation is a simple and fast interpolation method, which can relatively well preserve the details and texture of the image without causing excessive smoothing, and is also suitable for real-time applications;

[0047] For F i To P i The present invention introduces an attention module (AM) in the lateral connection; the introduction of the attention mechanism in the lateral connection enables the network to automatically learn which features are more important, thereby improving the utilization of detail information; at the same time, the introduction of the attention mechanism will also help the network suppress unimportant features, thereby reducing noise interference and making the network pay more attention to information that is beneficial to the task; after the data passes through the feature pyramid network, a feature map containing multi-scale information will be obtained.

[0048] In step 3 of this embodiment, before using the symmetrical face segmentation strategy, it is necessary to first perform face head posture detection. If the head rotation angle is within a certain range, it can continue to be used. Specifically, firstly, the rotation angle of the head of the face in the figure is obtained by the head posture estimation provided by OpenFace2.0, and then the core area of ​​the face is cropped according to a specific formula. If the head rotation angle exceeds the given range, the symmetrical face segmentation strategy cannot be used well. For details, refer to Figure 2 In the figure, (a) represents the data after face target detection, and the blue frame represents the position of the face area delineated after face target detection, in preparation for the subsequent symmetrical face division; (b) is the symmetrical face quartering method, first of all, it is cropped according to the position of the blue frame; after cropping the face part, the face area is symmetrically divided according to the red straight line, and the final data is shown in (c); the divided data will be sent to the subsequent three-branch convolutional neural network for training, the upper left part of the face area and the lower left part of the face area will be sent to the left branch neural network for training, and the upper right part of the face area and the lower right part of the face area will be sent to the right branch neural network for training.

[0049] In this embodiment, in step 4, the three-branch convolutional neural network includes SFD, and three types are formed at the output end of SFD, including UM 3 Module and S 2 A 2 M module, where

[0050] UM 3 The module is used in the field of macro facial expression recognition. It mixes information by traversing all possible combinations of channel subsets to solve the problem of insufficient information mixing of various subsets in the existing multi-scale module based on Res2Net.

[0051] S 2 A 2 The M module is modified from the MA-Net model; the original MA-Net model processes channel attention and spatial attention at the same time, which can easily confuse statistical data with saliency information. 2 A 2 The M module processes channel attention and spatial attention separately and puts the attention fusion mechanism at the end, so that S 2 A 2 The M module can extract the channel and spatial attention of saliency and statistics in two sub-branches and make S 2 A 2 The M module avoids the information confusion caused by the early channel attention fusion in the convolutional block attention module, thereby improving local feature extraction and making local feature extraction more accurate.

[0052] In this embodiment, after the input RGB image sequence passes through the feature pyramid network and SFD, the data will be divided into three types, which are as follows:

[0053] The first type is the uncut complete face image data, which is used as global information and put into the global branch of the three-branch convolutional neural network; the global branch consists of several UM 3 The module consists of a convolutional layer and a 3 After the module traverses all possible combinations of channel subsets to mix information, it passes through a convolutional layer to obtain the output Y global , and calculate the loss value L through a fully connected layer global ;

[0054] The left and right face data after SFD cropping are sent to the second and third branches of the three-branch convolutional neural network, namely the left branch and the right branch for training, respectively. The left branch and the right branch are composed of several S 2 A 2 M module and a convolutional layer, through S 2 A 2 After extracting the significant and statistical channel and spatial attention in the branch, M passes through a convolution layer and merges with other branches to get the output; the output of the left branch and the global branch is combined and calculated through a fully connected layer L left ; The output of the right branch and the global branch are combined and L is calculated through a fully connected layer right ; Finally, the outputs of the left branch and the right branch are combined to get the output Y local And calculate L through a fully connected layer local .

[0055] In this embodiment, Y global and Y local are all one-dimensional tensors, representing the prediction results of different branches of the model, where Y global is the output result after the data passes through the global branch of the network, and Y local It is the output result obtained by adding the data after passing through the left branch and the right branch of the network and then dividing it by 2. The final prediction result Y formula output by the model is as follows:

[0056] Y=αY local +(1-α)Y global (1)

[0057] L global , L left , L right and L local Will form the corresponding loss value, L global represents the loss value calculated by the global branch of the model; L leftRepresents the loss value calculated by combining the global branch of the model with the left branch; L right Represents the loss value calculated by combining the global branch of the model with the right branch; L local Represents the loss value calculated by combining the left branch and the right branch of the model. The final loss function formula is as follows:

[0058]

[0059] α and β in the above equation represent trade-off parameters.

[0060] In this embodiment, the value of α is 0.7, and the value of β is 0.6.

[0061] The present invention also discloses a micro-expression recognition system based on an improved feature pyramid network, including a micro-expression recognition method. The system includes a data acquisition module, a micro-expression analysis module, and a front-end interaction module.

[0062] The data acquisition module obtains the required data through the video files uploaded by the user. The system supports a variety of common video formats, such as MP4, AVI, etc. Through the system interface, users can easily upload target videos. The data acquisition module is also equipped with upload progress monitoring and format detection functions to ensure that the video files can be correctly parsed and processed by the system;

[0063] The micro-expression analysis module is implemented based on PyTorch. The system loads a pre-trained deep learning model that has been trained and verified on a large-scale micro-expression dataset. Through the model loading interface provided by PyTorch, the system can accurately classify and locate micro-expressions on uploaded video frames, and use the tensor operations and automatic differentiation functions provided by PyTorch for forward reasoning and back propagation; this ensures the system's accurate recognition and feedback of micro-expressions. The system also makes full use of PyTorch's parallel computing capabilities, and accelerates the speed of micro-expression analysis through GPU acceleration, allowing the system to detect micro-expressions in real time or near real time;

[0064] The front-end interaction module includes displaying data on the interface through a display device and providing user interaction and data call.

[0065] In this embodiment, a preprocessing module is also provided on the data acquisition module, which is used for performing video decoding, frame extraction and noise reduction on the input video, and using the prepared video frames as input for micro-expression analysis to ensure the flexibility and user-friendliness of the system, so that users can easily upload the target video to the system for subsequent micro-expression analysis.

[0066] Although embodiments of the present invention have been shown and described (see the above detailed description for details), it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A micro-expression recognition method based on an improved feature pyramid network, characterized in that: The specific steps are as follows: Step 1: Data preprocessing. For a given micro-expression video sequence, randomly extract a frame from the frames around the marked start frame as the start frame of the model training. Similarly, randomly extract a frame from the frames around the marked vertex frame as the vertex frame of the model training. The random frame extraction method can reduce the overfitting of the model to a certain extent. After the frame extraction, the Dlib tool is used to detect the landmarks on the face of the image, and the image is cropped according to these landmarks. All cropped images will be resized to 224×224 pixels. Step 2: After data preprocessing, the feature pyramid network is used to further perform multi-scale global feature enhancement on all sequence images to reduce the impact of micro-expressions on model recognition; Step 3: Use the local feature enhancement method of symmetrical facial segmentation strategy to reduce the interference of redundant background on the model; Step 4: Use a three-branch convolutional neural network to perform micro-expression recognition on the data with enhanced features.

2. The micro-expression recognition method based on the improved feature pyramid network according to claim 1, characterized in that: The improved feature pyramid network in step 2 mainly includes three parts: a top-down path, a bottom-up path and a lateral connection.

3. The micro-expression recognition method based on improved feature pyramid network according to claim 1, characterized in that: In the step 3, before using the Symmetry Facial Division (SFD) strategy, it is necessary to first perform face head posture detection. If the head rotation angle is within a certain range, it can continue to be used.

4. The micro-expression recognition method based on improved feature pyramid network according to claim 1, characterized in that: In step 4, a three-branch convolutional neural network is constructed by UM 3 Module and S 2 A 2 M modules, among which UM 3 The module is used in the field of macro facial expression recognition. It mixes information by traversing all possible combinations of channel subsets to solve the problem of insufficient information mixing of various subsets in the existing multi-scale module based on Res2Net. S 2 A 2 The M module processes channel attention and spatial attention separately and puts the attention fusion mechanism at the end, so that S 2 A 2 The M module can extract the channel and spatial attention of saliency and statistics in two sub-branches and make S 2 A 2 The M-module avoids information confusion caused by early channel attention fusion in the convolutional block attention module.

5. The micro-expression recognition method based on improved feature pyramid network according to claim 4, characterized in that: After the input RGB image sequence passes through the feature pyramid network and SFD, the data will be divided into three types, which are as follows: The first type is the uncut complete face image data, which is used as global information and put into the global branch of the three-branch convolutional neural network; the global branch consists of several UM 3 The module consists of a convolutional layer and a 3 After the module traverses all possible combinations of channel subsets to mix information, it passes through a convolutional layer to obtain the output Y global , and calculate the loss value L through a fully connected layer global ; The left and right face data after SFD cropping are sent to the second and third branches of the three-branch convolutional neural network, namely the left branch and the right branch for training, respectively. The left branch and the right branch are composed of several S 2 A 2 M module and a convolutional layer, through S 2 A 2 After extracting the significant and statistical channel and spatial attention in the branch, M passes through a convolution layer and merges with other branches to get the output; the output of the left branch and the global branch is combined and calculated through a fully connected layer L left ; The output of the right branch and the global branch are combined and L is calculated through a fully connected layer right ; Finally, the outputs of the left branch and the right branch are combined to get the output Y local And calculate L through a fully connected layer local .

6. The micro-expression recognition method based on improved feature pyramid network according to claim 5, characterized in that: The Y global and Y local are all one-dimensional tensors, representing the prediction results of different branches of the model, where Y global is the output result after the data passes through the global branch of the network, and Y local It is the output result obtained by adding the data after passing through the left branch and the right branch of the network and then dividing it by 2. The final prediction result Y formula output by the model is as follows: Y=αY local +(1-α)Y global (1) L global , L left , L right and L local Will form the corresponding loss value, L global represents the loss value calculated by the global branch of the model; L left Represents the loss value calculated by combining the global branch of the model with the left branch; L right Represents the loss value calculated by combining the global branch of the model with the right branch; L local Represents the loss value calculated by combining the left branch and the right branch of the model. The final loss function formula is as follows: α and β in the above equation represent trade-off parameters.

7. The micro-expression recognition method based on improved feature pyramid network according to claim 6, characterized in that: The value of α is 0.7, and the value of β is 0.

6.

8. A micro-expression recognition system based on an improved feature pyramid network, comprising the micro-expression recognition method according to any one of claims 1 to 7, characterized in that: The system includes a data acquisition module, a micro-expression analysis module, and a front-end interaction module. The data acquisition module acquires the required data through the video files uploaded by the user. The data acquisition module is also provided with upload progress monitoring and format detection functions to ensure that the video files can be correctly parsed and processed by the system; The micro-expression analysis module is implemented based on PyTorch. The system loads a pre-trained deep learning model that has been trained and verified on a large-scale micro-expression dataset. Through the model loading interface provided by PyTorch, the system can accurately classify and locate micro-expressions on the uploaded video frames. The front-end interaction module includes displaying data on an interface through a display device and providing user interaction and data call.

9. The micro-expression recognition system based on improved feature pyramid network according to claim 8, characterized in that: The data acquisition module is also provided with a preprocessing module, which is used for performing video decoding, frame extraction and noise reduction processing on the input video.

Images