Micro-expression recognition method and system based on three-branch network

Abstract

The invention provides a micro-expression recognition method and system based on a three-branch network, relates to the technical field of micro-expression recognition, and aims to solve the problems that according to an existing micro-expression recognition technology, micro-expression samples are insufficient, multi-type features are difficult to learn, and the model recognition precision is low. The method comprises the following steps: acquiring a video sequence, preprocessing the video sequence to obtain an optical flow diagram, further processing the optical flow diagram to obtain an optical strain diagram, respectively inputting the video sequence, the optical flow diagram and the optical strain diagram into a three-branch network, extracting human face features, optical flow features and optical strain features, and fusing the human face features, the optical flow features and the optical strain features to obtain a fusion result. And performing micro-expression classification on the fused features to obtain a micro-expression classification result. According to the method, the problems of insufficient micro-expression samples and the like are solved, the loss of fine face features in the recognition process can be reduced by utilizing implied different information, the model can more effectively process samples which are difficult to classify, and the micro-expression recognition accuracy is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of micro-expression recognition, and in particular relates to a micro-expression recognition method and system based on a three-branch network. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Facial expressions are a way for a person to express his or her emotional state, but people often hide their true emotions. Psychology believes that real emotions are sometimes expressed in a very short and fast facial expression, which usually lasts 1 / 25s to 1 / 3s and appears in a specific area of ​​the face. This expression is called a micro-expression. Compared with macro-expressions, micro-expressions are facial emotions that people trigger unintentionally. They are spontaneous and uncontrollable and can reflect people's true feelings. Therefore, the study of micro-expressions plays an important role and has a wide range of applications in public security, judicial investigation, people's daily life and other fields.

[0004] Micro-expression recognition has three significant characteristics: (1) short duration, usually lasting 1 / 25s to 1 / 3s, (2) small amplitude and low intensity of change, and (3) local movement only occurs in fixed facial areas. Current micro-expression recognition methods generally extract features after data preprocessing, and then classify micro-expressions based on the extracted features. In terms of feature extraction, traditional methods mostly extract features from original images through feature descriptors, such as LBP-TOP, LBP-SIP, etc. Some methods use optical flow methods to improve the accuracy of micro-expression feature extraction, such as Bi-WOOF, MDMO, etc. However, these methods mostly extract some basic and intuitive features and information that describe facial expressions, and do not involve a deep understanding of facial expressions.

[0005] In order to obtain richer features, most of the feature extraction methods based on deep learning are currently used, such as using recurrent convolutional neural networks (RCN) to extract features from the original image sequence and features from the optical flow respectively, combining convolutional neural networks (CNN) and long short-term memory networks (LSTM) to extract features for micro-expression recognition, etc. However, these network models have complex structures, require a lot of computing resources, and are prone to overfitting problems during training. There are also methods that use shallow convolutional networks with multi-stream input information to extract features. This method is simpler and easier to understand in micro-expression recognition, but it cannot capture complex features. 3D convolutional neural networks (3DCNN) can extract temporal and spatial information at the same time, and learn features from low to high levels, which helps to construct and express complex spatiotemporal features layer by layer. This network provides a good tool for micro-expression recognition. However, when applied to micro-expression datasets with small sample sizes, 3DCNN is prone to the risk of overfitting. Due to the unclear boundaries of micro-expression classification and insufficient data set samples, many current deep learning-based methods, including 3DCNN, are still limited in feature learning ability and model generalization and fitting ability, resulting in low recognition rates.

[0006] In summary, the existing micro-expression recognition technology has certain shortcomings:

[0007] (1) Most models extract some basic and intuitive features and information describing facial expressions, but do not involve a deep understanding of facial expressions and cannot capture complex features;

[0008] (2) Although some models can extract complex micro-expression features, their model structures are complex, require a lot of calculations, and are prone to overfitting problems during training;

[0009] (3) Some models have overcome the problems of complex structure and large amount of computation required. However, for micro-expression datasets with small sample sizes, there is still the problem of overfitting. Insufficient micro-expression samples make it difficult to learn multiple types of features, resulting in low recognition accuracy. Summary of the invention

[0010] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a micro-expression recognition method and system based on a three-branch network. A deep convolutional neural network is combined with a shallow 3D convolutional neural network and multiple attention modules are integrated to construct a micro-expression recognition model based on a three-branch network. The three-branch network is used to extract and process facial features, optical flow features, and optical strain features. At the same time, the IM Loss function is introduced into the model and combined with the Focal Loss function, so that the model can more effectively process samples that are difficult to classify, solving the problems of insufficient micro-expression samples, difficulty in learning multiple types of features, and low recognition accuracy.

[0011] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0012] The first aspect of the present invention discloses a micro-expression recognition method based on a three-branch network, comprising:

[0013] A video sequence is obtained, and the video sequence is preprocessed to obtain an optical flow map, and the optical flow map is further processed to obtain an optical strain map, wherein the video sequence includes video frames;

[0014] Input the video frame into the first branch network, and the first branch network extracts facial features based on deep learning and deep enhanced channel attention;

[0015] The optical flow map is input into the second branch network, and the second branch network uses a second shallow convolutional neural network and a second facial feature enhanced attention to extract features of the optical flow map;

[0016] The light strain map is input into the third branch network, and the third branch network extracts features of the light strain map by using a third shallow convolutional neural network and a third facial feature enhanced attention;

[0017] The facial features, the features of the optical flow map, and the features of the optical strain map are integrated;

[0018] The fused features are used to classify micro-expressions to obtain the classification results of micro-expressions.

[0019] As a further technical solution, the video frame is a vertex frame, which refers to the frame in which the intensity or expression of the micro-expression is most obvious, or a sequence of intermediate frames consisting of the vertex frame and its preceding and following frames.

[0020] As a further technical solution, the video sequence is preprocessed, including vertex frame positioning, for determining the position of the vertex frame in the micro-expression sequence, for extracting time information before and after the frame;

[0021] The vertex frame and the start frame are processed to obtain an optical flow map representing the dynamic changes of micro-expressions, and the optical flow map is processed to obtain an optical strain map.

[0022] As a further technical solution, the second branch network and the third branch network have the same structure, both including: three parallel streams, a channel layer, a facial feature enhanced attention module, a second fusion module, and a fully connected layer;

[0023] In the second branch network, each parallel stream uses a convolution kernel with different parameters to extract features of different sizes. After passing through the parallel streams, the features in the three parallel streams are spliced ​​together at the channel level to obtain the first fusion feature;

[0024] The first facial feature enhanced attention module combines and weighted averages the first fused features;

[0025] The feature obtained by the first facial feature enhanced attention module is further fused with the first fused feature to obtain a second fused feature;

[0026] The second fusion feature is processed by the fully connected layer and the features of the optical flow map are output.

[0027] As a further technical solution, the facial feature enhancement attention module includes a convolutional attention module, a double pooling layer and a fully connected layer, and the convolutional attention module includes a channel attention module and a spatial channel attention module;

[0028] Perform maximum pooling on the first fusion feature to obtain the maximum pooling layer output feature;

[0029] Perform average pooling on the first fusion feature to obtain the average pooling layer output feature;

[0030] The output features of the maximum pooling layer and the average pooling layer are combined and input into the fully connected layer to obtain the output features of the fully connected layer;

[0031] Input the output features of the fully connected layer into the channel attention module to obtain the channel attention features;

[0032] Perform channel attention weighting on the channel attention feature and the fully connected layer output feature to obtain the channel attention weighted feature;

[0033] Input the channel attention weighted features into the spatial attention module to obtain the spatial attention features;

[0034] Perform spatial attention weighting on the spatial attention feature and the channel attention weighted feature to obtain the spatial attention weighted feature;

[0035] The spatial attention feature is fused with the first fusion feature to obtain the second fusion feature.

[0036] As a further technical solution, the second fusion feature is processed by a fully connected layer, wherein the fully connected layer includes three layers, and the processing includes:

[0037] The first two fully connected layers map the fused second fusion features to higher-level abstract representations and gradually reduce the feature dimensions;

[0038] The third fully connected layer transforms the features output by the first two fully connected layers into a dimension suitable for the output of the second branch network.

[0039] As a further technical solution, the optical flow map is resampled before being input into the second branch network and the optical strain map is resampled before being input into the third branch network.

[0040] As a further technical solution, the first branch network includes a deep convolutional neural network and a channel attention module; the channel attention module includes a global average pooling module and a maximum pooling module;

[0041] After the video frames are input into the deep convolutional neural network, the original features of the face are extracted;

[0042] The original features of the face are input into the adaptive average pooling, average pooling is performed on each channel, and the average pooling channel attention weight is obtained through convolution and function activation processing;

[0043] Multiply the average pooling channel attention weight with the original face feature to obtain the global average pooling output feature;

[0044] The original features of the face are input into the adaptive maximum pooling for maximum pooling, and the maximum pooling channel attention weight is obtained through convolution and function activation processing;

[0045] Multiply the maximum pooling channel attention weight with the original face feature to obtain the maximum pooling output feature;

[0046] The global average pooling output features are fused with the maximum pooling output features to obtain the facial features.

[0047] As a further technical solution, after obtaining the micro-expression classification results, the total loss function is constructed through the two loss functions of IM Loss and Focal Loss to optimize the model.

[0048] The second aspect discloses a micro-expression recognition system based on a three-branch network, comprising:

[0049] A preprocessing module, used for acquiring a video sequence, preprocessing the video sequence to obtain an optical flow map, and further processing the optical flow map to obtain an optical strain map, wherein the video sequence includes video frames;

[0050] The feature extraction and fusion module is used to input the video frame into the first branch network, the first branch network extracts facial features based on deep learning and deep enhanced channel attention; input the optical flow map into the second branch network, the second branch network uses the second shallow convolutional neural network and the second facial feature enhanced attention to extract the features of the optical flow map; input the optical strain map into the third branch network, the third branch network uses the third shallow convolutional neural network and the third facial feature enhanced attention to extract the features of the optical strain map; fuse the facial features, the features of the optical flow map, and the features of the optical strain map;

[0051] The recognition and classification module is used to classify micro-expressions based on the fused features to obtain the classification results of micro-expressions.

[0052] One or more of the above technical solutions have the following beneficial effects:

[0053] In this embodiment, by constructing a micro-expression recognition model based on a three-branch network, a three-branch network composed of a deep convolutional neural network and a shallow 3D convolutional neural network, and incorporating multiple attention modules, the problems of insufficient micro-expression samples, difficulty in learning multiple types of features, low recognition accuracy and complex model are solved.

[0054] In this embodiment, a deep enhanced channel attention module ECANet is introduced in the deep convolutional neural network, and a new facial feature enhanced attention module is proposed in the shallow 3D convolutional neural network. The facial features, optical flow features and optical strain features are extracted and processed respectively through three branches and fused layer by layer, so as to make full use of the different information implicit in the features of each layer, reduce the loss of subtle facial features during the recognition process, and significantly improve the accuracy of micro-expression recognition.

[0055] In this embodiment, the IM Loss function is introduced into the micro-expression recognition model and combined with the Focal Loss function, so that the model can more effectively process samples that are difficult to classify, effectively improving the model's recognition ability for difficult samples.

[0056] The experiments of this embodiment on CASME II, SMIC, SAMM micro-expression datasets and MEGC2019 composite micro-expression dataset show that the proposed method is superior to traditional methods and existing deep learning methods in micro-expression recognition. The method of this embodiment can be effectively applied to multi-classification micro-expression recognition.

[0057] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0059] Figure 1 This is a flow chart of a micro-expression recognition method based on a three-branch network in the first embodiment of the present invention;

[0060] Figure 2 An intermediate frame sequence constructed for samples in the CASMEII data set of the first embodiment;

[0061] Figure 3 3DCNN-FEAM network schematic diagram of the first embodiment;

[0062] Figure 4 Schematic diagram of the facial feature enhanced attention module FEAM of the first embodiment;

[0063] Figure 5 This is a schematic diagram of the ECANet module of the first embodiment;

[0064] Figure 6 This is the confusion matrix of the MEGC2019 dataset in this embodiment 1 for different methods. DETAILED DESCRIPTION

[0065] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0066] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0067] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0068] Embodiment 1

[0069] This embodiment discloses a micro-expression recognition method based on a three-branch network.

[0070] In order to more clearly illustrate this embodiment, the micro-expression recognition process based on the three-branch network can be specifically described as follows:

[0071] This embodiment provides a micro-expression recognition method based on a three-branch network, including:

[0072] S1, obtaining a video sequence, preprocessing the video sequence to obtain an optical flow map, and further processing the optical flow map to obtain an optical strain map, the video sequence includes video frames;

[0073] S2. Input the video frame into the first branch network, and the first branch network extracts facial features based on deep learning and deep enhanced channel attention;

[0074] S3, inputting the optical flow map into the second branch network, and the second branch network uses the second shallow convolutional neural network and the second facial feature to enhance attention and extract features of the optical flow map;

[0075] S4, inputting the light strain map into the third branch network, and the third branch network extracts features of the light strain map using a third shallow convolutional neural network and a third facial feature enhanced attention;

[0076] S5, fusing the facial features, the features of the optical flow map, and the features of the optical strain map;

[0077] S6. Classify micro-expressions based on the fused features to obtain micro-expression classification results.

[0078] like Figure 1 , Figure 2 As shown, in step S1, a video sequence is acquired, and the video sequence is preprocessed to obtain an optical flow map, and the optical flow map is further processed to obtain an optical strain map, and the video sequence includes video frames.

[0079] In a micro-expression video sequence, the start frame indicates the moment when the expression begins.

[0080] The video frame is a apex frame, which refers to the frame where the intensity or expression of the micro-expression is most obvious, or a sequence of intermediate frames consisting of the apex frame and its preceding and following frames.

[0081] S101, preprocessing the video sequence, including positioning the vertex frame, for determining the position of the vertex frame in the micro-expression sequence, for extracting time information before and after the frame;

[0082] S1011. Obtain a data set containing a video sequence.

[0083] In this embodiment, four datasets, CASME II, SMIC, SAMM and MEGC2019, are used.

[0084] Specifically, micro-expressions in the SMIC dataset are divided into three categories: positive, negative, and surprise.

[0085] The CASMEII and SAMM datasets were repartitioned to unify the sample labels, and the original labels were remapped to the new label space to better avoid the ambiguity of emotion categories caused by different stimuli and environmental settings.

[0086] The specific operation is as follows: "disgust", "anger", "depression", "contempt", "sadness" and "fear" samples are classified as "negative" samples; "happy" samples are classified as "positive" samples; "surprise" samples remain unchanged; "other" samples cannot be classified and are therefore not used.

[0087] MEGC2019 is a composite dataset with 24, 16 and 28 subjects drawn from CASME II, SMIC and SAMM, respectively, and contains 442 samples from 68 subjects from different backgrounds (race, environment and gender).

[0088] Each time, all facial micro-expression data of one subject in the dataset are kept for testing, and the rest are used for training.

[0089] S1012, obtaining a starting frame and a vertex frame of a micro-expression sequence from a data set.

[0090] In a micro-expression sequence, the apex frame refers to the frame where the intensity or expression of the micro-expression is most obvious, and apex frame positioning refers to determining the position of the apex frame in the micro-expression sequence. The information contained in the apex frame can largely represent the characteristics of the entire micro-expression sample. After determining the position of the apex frame, the time information before and after the frame can be extracted to further analyze the temporal characteristics of the micro-expression.

[0091] Get the start frame and vertex frame from the dataset.

[0092] Among them, the CASMEII, SAMM datasets and MEGC2019 have marked the start frame and vertex frame. The marked datasets show that the vertex frame of micro-expressions usually appears in the middle part of the video. The frame sequence before and after the vertex frame of micro-expressions has relatively small expression changes. For the unlabeled SMIC dataset, the middle frame of the sample video is also selected as the vertex frame.

[0093] S102, intermediate frame enhancement: the vertex frame is replaced with an intermediate frame sequence consisting of previous and next frames.

[0094] Micro-expressions have instantaneous characteristics. The apex frame of a micro-expression is the frame with the highest expression intensity. The expression difference between the apex frame of a micro-expression and the frames before and after it is very small, and the change in expression intensity is very subtle.

[0095] In this embodiment, in order to enrich the sample data, the vertex frame and the three frames before and after it are combined into an intermediate frame sequence to achieve intermediate frame enhancement, such as Figure 2 As shown, the intermediate frame sequence is constructed for the samples in the CASMEII dataset.

[0096] In this embodiment, the vertex frame and the preceding and following 0 frames, 3 frames, 5 frames, 7 frames, 10 frames and 15 frames are respectively taken from multiple data sets to perform micro-expression recognition experiments under the same conditions, and the unweighted average recall rate (UAR) is calculated as the evaluation criterion. Table 1 shows the experimental results on the CASME II and SAMM data sets.

[0097] Table 1 Comparison of experimental results of different intermediate frame sequences selected in CASME II and SAMM datasets

[0098]

[0099] It can be seen that compared with selecting 0 frames before and after the vertex frame (i.e., selecting only the vertex frame), selecting the vertex frame and the frames before and after to form an intermediate frame sequence as the input of the model can improve the UAR results of the micro-expression recognition experiment, indicating that incorporating the frames before and after the vertex frame is effective in improving the recognition accuracy. In addition, the UAR result obtained by selecting the vertex frame and the 3 frames before and after on the CASMEII dataset is the best and has the highest time efficiency. Although the UAR result obtained by selecting the vertex frame and the 5 frames before and after on the SAMM dataset is the best, it is very close to the UAR result obtained by selecting the vertex frame and the 3 frames before and after, and the time efficiency is significantly lower than that of selecting the vertex frame and the 3 frames before and after. Therefore, considering the evaluation results and time efficiency, the effect of selecting the vertex frame and the 3 frames before and after is better.

[0100] The intermediate frame enhancement technology can select reasonable frame samples from a given set of micro-expression sequences with better efficiency to enrich the input of the network model, helping the network model to better learn the characteristics of micro-expressions, thereby achieving better detection results. Therefore, the intermediate frame enhancement technology can improve the problem of low model prediction accuracy caused by insufficient micro-expression sample data to a certain extent.

[0101] S103, processing the vertex frame and the start frame to obtain an optical flow map representing the dynamic changes of micro-expressions, and processing the optical flow map to obtain an optical strain map.

[0102] S1031, processing the vertex frame and the start frame to obtain an optical flow graph representing the dynamic changes of micro-expressions.

[0103] Optical flow is a method for estimating the motion of objects in a video. It is an approximation of the image pattern based on the local derivatives between two frames of images, and can show obvious motion changes between frames. The optical flow method reflects the motion of objects in a video in two consecutive frames through the optical flow field. The optical flow field is a two-dimensional motion field that represents the magnitude and direction of the motion of each pixel on the image. The optical flow field reflects the area where micro-expression movements occur on the face more clearly than the original image, allowing the model to learn more complex and abstract features related to the tiny movements and deformations of micro-expressions. For small facial pixel movements, the optical flow field helps to express the degree of deformation of facial muscle tissue.

[0104] In a micro-expression video sequence, the start frame indicates the moment when the expression begins, and the vertex frame is the moment when the expression reaches its maximum intensity. The frame difference between the two defines the time span of the expression movement. The selection of the start frame and the vertex frame mainly determines the time range of the optical flow calculation, so both directly affect the quality of the generated optical flow map.

[0105] In this embodiment, for a micro-expression video sequence, I (x, y, t) represents the brightness value of the image at the pixel point (x, y) at time t. Based on the assumption of constant optical flow (brightness consistency assumption), the position of each pixel in the starting frame is (x, y). After the vertex frame, the pixel may move (x+dx, y+dy). The u and v in the optical flow field represent the movement speed of the pixel in the horizontal direction (dx) and vertical direction (dy), respectively. u=dt / dx, v=dt / dy, where dt is the time difference between the starting frame and the vertex frame. The optical flow field calculated from these two frames is as follows:

[0106] Oi={(u(x,y)),v(x,y))|x=1,2,..,X,y=1,..,Y} (1)

[0107] Where X and Y represent the frame f i,j The width and height of , u(x,y) and v(x,y) are the optical flow components in the horizontal and vertical directions respectively.

[0108] The optical flow vector field O is visualized to generate an optical flow map, where color and intensity represent the direction and speed of movement.

[0109] The optical flow field is a vector field that describes the movement direction and speed of each pixel in the image and is the core result of the calculation process.

[0110] The optical flow map is a visual representation of the optical flow field, and color coding is usually used to show the direction and intensity of movement. The color indicates the direction of movement, and the intensity (brightness) indicates the speed of movement. The optical flow map is the result of visualizing the optical flow field.

[0111] After the above steps, the optical flow map can help us intuitively understand the direction and amplitude of facial movement.

[0112] S1032: Process the optical flow map to obtain an optical strain map.

[0113] Since the strain pattern is only related to the deformation of the face, it is not easily affected by factors such as lighting conditions and facial occlusion, and has a good performance in micro-expression recognition tasks. Therefore, given the optical flow vector, optical strain can be derived to describe the facial movement pattern. For sufficiently small facial pixel movements, optical strain can represent the deformation size of facial muscle tissue. Optical strain is expressed as follows:

[0114]

[0115] Among them, u=[u,v] T is a displacement vector, which represents the projection of the displacement caused by the facial expression deformation in three-dimensional space onto the two-dimensional image; It means to take the derivative of u.

[0116] Expand formula (2) into matrix form:

[0117]

[0118] Among them, the diagonal strain component (ε xx ,ε yy ) is the normal strain component, (ε xy ,ε yx ) is the shear strain component.

[0119] Specifically, normal strain measures the change in length perpendicular to a specific direction (such as the x or y direction), while shear strain measures the change in angular displacement in a plane due to shear force. Since muscle movement during micro-expression movements may involve multiple directions, the optical strain magnitude of each pixel can be calculated by the sum of the squares of the normal strain component and the shear strain component, as follows:

[0120]

[0121] Optical strain calculation describes the deformation information of local facial movement and quantifies the degree of pixel displacement. The optical strain map converts this information into an intuitive expression through visualization, namely the optical strain map.

[0122] After the above steps, the optical strain is visualized to obtain the optical strain map, which is usually used to analyze the deformation and intensity of the facial micro-expression area. The optical flow map mainly reflects the direction and speed of movement, while the optical strain map further quantifies the degree of deformation caused by movement. The two together provide important feature information for micro-expression recognition.

[0123] like Figure 1 , Figure 5 As shown, in step S2, the video frame is input into the first branch network, and the first branch network extracts facial features based on deep learning and deep enhanced channel attention.

[0124] In this embodiment, the three-branch network includes a first branch network, a second branch network and a third branch network.

[0125] The first branch network includes a deep convolutional neural network and a channel attention module, and the channel attention module includes a global average pooling module and a maximum pooling module.

[0126] In this embodiment, the first branch network introduces ResNet-18 as a migration model and adds a deep enhanced channel attention module ECANet to extract facial features.

[0127] ResNet-18 is a classic deep convolutional neural network suitable for image feature extraction. It has multiple layers of convolution and pooling layers, which can automatically learn and capture features at different levels, including low-level textures and high-level semantic information. Micro-expressions often contain small but important facial dynamic changes, which ResNet-18 cannot capture. The ECANet attention mechanism focuses on enhancing the network's attention to specific channels, which is particularly important for micro-expression recognition because different channels may contribute differently to different facial features. By adaptively adjusting the weights of the channels, ECANet can highlight those channels that contain micro-expression information and reduce attention to irrelevant information. This attention mechanism helps to improve the perception and expressiveness of the network, so ResNet-18 is combined with the ECANet attention mechanism.

[0128] The ECANet module includes the Adaptive Average Pooling (GAP) module and the Adaptive Max Pooling (AMP) module, which are used to capture the key features of the input tensor. Channel attention is calculated through Adaptive Average Pooling (Global Average Pooling) and Adaptive Max Pooling (Adaptive Max Pooling), thereby enhancing the representation ability of input features. The ECANet module adopts a local cross-channel interaction strategy without dimensionality reduction, which can avoid the adverse effects of dimensionality reduction operations on channel attention prediction, thereby effectively reducing the complexity of the model while maintaining good performance.

[0129] The first branch network extracts facial features. The specific process is as follows:

[0130] S201, after inputting the video frame into the deep convolutional neural network, extracting the original features of the face.

[0131] like Figure 1 , Figure 5As shown, in this embodiment, the video frame is input into the ResNet-18 network to obtain the original face feature X, and the output feature X is respectively input into the adaptive average pooling (GAP) module and the adaptive maximum pooling (AMP) module of the ECANet module.

[0132] S202, inputting the original features of the face into adaptive average pooling, performing average pooling on each channel, obtaining the average pooling channel attention weight through convolution and function activation processing, multiplying the average pooling channel attention weight with the original features of the face, and obtaining the global average pooling output features.

[0133] In this embodiment, the original face feature X is input into the adaptive average pooling (GAP) module, and the feature vector X with a dimension of H×W×C is average pooled on each channel to generate a feature vector with a spatial dimension of 1×1×C, where C is the number of channels. Then, the channel attention weight λ is obtained by convolution and Sigmoid activation. 1 , the weight λ 1 Multiply with the original input X to get the global average pooling output feature

[0134] S203, inputting the original features of the face into the adaptive maximum pooling for maximum pooling, obtaining the maximum pooling channel attention weight through convolution and function activation processing, multiplying the maximum pooling channel attention weight with the original features of the face, and obtaining the maximum pooling output features.

[0135] In this embodiment, the original face feature X is input into the adaptive maximum pooling (AMP) module, the adaptive maximum pooling is applied to the input feature X, and the channel attention weight λ is obtained by convolution and Sigmoid activation processing. 2 , multiply the weight by the original input X to get the output feature of adaptive max pooling

[0136] S204, performing feature fusion on the global average pooling output features and the maximum pooling output features to obtain facial features.

[0137] In this embodiment, the output features of the adaptive maximum pooling module are And the output features of adaptive max pooling Perform feature fusion to obtain facial features.

[0138] After the above steps, deeper feature extraction and representation can be achieved in the micro-expression recognition task. ResNet-18 provides basic feature extraction capabilities, while the ECANet attention mechanism further optimizes and strengthens these features. This synergy can effectively extract and capture key information in micro-expression images and improve the performance and robustness of the model.

[0139] like Figure 1 , Figure 3 and Figure 4 As shown, in step S3, the optical flow map is input into the second branch network, and the second branch network uses a second shallow convolutional neural network and a second facial feature enhanced attention to extract features of the optical flow map.

[0140] like Figure 3 , Figure 4 As shown, in this embodiment, the second branch network adopts a trained shallow 3DCNN model and combines it with a new facial feature enhanced attention module FEAM (Face-Enhanced Attention Module) to extract the features of the optical flow map.

[0141] The optical flow and optical strain features contain important information about motion, shape change, and time series. Obviously, the use of 3DCNN networks helps to extract more information about these spatiotemporal characteristics. Therefore, the second and third branches of the model in this paper introduce 3DCNN networks to extract optical flow features and optical strain features, respectively.

[0142] The convolution layers in 3D convolutional neural networks (3DCNN) are all three-dimensional, and use 3D convolution kernels with different parameters to extract spatiotemporal features in different time domains, while extracting features in both time and space domains. Each 3D convolution layer of the 3DCNN network has the ability to learn features at different levels, from low-level features such as edges and textures to high-level features, enabling the network to build and express complex feature hierarchies layer by layer. In addition, the 3DCNN network has fewer parameters and computational complexity, so it can extract both discriminative high-level features and micro-expression details through lightweight computation.

[0143] Since micro-expression movements are localized and only appear in some areas of the face, there are no effective classification features in most facial areas. Only a few areas with micro-expression movements can provide information that helps to classify micro-expressions. The attention mechanism helps to focus on specific facial areas and learn and obtain important features. Therefore, based on the use of 3DCNN to learn optical flow features and optical strain features, the new FEAM facial feature enhancement attention module is added to construct a new branch network 3DCNN-FEAM.

[0144] The second branch network and the third branch network have the same structure, both including: three parallel streams, a channel layer, a facial feature enhanced attention module, a second fusion module, and a fully connected layer.

[0145] Extract the features of the optical flow map. The specific process is:

[0146] S301. In the second branch network, each parallel stream uses a convolution kernel with different parameters to extract features of different sizes. After passing through the parallel streams, the features in the three parallel streams are spliced ​​together at the channel level to obtain the first fused feature.

[0147] like Figure 3 As shown, in this embodiment, the optical flow image is resampled and then input into the second branch network 3DCNN-FEAM.

[0148] In the second branch network, the optical flow map passes through three parallel streams, each of which consists of a convolution layer and a maximum pooling layer. Each parallel stream uses convolution kernels with different parameters to extract features of different sizes, and supplements small-scale inputs with different numbers of 5*5 convolution kernels to avoid data underfitting problems. The feature maps are then stacked by channel and enter the maximum pooling layer. The maximum pooling operation can eliminate redundancy while highlighting the main features. After passing through the parallel streams, the features in the three streams are spliced ​​together at the channel level to achieve the first fusion and obtain the first fusion feature F.

[0149] S302: The first facial feature enhanced attention module combines and weighted averages the first fusion features.

[0150] The first fusion features are input into the FEAM facial feature enhanced attention module, and the features are combined and weighted averaged to enhance the model's ability to capture key information, thereby obtaining the features of the FEAM facial feature enhanced attention module.

[0151] The facial feature enhanced attention module includes a convolutional attention module, a double pooling layer and a fully connected layer. The convolutional attention module includes a channel attention module and a spatial channel attention module.

[0152] like Figure 4 As shown, in this embodiment, the facial feature enhanced attention module FEAM introduces a double pooling layer and a fully connected layer based on the convolutional attention module structure to further improve the expressiveness of the feature map, so that the neural network can focus more on capturing subtle changes in the face. This module enables the neural network to place more emphasis on the extraction of facial features during the learning process, which helps to more accurately capture key information such as facial expressions and morphology.

[0153] The dual pooling layer includes average pooling and maximum pooling. Average pooling captures the overall information of the feature, while maximum pooling focuses on the most significant part of the feature. The dual pooling operation enables the network to consider information of different scales simultaneously when processing features, thereby more comprehensively understanding the structure of the face. The processing of the fully connected layer helps to further integrate feature information, so that the network can better understand the relationship between different features during the learning process and improve the perception of facial details.

[0154] S3021. Perform maximum pooling on the first fusion feature to obtain the maximum pooling layer output feature, perform average pooling on the first fusion feature to obtain the average pooling layer output feature, merge the maximum pooling layer output feature and the average pooling layer output feature and input them into the fully connected layer to obtain the fully connected layer output feature.

[0155] In this embodiment, the first fusion feature F is input to the maximum pooling layer and the average pooling layer of the FEAM module respectively, and the maximum pooling layer output feature F is obtained respectively. max , average pooling layer output feature F avg , the maximum pooling layer output feature F avg And the average pooling layer output feature F avg Merge the input to the fully connected layer to obtain the fully connected layer output feature F fully , the formula is:

[0156]

[0157] Among them, F fully represents the output feature of the fully connected layer, F max represents the output feature of the maximum pooling layer, F avg Average pooling layer output features, W 1 Represents the weight parameters in the fully connected layer, which is a trainable matrix used to linearly transform the feature input F max and F avg The feature representation of .

[0158] S3022. Input the output features of the fully connected layer into the channel attention module to obtain the channel attention features.

[0159] The fully connected layer outputs the feature F fully Input to the channel attention module to obtain the channel attention feature M chan , the formula is:

[0160] M chan =σ(W 2 ·F fully ) (6)

[0161] Among them, W 2Represents the weight parameter of the channel attention module, which is another trainable matrix used to adjust F full y The importance of each channel in .

[0162] S3023. Perform channel attention weighting on the channel attention feature and the fully connected layer output feature to obtain a channel attention weighted feature.

[0163] The channel attention feature M chan And the fully connected layer output feature F fully Perform channel attention weighting to obtain the channel attention weighted feature F c h an , the formula is:

[0164]

[0165] S3024. Input the channel attention weighted features into the spatial attention module to obtain the spatial attention features.

[0166] The channel attention weighted feature F chan Input into the spatial attention module to obtain the spatial attention feature M spat , the formula is:

[0167] M spat =σ(f conv (F chan ))(8)

[0168] Among them, f conv is the convolution operation.

[0169] S3025. Perform spatial attention weighting on the spatial attention feature and the channel attention weighted feature to obtain the spatial attention weighted feature.

[0170] The spatial attention feature M spat and the channel attention weighted feature F chan Perform spatial attention weighting to obtain the spatial attention weighted feature F spat , the formula is:

[0171]

[0172] S303, further fusing the features obtained by the first facial feature enhanced attention module with the first fused features to obtain a second fused feature.

[0173] The spatial attention feature F spat Fuse with the first fusion feature F to obtain the final output feature F out , the formula is:

[0174] Fout =concat(F,F spat ) (10)

[0175] Achieve the second fusion and obtain the second fusion features.

[0176] S304, processing the second fusion feature through a fully connected layer and outputting the features of the optical flow map.

[0177] The second fused features are processed by a fully connected layer, wherein the fully connected layer includes three layers, and the processing includes: the first two fully connected layers respectively map the fused second fused features to a higher level of abstract representation and gradually reduce the feature dimension;

[0178] The third fully connected layer transforms the features output by the first two fully connected layers into a dimension suitable for the output of the second branch network.

[0179] Specifically, the second fusion features are passed through three fully connected layers. The first two fully connected layers, also called dense layers of different dimensions, namely dense layer 1 and dense layer 2, respectively map the second fusion features to a higher level of abstract representation and gradually reduce the feature dimensions. The third fully connected layer converts the features into dimensions suitable for the network output to obtain optical flow features, thereby realizing the extraction of optical flow features.

[0180] like Figure 1 , Figure 3 and Figure 4 As shown, in step S4, the optical strain map is input into the third branch network, and the third branch network uses the third shallow convolutional neural network and the third facial feature enhanced attention to extract the features of the optical strain map.

[0181] The third branch network uses a trained shallow 3DCNN model and combines it with a new face-enhanced attention module FEAM (Face-Enhanced Attention Module) to extract the features of the optical strain map.

[0182] The third branch network has the same structure as the second branch network, including: three parallel streams, channel layer, second facial feature enhanced attention module, second fusion module, and fully connected layer;

[0183] like Figure 3 As shown, in this embodiment, the optical strain image is resampled and then input into the third branch network 3DCNN-FEAM to extract the optical strain features.

[0184] The specific extraction process is the same as step S301, and the features of the optical strain map are obtained, thereby realizing feature extraction of the optical strain map.

[0185] like Figure 1As shown, in step S5, the facial features, the features of the optical flow map, and the features of the optical strain map are fused.

[0186] S501, fusing facial features, optical flow features and optical strain features to obtain aggregated features.

[0187] S5011. Merge the features of the optical flow map and the optical strain map output by the two 3DCNN-FEAM branch networks to form a new optical flow feature block.

[0188] In this embodiment, the feature F of the optical flow graph optical The characteristic F of the optical strain map strain Through feature concatenation, the fused feature F fused As the input of the subsequent feature processing module, the formula is:

[0189]

[0190] S5012. Perform L2 regularization on the optical flow feature block and the face feature block output by the ResNet-18-ECA network respectively.

[0191] In this embodiment, the optical flow feature block F fused And the face feature block F output by the ResNet-18-ECA network face L 2 Regularization processing, the formula is:

[0192]

[0193] Among them, ||F|| 2 Represents the L2 norm of the feature.

[0194] Through the L2 regularization operation, the eigenvalues ​​are normalized to control the model complexity and reduce the risk of overfitting, and by constraining the weight parameters during the learning process, unnecessary noise and overfitting problems in the model are prevented.

[0195] S5013, fusing the regularized optical flow feature block and the face feature block to obtain aggregated features.

[0196] In this embodiment, the regularized optical flow feature blocks and face feature blocks Perform feature fusion to obtain the aggregated feature F final , the formula is:

[0197]

[0198] After the above steps, effective recognition features are obtained, which provide data support for subsequent feature classification.

[0199] like Figure 1 As shown, in step S6, the fused features are subjected to micro-expression classification to obtain micro-expression classification results.

[0200] S601. Probability prediction.

[0201] Use the normalized exponential function (Softmax) to predict the aggregated features. The prediction formula is:

[0202]

[0203] According to formula (14), the probability values ​​of the three micro-expression categories of positive, negative and surprised are obtained respectively. The category with the highest probability value is the classification result of the micro-expression. The formula is:

[0204]

[0205] Among them, y predicted Represents the final predicted category (positive, negative, or surprised) of the input micro-expression video.

[0206] S602: Constrain the prediction probability through a loss function.

[0207] For each sample in the micro-expression dataset (i.e., a micro-expression sequence), a corresponding probability distribution will be obtained after being processed by the above network model. This distribution is composed of the probability of each category output by the last Softmax layer of the network. The lack of micro-expression sample data is one of the important reasons for the imbalanced distribution of sample categories. In order to effectively solve the problem of unbalanced distribution of micro-expression data sample categories, a new loss function is proposed to constrain the predicted probability distribution of the network model to improve the accuracy of the model in micro-expression recognition. Introducing the IM Loss (Information Maximizing Loss) function into micro-expression recognition and combining it with the Focal Loss function to jointly constrain the model can better improve the problem of unbalanced sample distribution caused by lack of sample data, which in turn leads to low model recognition accuracy.

[0208] Among them, the IM Loss function constrains the probability distribution corresponding to the sample, and its goal is to maximize the degree of correlation between the predicted probability distribution and the true label distribution, that is, the amount of information shared between the two distributions, so as to make the probability distribution generated by the model as close to the true distribution as possible, and then enhance the model's recognition ability for all categories. The Focal Loss function constrains the predicted probability of a specific category, that is, it focuses on a single probability value, especially the probability value of samples that are difficult to predict or misclassified. By adjusting their loss weights, the model's recognition ability for minority classes can be improved. Since the probability distribution depends on the probability value, the Focal Loss function can essentially be regarded as a constraint on the probability distribution corresponding to the sample.

[0209] The IM Loss function was introduced into micro-expression recognition for the first time. The IM Loss function consists of two parts: the entropy minimization part and the diversity maximization part.

[0210] The purpose of entropy minimization is to ensure that the predicted label distribution of the model output is as certain as possible for a given input sample, that is, to reduce the uncertainty of the model. Mathematically, entropy is a measure of the uncertainty of a random variable. For a probability distribution, the smaller the entropy, the more the distribution tends to a certain category and the smaller the uncertainty. In classification tasks, entropy minimization usually means that the model is very confident in its prediction results.

[0211] The diversity maximization part aims to ensure that the model gives different categories of prediction results for different input samples, that is, to increase the diversity of the model output. This is especially important for tasks in scenarios where samples are in different domains or sample categories are unbalanced, because the model may overfit samples of a large number of categories and ignore the minority categories. By maximizing diversity, it can be ensured that each category receives appropriate attention, making the overall prediction more balanced.

[0212] The purpose of the IM Loss function is to make the classification labels as diverse as possible, that is, each category is relatively average. To achieve this goal, it is only necessary to calculate the average value of the output of the entire target domain to maximize its entropy, keep those that are difficult to classify away from the classification boundary, and increase confidence.

[0213] The IM Loss function is defined as:

[0214]

[0215] Among them, L ent represents the entropy minimization part of the IM Loss function, which is used to calculate the cross entropy loss of the prediction probability of each sample and average it over the entire dataset; R represents all micro-expression datasets in this embodiment, R = {x 1 ,x2 ,...,x V}, where sample x t ,t=1,2,...V represents a micro-expression video sequence in the micro-expression dataset R, and V is the total number of samples in the dataset R; Represents each sample x in the micro-expression dataset R t The corresponding losses are summed and averaged, for each sample x t The corresponding loss is the sample x under the current model parameters t The sum of the cross entropy losses of the predicted probabilities of all possible categories; K is the total number of sample categories in the dataset, Predict sample x for the model t The probability value of belonging to category k; L div Represents the diversity maximization part of the IMLoss function, which is used to directly calculate each sample x t The KL divergence between the predicted probability distribution and the uniform distribution is used to promote the diversity of predictions and avoid bias towards certain categories, and these values ​​are summed; is the average probability that each sample is predicted to be class k, is the expected probability of a uniform distribution, and this paper assumes that the probability of each category being selected is equal.

[0216] Through the above steps, the diversity and averageness of the classification labels are ensured while enhancing the confidence of the samples that are difficult to classify in the target domain, enhancing the confidence of the model in micro-expression classification and ensuring balanced recognition of various micro-expressions.

[0217] In traditional cross entropy loss, all samples are treated equally, which can easily lead to the model being dominated by a large number of easy-to-classify samples, thereby ignoring the few but difficult-to-classify samples. To solve this problem, an adjustable factor is introduced in the Focal Loss function, which can dynamically adjust the weight of each sample.

[0218] The Focal Loss function is defined as:

[0219]

[0220] in, Indicates that the model will x t The prediction is the probability value of category k, and Focal Loss calculates its loss independently for each sample in the dataset; where k∈[k 1 ,k 2 ,k 3 ],k 1 For class positive, k 2 is the negative class, k 3 Surprised by the category; αk It is a preset weight parameter used to balance the weights of different categories. The weight of negative samples is set smaller. γ is an adjustment parameter. By setting γ, the weight of difficult-to-classify samples can be increased, thereby increasing the model's attention to difficult-to-classify samples.

[0221] After the above steps, the problems caused by category imbalance can be alleviated, especially for a small number of samples that are difficult to classify, so that the model can pay more attention to those difficult-to-classify samples, so that the model can have better classification effects when facing samples of various categories.

[0222] In order to improve the model's ability to identify difficult-to-classify samples and ensure the diversity and certainty of the output, a strategy of weighting samples based on the difficulty of classifying the sample categories is adopted. Specifically, Focal Loss pays more attention to difficult-to-classify samples by dynamically adjusting weights to address the problem of class imbalance, while the IM Loss function increases the diversity of the model output while improving the certainty of the model.

[0223] The Total Loss function is defined as:

[0224] L Total (x t )=αL focal (p k (x t )+βL LM (p k (x t )) (18)

[0225] Among them, α and β are weight coefficients used to adjust the proportion of the two losses, p k(xt ) indicates that the model converts x t The probability value predicted as category k, L focal (p k (x t )) represents the Focal Loss function, L IM (p k (x t )) represents the IM Loss function.

[0226] After the above steps, the model can not only pay enough attention to difficult samples at the overall level of the dataset, but also distinguish difficult-to-distinguish samples within a specific category, thereby improving the classification accuracy and robustness of the model.

[0227] Through steps S1-S6, the micro-expression recognition model based on the three-branch network is trained and tested. Each time, all micro-expressions of a subject are selected for testing, and the micro-expressions of other subjects are used as training data. After the trained model is obtained, the micro-expressions of the selected subject are input into the model for testing. To ensure that the information of the tester is excluded during the training phase, iterate the cross-validation y times and end the test. y is set according to the number of subjects.

[0228] In this embodiment, the evaluation criteria of unweighted average recall (UAR) and unweighted F1 score are used for evaluation. UF1 is also called macro-average F1 score, which is determined by averaging the F1 scores of each class. Among them, the F1 score can be interpreted as the weighted average of precision and recall, reaching the best value when the F1 score is equal to 1 and the worst value when it is equal to 0. The relative contributions of precision (precision, P) and recall (recall, R) to the F1 score are the same. Therefore, UF1 is a good choice in multi-classification problems, as it can equally emphasize rare classes. In order to calculate UF1, the F1 value of each category k is first calculated, and then the average of the F1 values ​​of each category is calculated, which is UF1, and the formula is:

[0229]

[0230]

[0231] Among them, Q is the number of labels of micro-expressions, F1 k is the F1 value of the classification result of category k, TP k , FP k and FN k They represent the number of true positive, false positive, and false negative samples in the classification results of category k respectively.

[0232] Unweighted average recall (UAR), also known as balanced accuracy, is a more reasonable evaluation criterion to replace weighted average recall. UAR is obtained by adding the accuracy of each category and averaging the number of categories. The formula is:

[0233]

[0234] Among them, TP c represents the number of true positive samples in the classification results of category c, P c is the predicted probability of category c, M c Represents the total number of samples in category c, that is, the number of samples that actually belong to category c in the dataset. It is used to weightedly calculate UAR to balance the problem of uneven number of class samples.

[0235] The advantage of this evaluation method is that compared with the weighted average recall, its prediction results will not be biased towards larger categories in the data. The number of samples of different categories in the MEGC2019 composite dataset is unbalanced. The ratio of the three types of expression samples, surprise, positive and negative, is approximately 1:1.3:3, which has a serious data category imbalance problem. Therefore, this paper uses the unweighted average recall (UAR) and unweighted F1-score (UF1) indicators to evaluate the performance of the method for three-category classification. These two indicators can well measure the performance of the model in terms of accuracy and precision of positive example recognition. By treating each class equally, it ensures that the model's ability to correctly identify classes with lower frequencies is also taken into account, thereby evaluating its performance more fairly.

[0236] In this embodiment, experiments are conducted on the data sets CASMEII, SMIC and SAMM, and the MEGC2019 composite data set. The method of this embodiment is compared with the method based on manual feature extraction and the method based on deep learning for recognition. The comparison results are shown in Table 2.

[0237] Table 2 Comparison of micro-expression recognition performance of different methods on three-category datasets

[0238]

[0239] According to Table 2, it can be seen that in all training and testing data sets, compared with other methods, the UF1 and UAR of the recognition method of this embodiment are significantly improved. The recognition results of the method of this embodiment on three categories are better than those of other methods, and it is more effective for three-category micro-expression recognition.

[0240] In addition, the method of this embodiment is also applied to multi-classification micro-expression recognition. The CASME II and SAMM data sets are used to classify and recognize five types of micro-expressions, namely “disgust”, “happiness”, “other”, “depression” and “surprise”. The recognition is compared with other methods. The recognition comparison results are shown in Table 3.

[0241] Table 3 Comparison of recognition rates of different methods for multi-classification micro-expression recognition

[0242]

[0243] For multi-category micro-expression recognition, the UF1 and UAR results of the recognition method in this embodiment are also significantly higher than those of other methods. The recognition results of the method in this embodiment are better than those of other methods in multiple categories, and it also has a significant effect on multi-category micro-expression recognition.

[0244] This embodiment discloses a micro-expression recognition method based on a three-branch network. One deep convolutional neural network and two shallow 3D convolutional neural networks in parallel in the model are used to extract facial features, optical flow features, and optical strain features, respectively. The full integration of these different types of features can effectively reduce the loss of subtle facial features and improve the accuracy of micro-expression recognition. A facial feature enhancement attention module is proposed in the shallow 3D CNN, and a deep enhancement channel attention module is introduced in the deep convolutional neural network. This multi-attention mechanism enables the model to capture key facial features more specifically and reduce the interference of redundant information. The new loss function that combines the IM Loss constraint term and the Focal Loss constraint term significantly improves the model's recognition ability and recognition accuracy for difficult samples.

[0245] In this embodiment, a subject is selected from the CASME II dataset to perform an ablation experiment.

[0246] (1) Network architecture ablation

[0247] The first network branch used to extract facial features in the model is denoted as N A , the second network branch used to extract optical flow features is recorded as N B , the third network branch for extracting photostrain features is denoted as N C , the complete network model is recorded as N D , the deep enhanced channel attention module ECANet in the first network branch is recorded as M E , the facial feature enhanced attention module FEAM in the second and third network branches is denoted as M F .

[0248] Network architecture ablation experiment 1 is the complete model with the first face feature extraction branch removed, architecture experiment 2 is the complete model with the second optical flow feature extraction branch removed, architecture experiment 3 is the complete model with the third optical strain feature extraction branch removed, architecture experiment 4 is the complete model with the ECANet module removed, architecture experiment 5 is the complete model with the second branch without the FEAM module removed, architecture experiment 6 is the complete model with the third branch without the FEAM module removed, and architecture experiment 7 is the complete model.

[0249] After the network architecture ablation experiment, the comparison results are shown in Table 4.

[0250] Table 4 Network architecture ablation experiment results

[0251]

[0252] It can be seen that removing any feature extraction branch or attention module will lead to a decrease in algorithm performance, which shows that the multi-feature fusion mechanism and multi-attention mechanism in this embodiment effectively improve the micro-expression recognition accuracy of the model.

[0253] (2) Loss function ablation

[0254] Five ablation experiments were designed. Since the Focal Loss function in the method of this embodiment, when its parameter γ is set to 0, degenerates into the cross entropy loss function commonly used in the existing micro-expression recognition method, the ablation experiment 1 adopts the most basic cross entropy loss function, denoted as L CE , and use this as the baseline. Ablation experiment 2 uses the cross entropy loss function L CE and IMLoss function L IM , ablation experiment 3 only uses the Focal Loss function L focal , ablation experiment 4 only uses IM Loss function L IM , ablation experiment 5 uses the Focal Loss function L focal and IM Loss function L IM , that is, the complete loss function L in this paper Total .

[0255] After the loss function ablation experiment, the comparison results are shown in Table 5.

[0256] Table 5 Loss function ablation experiment results

[0257]

[0258] It can be seen that the total loss function composed of the Focal Loss function and the IM Loss function in this embodiment has the best effect.

[0259] Embodiment 2

[0260] This embodiment discloses a micro-expression recognition system based on a three-branch network, comprising:

[0261] A preprocessing module, used for acquiring a video sequence, preprocessing the video sequence to obtain an optical flow map, and further processing the optical flow map to obtain an optical strain map, wherein the video sequence includes video frames;

[0262] The feature extraction and fusion module is used to input the video frame into the first branch network, the first branch network extracts facial features based on deep learning and deep enhanced channel attention; input the optical flow map into the second branch network, the second branch network uses the second shallow convolutional neural network and the second facial feature enhanced attention to extract the features of the optical flow map; input the optical strain map into the third branch network, the third branch network uses the third shallow convolutional neural network and the third facial feature enhanced attention to extract the features of the optical strain map; fuse the facial features, the features of the optical flow map, and the features of the optical strain map;

[0263] The recognition and classification module is used to classify micro-expressions based on the fused features to obtain the classification results of micro-expressions.

[0264] Based on providing a micro-expression recognition system based on a three-branch network, the method steps in embodiment 1 are implemented.

[0265] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0266] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A micro-expression recognition method based on a three-branch network, characterized in that: include: A video sequence is obtained, and the video sequence is preprocessed to obtain an optical flow map, and the optical flow map is further processed to obtain an optical strain map, wherein the video sequence includes video frames; Input the video frame into the first branch network, and the first branch network extracts facial features based on deep learning and deep enhanced channel attention; The optical flow map is input into the second branch network, and the second branch network uses a second shallow convolutional neural network and a second facial feature enhanced attention to extract features of the optical flow map; The light strain map is input into the third branch network, and the third branch network extracts features of the light strain map by using a third shallow convolutional neural network and a third facial feature enhanced attention; The facial features, the features of the optical flow map, and the features of the optical strain map are integrated; The fused features are used to classify micro-expressions to obtain the classification results of micro-expressions.

2. A micro-expression recognition method based on a three-branch network as claimed in claim 1, characterized in that: The video frame is a apex frame, which refers to the frame where the intensity or expression of the micro-expression is most obvious, or a sequence of intermediate frames consisting of the apex frame and its preceding and following frames.

3. A micro-expression recognition method based on a three-branch network as claimed in claim 1, characterized in that: Preprocessing the video sequence, including positioning the vertex frame, for determining the position of the vertex frame in the micro-expression sequence, for extracting the time information before and after the vertex frame; The vertex frame and the start frame are processed to obtain an optical flow map representing the dynamic changes of micro-expressions, and the optical flow map is processed to obtain an optical strain map.

4. A micro-expression recognition method based on a three-branch network as claimed in claim 1, characterized in that: The second branch network and the third branch network have the same structure, both of which include: three parallel streams, a channel layer, a facial feature enhanced attention module, a second fusion module, and a fully connected layer; In the second branch network, each parallel stream uses a convolution kernel with different parameters to extract features of different sizes. After passing through the parallel streams, the features in the three parallel streams are spliced ​​together at the channel level to obtain the first fusion feature; The first facial feature enhanced attention module combines and weighted averages the first fused features; The feature obtained by the first facial feature enhanced attention module is further fused with the first fused feature to obtain a second fused feature; The second fusion feature is processed by the fully connected layer and the features of the optical flow map are output.

5. A micro-expression recognition method based on a three-branch network as claimed in claim 1, characterized in that: The facial feature enhancement attention module includes a convolutional attention module, a double pooling layer, and a fully connected layer. The convolutional attention module includes a channel attention module and a spatial channel attention module. Perform maximum pooling on the first fusion feature to obtain the maximum pooling layer output feature; Perform average pooling on the first fusion feature to obtain the average pooling layer output feature; The output features of the maximum pooling layer and the average pooling layer are combined and input into the fully connected layer to obtain the output features of the fully connected layer; Input the output features of the fully connected layer into the channel attention module to obtain the channel attention features; Perform channel attention weighting on the channel attention feature and the fully connected layer output feature to obtain the channel attention weighted feature; Input the channel attention weighted features into the spatial attention module to obtain the spatial attention features; Perform spatial attention weighting on the spatial attention feature and the channel attention weighted feature to obtain the spatial attention weighted feature; The spatial attention feature is fused with the first fusion feature to obtain the second fusion feature.

6. A micro-expression recognition method based on a three-branch network as claimed in claim 1, characterized in that: The second fusion feature is processed by the fully connected layer. The fully connected layer consists of three layers, including: The first two fully connected layers map the fused second fusion features to higher-level abstract representations and gradually reduce the feature dimensions; The third fully connected layer transforms the features output by the first two fully connected layers into a dimension suitable for the output of the second branch network.

7. A micro-expression recognition method based on a three-branch network as claimed in claim 1, characterized in that: The optical flow map is input into the second branch network and the optical strain map is input into the third branch network before being resampled.

8. A micro-expression recognition method based on a three-branch network as claimed in claim 1, characterized in that: The first branch network includes a deep convolutional neural network and a channel attention module; the channel attention module includes a global average pooling module and a maximum pooling module; After the video frames are input into the deep convolutional neural network, the original features of the face are extracted; The original features of the face are input into the adaptive average pooling, average pooling is performed on each channel, and the average pooling channel attention weight is obtained through convolution and function activation processing; Multiply the average pooling channel attention weight with the original face feature to obtain the global average pooling output feature; The original features of the face are input into the adaptive maximum pooling for maximum pooling, and the maximum pooling channel attention weight is obtained through convolution and function activation processing; Multiply the maximum pooling channel attention weight with the original face feature to obtain the maximum pooling output feature; The global average pooling output features are fused with the maximum pooling output features to obtain the facial features.

9. A micro-expression recognition method based on a three-branch network as claimed in claim 1, characterized in that: After obtaining the micro-expression classification results, the total loss function is constructed through the two loss functions of IM Loss and Focal Loss to optimize the model.

10. A micro-expression recognition system based on a three-branch network, characterized in that: include: A micro-expression recognition model construction system based on a three-branch network, comprising: A preprocessing module, used for acquiring a video sequence, preprocessing the video sequence to obtain an optical flow map, and further processing the optical flow map to obtain an optical strain map, wherein the video sequence includes video frames; The feature extraction and fusion module is used to input the video frame into the first branch network, the first branch network extracts facial features based on deep learning and deep enhanced channel attention; input the optical flow map into the second branch network, the second branch network uses the second shallow convolutional neural network and the second facial feature enhanced attention to extract the features of the optical flow map; input the optical strain map into the third branch network, the third branch network uses the third shallow convolutional neural network and the third facial feature enhanced attention to extract the features of the optical strain map; fuse the facial features, the features of the optical flow map, and the features of the optical strain map; The recognition and classification module is used to classify micro-expressions based on the fused features to obtain the classification results of micro-expressions.

Images