A micro-expression recognition method and system

Abstract

The application discloses a micro-expression recognition method and system, a face image set is obtained by preprocessing a micro-expression video sequence; a network model is built, the network model adopts ResNet as a backbone network, two branches are added after each of ResNet, one branch is a SE module, and the second branch is a Triplet Attention module, feature maps output by the two branches are multiplied and fused; a feature refinement module is added after each of ResNet, before final classification of extracted features, according to the number of classified categories; a training set is used to train the network model, a classification task is performed, and a classification model is generated; and the trained network model is tested by using a test set. The embedded enhanced scSE module is used to inhibit irrelevant information, emphasize extraction of effective feature information, and help the network to extract more effective feature information.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and artificial intelligence, and particularly relates to a micro-expression recognition method and system. BACKGROUND

[0002] Micro-expression is a rapid and unconscious facial movement produced by a person when experiencing strong emotions. It is produced when a person tries to hide their inner emotions and cannot be faked or suppressed, and can be used as an important basis for judging a person's subjective emotions.

[0003] Micro-expression usually has the characteristics of short duration and low motion intensity, generally lasting 1 / 25s to 1 / 3s, and the motion amplitude is very small, and it does not appear on the upper half of the face and the lower half of the face at the same time.

[0004] Micro-expression has great research value and important applications in the fields of clinical diagnosis, emotional intelligence, and judicial interrogation. More and more researchers are entering this field to study micro-expression.

[0005] Traditional manual feature extraction mainly relies on manually designed feature extraction rules, and the features obtained often cannot explain the physical meaning of each dimension, requiring professional knowledge and a complex parameter adjustment process, and the generalization performance and robustness are poor. The method based on deep learning has the problems of large parameter quantity and insufficient feature extraction. SUMMARY

[0006] The present application provides a micro-expression recognition method, which can solve the problems of insufficient micro-expression samples and insufficient feature extraction in the prior art, and also provides a micro-expression recognition system.

[0007] Technical scheme: on one hand, the present application provides a micro-expression recognition method, comprising:

[0008] Step (1) pre-processes the micro-expression video sequence to obtain a face image set, and divides the face image set into a training set and a test set;

[0009] Step (2) Build the network model DA_DenseNet. The network model DA_DenseNet uses DenseNet-121 as the backbone network and adds two branches after each Dense Block of DenseNet-121. One branch is the en-scSE module and the second branch is the Triplet Attention module. The feature maps output by the two branches are multiplied and fused. Triplet Attention is added after each Transition Layer. Before the extracted features are finally classified, a feature refinement module is added according to the number of classes to be classified. The en-scSE module is a parallel module of an improved channel-level attention module and an improved channel compression-spatial activation module. The Triplet Attention module is an interaction processing module of different dimensions.

[0010] The cSE module will display the face image. Using global average pooling from Become Then, after two more convolution operations, a final result is obtained. The vectors are normalized using the sigmoid function, and finally multiplied channel-wise to obtain the output features; the en-scSE module adds two linear layers before the channel-wise multiplication and uses the ReLU activation function;

[0011] The sSE module compresses face images along the channels. And spatially stimulated, facial images utilize The convolution, from Become Use the sigmoid function The activation is then multiplied with the original face image to obtain features; the en-scSE module adds a global average pooling operation, which is multiplied with the features obtained from the original sSE, suppressing unimportant information in the space and better extracting important features;

[0012] Step (3) Train the network model DA_DenseNet using the training set, perform the classification task, and generate a classification model;

[0013] Step (4) Test the trained network model DA_DenseNet using the test set.

[0014] Furthermore, including:

[0015] In step (1), the micro-expression video sequence is preprocessed, including:

[0016] (11) positioning the starting frame and the vertex frame of each micro-expression video sequence, and segmenting the face region from the positioned starting frame and vertex frame; and adding two frames before and after the micro-expression vertex frame as training or test samples;

[0017] (12) using random cropping to directly crop the micro-expression video sequence into fixed-size micro-expression samples, randomly rotating the micro-expression samples, and performing color jittering operation on the micro-expression samples for data enhancement.

[0018] Further, comprising:

[0019] In the step (2), the structure of the Triplet Attention is:

[0020] The Triplet Attention is composed of 3 parallel branches, the first branch establishes interaction between the H dimension and the C dimension, the second branch establishes interaction between the C dimension and the W dimension, and the third branch establishes interaction between the H dimension and the W dimension, and the Z-pool layer is responsible for reducing the Tensor of the C dimension to 2 dimensions, and connecting the average pooling features and the maximum pooling features on the dimension, and the formula is:

[0021]

[0022] Where Maxpool represents the maximum pooling, Avgpool represents the average pooling, and 0d is the 0th dimension on which the maximum and average pooling operations occur.

[0023] The outputs of the final three branches are aggregated using the average, and the formula is:

[0024] ;

[0025] Where, represents the tensor after rotation of the first branch, represents the tensor after rotation of the second branch, represents the tensor of the third branch, represents the tensor after Z-pool, represents the tensor after Z-pool, represents the tensor after Z-pool, represents the sigmoid activation function, represents a standard two-dimensional convolution layer defined by the convolution kernel size k in the three branches.

[0026] Further, comprising:

[0027] In step (2), the structure of the feature refinement module is as follows:

[0028] Each feature refinement branch consists of a global average pooling layer and two fully connected layers. The fully connected layers are used for classification, and the fully connected layers are followed by a sigmoid activation layer. If the micro-expression samples of the training samples belong to the k-th class, the truth value in the k-th branch is 1, and the truth value in other branches is 0, allowing the network to generate expression-specific features for each expression class.

[0029] On the other hand, the present invention also provides a micro-expression recognition system, comprising:

[0030] The preprocessing module is used to preprocess the micro-expression video sequence to obtain a set of face images, and divide the set of face images into a training set and a test set;

[0031] The model building module is used to build the network model DA_DenseNet. The network model DA_DenseNet uses DenseNet-121 as the backbone network, and adds two branches after each Dense Block of DenseNet-121. One branch is an en-scSE module, and the second branch is a Triplet Attention module. The feature maps output by the two branches are multiplied and fused. Triplet Attention is added after each Transition Layer. Before the extracted features are finally classified, a feature refinement module is added according to the number of classes to be classified. The en-scSE module is a parallel module of an improved channel-level attention module and an improved channel compression-spatial activation module. The Triplet Attention module is an interaction processing module of different dimensions.

[0032] The cSE module will display the face image. Using global average pooling from Become Use it twice more. The convolution operation ultimately yields a The vectors are normalized using the sigmoid function, and finally multiplied channel-wise to obtain the output features; the en-scSE module adds two linear layers before the channel-wise multiplication and uses the ReLU activation function;

[0033] The sSE module compresses face images along the channels. And spatially stimulated, facial images utilize The convolution, from Become Use the sigmoid function Activation, then multiplied with the original human face image to obtain the feature; the en-scSE module adds a global average pooling operation, and the feature obtained by the original sSE is multiplied, the unimportant information in the space will be suppressed, and the important features are better extracted;

[0034] The training module is configured to train the network model DA_DenseNet by using a training set, perform a classification task, and generate a classification model.

[0035] The testing module is configured to test the trained network model DA_DenseNet by using a test set.

[0036] On the basis of the above, the application further provides a computer storage medium having a computer program stored thereon, the computer program being executed by a computer processor to implement the method described above.

[0037] Advantages: (1) the application adopts DenseNet as the backbone network, reduces the gradient disappearance problem, deep layer gradient can be directly transmitted back to the shallow layer, strengthens the propagation of features, encourages the reuse of features, and the features output by each layer are used by the subsequent layers; (2) the enhanced scSE (en-scSE) module is embedded after the Dense Block to suppress irrelevant information and emphasize the extraction of effective feature information, helping the network to extract more effective feature information; (3) the TripletAttention module is added after the TL layer, which establishes the dependency relationship between dimensions through rotation operation and residual transformation without increasing the parameters; (4) the feature refinement module is used to extract specific expression features of different types of expressions, improving the classification accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The flowchart of the recognition method described in the embodiments of the application;

[0039] Figure 2 The structure diagram of the DA_DenseNet described in the embodiments of the application;

[0040] Figure 3 The structure diagram of the en-scSE described in the embodiments of the application;

[0041] Figure 4 The structure diagram of the Triplet Attention described in the embodiments of the application;

[0042] Figure 5 The structure diagram of the feature refinement described in the embodiments of the application. DETAILED DESCRIPTION

[0043] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0044] First, the present application discloses a micro-expression recognition method, such as Figure 1 is a flowchart of the method of the present application, comprising the following steps:

[0045] Step 1: Preprocess the micro-expression video sequence, obtain the starting frame and the vertex frame, and add two frames before and after the vertex frame as training test samples, and use cropping, random rotation, random horizontal flipping, changing image attributes, etc. to perform data enhancement to solve the problem of too small micro-expression dataset; Specifically:

[0046] Among them, the specific content and method of data preprocessing in step 1 are:

[0047] (1) Locate the starting frame and the vertex frame of each micro-expression sequence, and segment the face region from the located starting frame and vertex frame;

[0048] (2) In order to increase the amount of available samples, add two frames before and after the micro-expression vertex frame as training / test samples;

[0049] (3) Randomly crop, directly crop a fixed size of micro-expression sample into the network, in order to increase the robustness of the model, randomly rotate the micro-expression sample, the rotation angle range is (-8°, 8°), and color jitter operation is performed on the micro-expression sample to perform data enhancement.

[0050] Step 2: Build a DA_DenseNet network model;

[0051] As shown in Figure 2 , the DA_DenseNet structure in step 2 is:

[0052] In order to alleviate the problem of gradient disappearance, the model selects DenseNet-121 as the backbone network, which strengthens the propagation of features, and shallow features can be directly used by deep layers; DenseNet-121 has 4 DBs, 3 TL layers, and TripletAttention is inserted between DB and TL.

[0053] Two branches are added after each DB (Dense Block): one is en-scSE (enhanced Spatial and Channel Squeeze & Excitation Block), and the second is a Triplet Attention module, which multiplies and fuses the feature maps output by the two branches. Triplet Attention is added after each TL (Transition Layer) to enhance the extraction of effective feature information and suppress ineffective feature information. Before the extracted features are finally classified, a feature refinement module is added according to the number of classes to extract features of different types of micro-expressions. The training scheme of this network model uses a marginal loss function.

[0054] like Figure 3 As shown, the en-scSE structure added after DB in the DA_DenseNet model is as follows:

[0055] scSE is a parallel combination of the cSE (Spatial Squeeze and Channel Excitation Block) module and the sSE (Channel Squeeze and Spatial Excitation Block) module.

[0056] The cSE module is equivalent to channel-level attention; its specific operation is to process feature maps... Using global average pooling from Become Use it twice more. The convolution operation ultimately yields a... The vector is then normalized using the sigmoid function, and finally multiplied with the original features channel-wise to obtain the output features. This output is the final output of the cSE module, used for fusion with sSE. In other words, this invention applies it to the micro-expression classification task. To improve the model's robustness, cSE is improved by adding two linear layers before the channel-wise layer and using the ReLU activation function.

[0057] The formula is as follows:

[0058]

[0059] Among them, input features View as a channel The combination Represents the first The importance of each channel is adaptively adjusted in the network learning process, so as to ignore unimportant channels and focus on relatively important channels.

[0060] sSE is a channel compression and spatial excitation module, which compresses the channel feature map along the channel direction and excites it in space. The feature map is converted from to using a sigmoid function activation, and then multiplied with the original feature map. The formula is as follows:

[0061]

[0062] where the input tensor is considered as , corresponding to the spatial position , , ; the spatial compression operation is to generate a projection tensor by convolution and weight matrix, and then pass it through a sigmoid activation layer , corresponding to the relative importance of the spatial position of the given feature map. In order to be more suitable for micro-expression recognition tasks, the present application adds a global average pooling operation to the sSE, and multiplies the features obtained by the original sSE, so that the unimportant information in space will be suppressed, and the important features will be better extracted.

[0063] As shown in Figure 4 , the structure of Triplet Attention added after TL in the DA_DenseNet model is as follows:

[0064] Triplet Attention consists of three parallel branches. The first branch establishes interaction between H dimension and C dimension, the second branch establishes interaction between C dimension and W dimension, and the third branch establishes interaction between H dimension and W dimension. The Z-pool layer is responsible for reducing the Tensor in C dimension to 2 dimensions, and connecting the average pooling features and the maximum pooling features in this dimension. The formula is as follows:

[0065]

[0066] where Maxpool represents maximum pooling, Avgpool represents average pooling, and 0d is the 0th dimension where maximum and average pooling operations occur.

[0067] The outputs of the final three branches are aggregated using average, and the formula is as follows:

[0068] ; ​

[0069] wherein, represents the tensor after rotation of the first branch, represents the tensor after rotation of the second branch, represents the tensor of the third branch, represents the tensor after Z-pool, represents the tensor after Z-pool, represents the tensor after Z-pool, represents a sigmoid activation function, represents a standard two-dimensional convolutional layer defined by a convolution kernel size k in the three branches.

[0070] As shown in Figure 5 , the structure of the feature refinement module in the DA_DenseNet model is as follows:

[0071] Each feature refinement branch is composed of a global average pooling and two fully connected layers, wherein the fully connected layers serve as classification, and the sigmoid layer is an activation layer; if the micro-expression sample of the training sample is the kth category, the true value in the kth branch is 1, and the true values in the other branches are 0, allowing the network to generate expression-specific features for each category of expression.

[0072] Step three: training the DA_DenseNet, performing a classification task and generating a classification model; step four: testing using the trained DA_DenseNet model using the LOSO validation method.

[0073] The specific embodiments of the present application are described below through classification instances on the composite data set selected from CASME II, SAMM and SMIC:

[0074] (1) Image preprocessing: locating the starting frame and vertex frame of each micro-expression sequence, and segmenting the face region from the located starting frame and vertex frame; in order to increase the amount of available samples, two frames before and after the micro-expression vertex frame are added as training / test samples; random cropping is adopted to directly crop fixed-size micro-expression samples into the network, and here the fixed size is 224*224; in order to increase the robustness of the model, the micro-expression samples are randomly rotated with a rotation angle range of (-8°, 8°), and color jitter operation is performed on the micro-expression samples for data enhancement.

[0075] (2) Building a DA_DenseNet network model and using a marginal loss function as a loss function to train the model.

[0076] (3) Using the trained DA_DenseNet network model to perform micro-expression recognition on the test set using the LOSO strategy.

[0077] (4) The method is compared with the existing method for a composite data set, and the results are shown in Table 1. Through comparison with other methods, it can be verified that the method proposed in the application has good superiority relative to other methods:

[0078] Table 1

[0079] Method UAR UF1 LBP-TOP 0.5787 0.5882 Bi-WOOF 0.6227 0.6296 GEME(Mutil-task) 0.7303 0.7221 DA_DenseNet 0.7601 0.7291

[0080] In another aspect, the application also provides a micro-expression recognition system, comprising:

[0081] A preprocessing module is configured to preprocess a micro-expression video sequence to obtain a face image set, and divide the face image set into a training set and a test set;

[0082] A model construction module is configured to build a network model DA_DenseNet, wherein the network model DA_DenseNet adopts DenseNet-121 as a backbone network, and adds two branches after each Dense Block of the DenseNet-121, one branch is an en-scSE module, and the second branch is a Triplet Attention module, and the feature maps output by the two branches are multiplied and fused; a Triplet Attention is added after each Transition Layer, and a feature refinement module is added according to the number of classified classes before the extracted features are finally classified; the en-scSE module is a parallel module of an improved channel-level attention module and an improved channel compression-space excitation module, and the Triplet Attention module is a different dimension interaction processing module;

[0083] The cSE module compresses the face image The global average pooling is used to compress the face image from to , and two convolution operations are used to finally obtain a dimensional vector, which is normalized by using a sigmoid function, and the output feature is obtained by channel-wise multiplication; the en-scSE module adds two linear layers before the channel-wise and uses a ReLu activation function;

[0084] The sSE module compresses the face image along the channel and excites in space, and the face image is compressed from to by using a convolution. , using sigmoid function Activation, and then multiplied with the original human face image to obtain the feature; the en-scSE module adds a global average pooling operation, and the feature obtained by the original sSE is multiplied, important information in the space is inhibited, and important features are better extracted;

[0085] The training module is configured to train the network model DA_DenseNet using a training set, perform a classification task, and generate a classification model.

[0086] The testing module is configured to test the trained network model DA_DenseNet using a test set.

[0087] Based on the above embodiment, in the embodiment of the present application, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the face living body detection method based on the double-flow convolutional neural network in any method embodiment.

[0088] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0089] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks.

[0090] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0091] These computer program instructions can also be loaded into computer or other programmable data processing devices to cause a series of operational steps to be performed on the computer or other programmable devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable devices provide steps for implementing the flow Figure 1 one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0092] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to encompass within their scope all possible variations and modifications of the preferred embodiments.

[0093] It is apparent that a person skilled in the art can make a variety of changes and modifications to the embodiments of the application without departing from the spirit and scope of the embodiments of the application. Thus, the application is also intended to encompass these changes and modifications within the scope of the claims of the application and their equivalents.

Claims

1. A micro-expression recognition method, characterized in that, include: Step (1) Preprocess the micro-expression video sequence to obtain a set of face images, and divide the set of face images into a training set and a test set; Step (2) Build the network model DA_DenseNet. The network model DA_DenseNet uses DenseNet-121 as the backbone network and adds two branches after each Dense Block of DenseNet-121. One branch is the en-scSE module and the second branch is the Triplet Attention module. The feature maps output by the two branches are multiplied and fused. Triplet Attention is added after each Transition Layer. Before the extracted features are finally classified, a feature refinement module is added according to the number of classes to be classified. The en-scSE module is a parallel module of an improved channel-level attention module and an improved channel compression-spatial activation module. The Triplet Attention module is an interaction processing module of different dimensions. The cSE module converts the face image from a global average pooling to , and finally obtains a vector of dimension, normalizes it using the sigmoid function, and finally obtains the output feature by channel-wise multiplication; the en-scSE module adds two linear layers before the channel-wise and uses the ReLu activation function; sSE module compresses the face image along the channel and spatially excites the face image using convolution, from to , using a sigmoid function activation, and then multiplied by the original face image to obtain the feature; en-sSE module adds a global average pooling operation, which is multiplied by the feature obtained by the original sSE, and the unimportant information in the space will be suppressed, and the important features will be better extracted; Step (3) Train the network model DA_DenseNet using the training set, perform the classification task, and generate a classification model; Step (4) Test the trained network model DA_DenseNet using the test set.

2. The micro-expression recognition method of claim 1, wherein, In step (1), the micro-expression video sequence is preprocessed, including: Step (11) Locate the start frame and vertex frame of each micro-expression video sequence, and segment the face region from the located start frame and vertex frame; and add two frames before and after the micro-expression vertex frame as training or test samples. Step (12) uses random cropping to directly crop micro-expression video sequences into micro-expression samples of a fixed size, randomly rotates the micro-expression samples, and performs color jittering on the micro-expression samples to enhance the data.

3. The micro-expression recognition method of claim 1, wherein, In step (2), the structure of TripletAttention is as follows: Triplet Attention consists of three parallel branches. The first branch establishes an interaction between the H and C dimensions, the second branch establishes an interaction between the C and W dimensions, and the third branch establishes an interaction between the H and W dimensions. The Z-pooling layer is responsible for reducing the C-dimensional tensor to 2-dimensional and concatenating the average pooling features and max pooling features in that dimension. The formula is as follows: ； Where Maxpool represents max pooling, Avgpool represents average pooling, and 0d is the 0th dimension where max and average pooling operations occur. The outputs of the three branches are finally aggregated using an average, as shown in the formula: ； wherein, denotes the rotated tensor of the first branch, denotes the rotated tensor of the second branch, denotes the tensor of the third branch, denotes the tensor after the Z-pool, denotes the tensor after the Z-pool, denotes the tensor after the Z-pool, denotes the sigmoid activation function, denotes a standard two-dimensional convolutional layer defined by a convolution kernel size k among the three branches.

4. The micro-expression recognition method of claim 1, wherein, In step (2), the structure of the feature refinement module is as follows: Each feature refinement branch is composed of a global average pooling and two fully connected layers, where the fully connected layers play a role in classification, and the activation layer sigmoid after the fully connected layer; if the micro-expression sample of the training sample is the kth category, the true value in the kth branch is 1, and the true value of other branches is 0, allowing the network to generate expression-specific features for each expression.

5. A micro-expression recognition system characterized in that, The method comprises the following steps: a preprocessing module is configured to preprocess the micro-expression video sequence to obtain a set of face images, and divide the set of face images into a training set and a test set; a model construction module is configured to construct a network model DA_DenseNet, wherein the network model DA_DenseNet adopts DenseNet-121 as a backbone network, and two branches are added after each Dense Block of the DenseNet-121, one branch is an en-scSE module, and the second branch is a Triplet Attention module, the feature maps output by the two branches are multiplied and fused; a Triplet Attention is added after each Transition Layer, and a feature refinement module is added according to the number of classified categories before the extracted features are finally classified; the en-scSE module is a parallel module of an improved channel-level attention module and an improved channel compression-space excitation module, and the Triplet Attention module is a different dimension interaction processing module; The cSE module will display the face image. Using global average pooling from Become Use it twice more The convolution operation ultimately yields a... The vectors are normalized using the sigmoid function, and finally multiplied channel-wise to obtain the output features; the en-scSE module adds two linear layers before the channel-wise multiplication and uses the ReLU activation function; sSE module compresses the face image along the channel and spatially excites the face image using convolution, from to , using the sigmoid function activation, and then multiplied by the original face image to obtain the feature; en-sSE module adds a global average pooling operation, which is multiplied by the feature obtained by the original sSE, and the unimportant information in the space will be suppressed, and the important features will be better extracted; a training module is configured to train the network model DA_DenseNet using the training set, perform a classification task, and generate a classification model; a test module is configured to test the trained network model DA_DenseNet using the test set.

6. A computer storage medium having stored thereon a computer program, characterized in that: The computer program, when executed by a computer processor, implements the method of any one of claims 1 to 4.

Images