A micro-expression recognition method, system, device and medium based on a multi-scale mixed channel

Abstract

The application belongs to the technical field of expression recognition, and discloses a micro-expression recognition method, system, device and medium based on a multi-scale mixed channel, which comprises the following steps: acquiring an image sequence corresponding to a micro-expression video; extracting an effective face region in the image sequence to obtain preprocessed micro-expression image data; inputting the preprocessed micro-expression image data into a micro-expression recognition model for classification prediction to obtain a recognition result; wherein the micro-expression recognition model comprises a multi-scale mixed channel module, a feature purification module and a full connection layer connected in sequence. The technical scheme disclosed by the application can effectively capture subtle changes of micro-expression and significantly improve recognition accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of expression recognition, and particularly relates to a micro-expression recognition method, system, device and medium based on a multi-scale mixed channel. BACKGROUND

[0002] In the field of micro-expression recognition, existing methods are mainly classified into two types according to input data, i.e., a method taking manually labeled key frames as input and a method taking video sequences as input. The method taking video sequences as input can capture spatio-temporal features, but is weak in modeling the dependency relationship of long sequences, is easily affected by sequence length and frame rate changes, and is prone to cause too much redundant information in input. The method taking key frames as input usually relies on motion features of start frames and peak value frames for recognition, which retains spatial features and temporal features while ignoring facial identity information.

[0003] In addition, the features of micro-expression are usually very subtle and are easily mixed with noise at a small scale, which further increases the difficulty of recognition. Existing methods have problems in processing multi-scale feature fusion, which limits the recognition accuracy. Therefore, it has become a core problem to be solved to research a micro-expression recognition method capable of effectively extracting multi-scale features and fusing overall information. SUMMARY

[0004] The application aims to provide a micro-expression recognition method, system, device and medium based on a multi-scale mixed channel to solve the problems in the prior art.

[0005] To achieve the above-mentioned purpose, the application provides a micro-expression recognition method based on a multi-scale mixed channel, comprising:

[0006] obtaining an image sequence corresponding to a micro-expression video;

[0007] extracting an effective face region in the image sequence to obtain preprocessed micro-expression image data;

[0008] inputting the preprocessed micro-expression image data into a micro-expression recognition model for classification prediction to obtain a recognition result; wherein the micro-expression recognition model comprises a multi-scale mixed channel module, a feature purification module and a full connection layer connected in sequence.

[0009] Optionally, the extraction of the effective face region in the image sequence specifically comprises:

[0010] performing face detection on the image sequence based on an MTCNN network, extracting the effective face region, and taking a first frame image in the image sequence as a start frame, calculating the optical flow change between the start frame and all subsequent frames according to a TVL1 method, and extracting motion features.

[0011] Optionally, the training process of the micro-expression recognition model specifically comprises:

[0012] acquiring training data, the training data comprising micro-expression image training data and corresponding expression recognition labels;

[0013] inputting the training data into the micro-expression recognition model for classification prediction, and training according to a target loss function to obtain a trained micro-expression recognition model.

[0014] Optionally, the processing process of the micro-expression recognition model specifically comprises:

[0015] sequentially inputting the preprocessed micro-expression image sequence data into the multi-scale mixed channel module to extract micro-expression motion features from global to local;

[0016] integrating the micro-expression motion features into a channel dimension through the feature purification module, and performing feature extraction and feature purification processing to obtain purified micro-expression motion features;

[0017] inputting the purified micro-expression motion features into the full connection layer for classification prediction, and outputting the recognition result.

[0018] Optionally, the processing process of the multi-scale mixed channel module specifically comprises:

[0019] gradually blocking the preprocessed micro-expression image data, and extracting micro-expression features along the channel dimension after each blocking, while gradually reducing the size of the input tensor and increasing the channel dimension to obtain blocking features;

[0020] processing the blocking features based on a rearrangement operation and a position-aware ring convolution module, and retaining relatively global facial motion information through residual connection to complete the processing of the multi-scale mixed channel module, and outputting micro-expression motion features.

[0021] A micro-expression recognition system based on a multi-scale mixed channel, comprising:

[0022] a data acquisition module configured to acquire image sequences corresponding to micro-expression videos, extract valid face regions in the image sequences, and obtain preprocessed micro-expression image data;

[0023] a micro-expression recognition module configured to input the preprocessed micro-expression image data into a micro-expression recognition model for classification prediction to obtain a recognition result, wherein the micro-expression recognition model comprises a multi-scale mixed channel module, a feature purification module, and a full connection layer connected in sequence.

[0024] An electronic device comprises a memory for storing a computer program and a processor for running the computer program to make the electronic device execute the micro-expression recognition method based on a multi-scale mixed channel.

[0025] A computer readable storage medium stores a computer program, which, when executed by a processor, implements the micro-expression recognition method based on a multi-scale mixed channel.

[0026] The technical effects of the present application are:

[0027] The present application gradually extracts different scale micro-expression features from global to local through a multi-scale mixed channel module (MC), effectively capturing the subtle changes of micro-expression; through a feature purification module (FPM), further focusing on micro-expression features in the channel dimension, reducing the degree of attention of the model to noise, and enhancing the recognition ability of the model; through the MLP layer, the features are classified, and the overall precision of the model is significantly improved; on the mixed data set composed of public data sets (CASMEII, SMIC, and SAMM), the UAR value is 0.8832 and the UF1 value is 0.8968, on the CASME II data set, the present application realizes the UF1 value of 96.89% and the UAR value of 96.88%, which is significantly better than the existing method. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0029] The drawings forming a part of the present application are used to provide a further understanding of the present application, the illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0030] Fig. 1 The flowchart in the embodiment of the present application.

[0031] Fig. 2 The structure schematic diagram and the overall network structure diagram of the multi-scale mixed channel module (MC) in the embodiment of the present application;

[0032] Fig. 3 The structure schematic diagram of the feature purification module (FPM) in the embodiment of the present application. DETAILED DESCRIPTION

[0033] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0034] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0035] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0036] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0037] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0038] like Figs. 1-3 As shown, this embodiment provides a micro-expression recognition method based on multi-scale hybrid channels, including: acquiring an image sequence corresponding to a micro-expression video; extracting an effective face region from the image sequence to obtain preprocessed micro-expression image data; inputting the preprocessed micro-expression image data into a micro-expression recognition model for classification and prediction to obtain a recognition result; wherein, the micro-expression recognition model includes a multi-scale hybrid channel module, a feature purification module, and a fully connected layer connected in sequence.

[0039] The embodiment aims to provide a micro-expression recognition method based on local-global time relationship, comprising the following steps: The embodiment aims to provide a micro-expression recognition method based on a multi-scale mixed channel, comprising the following steps: obtaining a micro-expression video and decomposing it into an image sequence; preprocessing the micro-expression image sequence, performing face detection by using MTCNN, and performing affine transformation according to face feature points to realize face alignment, and extracting an effective face region; constructing a multi-scale mixed channel module (MC): extracting micro-expression motion features from the global to the local angle, gradually blocking the input image, and extracting micro-expression features along the channel dimension after each blocking, and simultaneously ensuring that the model continuously pays attention to information from large-scale features through residual connection; constructing a feature purification module (FPM): combining the remaining spatial features into the channel dimension through the combination of convolution and pooling layers, and further extracting and purifying micro-expression features by using 11-core convolution operation and transformer encoder; inputting the output of MC and FPM into the MLP layer for classification, and the MLP layer comprises a LayerNorm (LN) layer, a Reduce layer and a Linear layer; training the multi-scale mixed channel network, saving the best weight model; preprocessing the micro-expression video to be recognized, inputting the trained multi-scale mixed channel network, and recognizing the micro-expression category.

[0040] The specific implementation process of the embodiment comprises:

[0041] Step 1: obtaining a batch of micro-expression videos and decomposing them into an image sequence;

[0042] Step 2: preprocessing the micro-expression image sequence, performing face detection by using MTCNN, and performing affine transformation according to face feature points to realize face alignment, and extracting an effective face region; and taking the first frame image as the starting frame and all subsequent frames as the peak frame, and calculating the optical flow change by using the TVL1 method.

[0043] Step 3: constructing a multi-scale mixed channel module (MC): extracting micro-expression motion features from the global to the local angle, gradually blocking the input image, and extracting micro-expression features along the channel dimension after each blocking, and simultaneously ensuring that the model continuously pays attention to information from large-scale features through residual connection, and the feature processing process can be represented as:

[0044]

[0045] 1≤x1≤2,1≤y1≤2,k∈[1,4]

[0046] wherein In the formula, x1∈[1,2]x1∈[1,2], y1∈[1,2], k1∈[1,4] represents the feature block order of the I1 feature from the top left corner to the bottom right corner. Here, k1∈[1,4] is the reordered digital label, P(.) represents the image processed by the ParC module, and R0 represents the previous rearrangement layer.

[0047] Step 4: Constructing the Feature Purification Module (FPM): Integrate the remaining spatial features into the channel dimension through the combination of convolution and pooling layers, and use an 11-core convolution operation and a transformer encoder to further extract and purify micro-expression features;

[0048] Step 5: Send the output of MC and FPM into the MLP layer for classification, which includes LayerNorm (LN) layer, Reduce layer and Linear layer;

[0049] Step 6: Train the multi-scale hybrid channel network and save the best weight model;

[0050] Step 7: Preprocess the micro-expression video to be recognized, input into the trained multi-scale hybrid channel network, and identify the micro-expression category.

[0051] Implementable, the preprocessing in step 2 includes:

[0052] Step 2.1: Face detection and five feature point detection are performed on each frame of image;

[0053] Step 2.2: Affine transformation is used to align the detected facial features;

[0054] Step 2.3: The effective face region with consistent size is cropped based on the inner corner of the left eye of each frame of face.

[0055] Step 2.4: The TVL1 method is used to calculate the motion feature changes of the first image in the sequence and all subsequent images.

[0056] Implementable, step 3 specifically includes:

[0057] Step 3.1: The input image is divided into blocks step by step, and the micro-expression feature is extracted along the channel dimension after each division, while the size of the input tensor is gradually reduced and the channel dimension is gradually increased;

[0058] Step 3.2: The divided features are processed by rearrangement operation and position-aware ring convolution (ParC) module to ensure that the attention of feature scale does not decrease too quickly;

[0059] Step 3.3: The relative global facial motion information is preserved through residual connection, while deeper features in the channel dimension are learned.

[0060] Implementable, step 4 specifically includes:

[0061] Step 4.1: Integrate the remaining spatial features into the channel dimension through the combination of convolution and pooling layers, and select the useful part of the feature map;

[0062] Step 4.2: Use 1x1 kernel convolution operation and transformer encoder to further extract and purify micro-expression features and enhance global feature extraction capability;

[0063] Step 4.3: Convert the output of FPM into text state, make the height (H) and width (W) 1, and further focus on micro-expression feature information through the transformer encoder.

[0064] Implementable, step 5 specifically includes:

[0065] Step 5.1: Input the output features of MC and FPM into the MLP layer, which includes the LayerNorm (LN) layer, the Reduce layer and the Linear layer;

[0066] Step 5.2: Normalize the features through the LayerNorm layer, reduce the feature dimension through the Reduce layer, and complete the final classification task through the Linear layer;

[0067] Step 5.3: Use the Softmax function to complete the final classification of micro-expression categories.

[0068] The micro-expression recognition method based on the multi-scale mixed channel provided in the embodiment has the following advantages in view of the deficiencies of the existing methods in multi-scale feature extraction and global-local feature fusion: the micro-expression features are gradually extracted from global to local through the multi-scale mixed channel module (MC), so as to effectively capture the subtle changes of the micro-expression; the features are further optimized and fused through the feature purification module (FPM), so as to enhance the recognition ability of the model; the features are classified through the MLP layer, so as to significantly improve the recognition accuracy; on the mixed dataset composed of the public datasets (CASMEII, SMIC, and SAMM), the UAR value is 0.8832 and the UF1 value is 0.8968, on the CASME II dataset, the UF1 value of 96.89% and the UAR value of 96.88% are achieved, which are significantly better than the existing methods. On the five-class mixed dataset composed of SAMM and CASME II, the overall UAR and UF1 of MMCN reach 0.8242 and 0.8476 respectively. Among them, on the CASMEII, the UAR value of 0.9496 and the UF1 value of 0.9346 are achieved. On the three-class task of the CASME 3 dataset, MMCN achieves the UAR value of 0.5679 and the UF1 value of 0.6125, which is better than other models in the field, proving the generalization and effectiveness of the model.

[0069] The micro-expression recognition system based on the multi-scale mixed channel provided in the embodiment can be implemented, and includes:

[0070] The data acquisition module is configured to acquire an image sequence corresponding to a micro-expression video, extract an effective face region in the image sequence, and obtain preprocessed micro-expression image data.

[0071] The micro-expression recognition module is configured to input the preprocessed micro-expression image data into a micro-expression recognition model for classification and prediction, and obtain a recognition result. The micro-expression recognition model includes a multi-scale mixed channel module, a feature purification module, and a fully connected layer connected in sequence.

[0072] The electronic device provided in the embodiment can be implemented, and includes a memory and a processor. The memory is configured to store a computer program, and the processor is configured to run the computer program to enable the electronic device to perform the micro-expression recognition method based on the multi-scale mixed channel.

[0073] The computer readable storage medium provided in the embodiment can be implemented, and stores a computer program. The computer program is executed by a processor to implement the micro-expression recognition method based on the multi-scale mixed channel.

[0074] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A micro-expression recognition method based on multi-scale hybrid channel, characterized in that, The method comprises the following steps: obtaining an image sequence corresponding to a micro-expression video; extracting an effective face region in the image sequence to obtain preprocessed micro-expression image data; inputting the preprocessed micro-expression image data into a micro-expression recognition model for classification prediction to obtain a recognition result; wherein the micro-expression recognition model comprises a multi-scale mixed channel module, a feature purification module and a full connection layer connected in sequence; the processing process of the micro-expression recognition model specifically comprises: inputting the preprocessed micro-expression image sequence data into the multi-scale mixed channel module in sequence to extract micro-expression motion features from global to local; integrating the micro-expression motion features into a channel dimension through the feature purification module, and performing feature extraction and feature purification processing to obtain purified micro-expression motion features; inputting the purified micro-expression motion features into the full connection layer for classification prediction to output a recognition result; the processing process of the multi-scale mixed channel module specifically comprises: gradually blocking the preprocessed micro-expression image data, and extracting micro-expression features along the channel dimension after each blocking, while gradually reducing the size of the input tensor and increasing the channel dimension to obtain blocking features; processing the blocking features based on a rearrangement operation and a position-aware ring convolution module, and retaining relatively global facial motion information through residual connection to complete the processing of the multi-scale mixed channel module and output micro-expression motion features.

2. The micro-expression recognition method based on multi-scale mixed channels according to claim 1, characterized in that, The effective face region in the image sequence is extracted specifically by: performing face detection on the image sequence based on an MTCNN network, extracting an effective face region, and taking the first frame image in the image sequence as the starting frame to calculate the optical flow change between the starting frame and all subsequent frames according to a TVL1 method to extract motion features.

3. The micro-expression recognition method based on multi-scale hybrid channel according to claim 1, characterized in that, The training process of the micro-expression recognition model specifically comprises: obtaining training data, wherein the training data comprises micro-expression image training data and corresponding expression recognition labels; inputting the training data into the micro-expression recognition model for classification prediction, and training according to a target loss function to obtain a trained micro-expression recognition model.

4. A multi-scale hybrid channel based micro-expression recognition system, applying the method of any one of claims 1-3, characterized in that, The method comprises the following steps: a data acquisition module for obtaining an image sequence corresponding to a micro-expression video; extracting an effective face region in the image sequence to obtain preprocessed micro-expression image data; a micro-expression recognition module for inputting the preprocessed micro-expression image data into a micro-expression recognition model for classification prediction to obtain a recognition result; wherein the micro-expression recognition model comprises a multi-scale mixed channel module, a feature purification module and a full connection layer connected in sequence.

5. An electronic device, comprising: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the micro-expression recognition method based on a multi-scale mixed channel according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer program stored in the memory is executed by the processor to implement the micro-expression recognition method based on a multi-scale mixed channel according to any one of claims 1-3.

Images