Micro-expression recognition method and device based on multi-task learning and global recurrent convolution

Abstract

This invention discloses a micro-expression recognition method and apparatus based on multi-task learning and global recurrent convolution. First, the training dataset is preprocessed to obtain a fixed-length amplified frame sequence. Optical flow is calculated between adjacent frames to obtain an optical flow sequence, and the coordinates of five feature points of the facial features are labeled. Then, a convolutional neural network module I is constructed to adjust the frame size and number of channels of the amplified frame sequence. Next, a global recurrent convolution module II is constructed to extract global features. Finally, a multi-task learning module III, including optical flow estimation, facial feature point regression, and 3D convolutional neural network micro-expression classification and prediction, is constructed for prediction. This invention employs an end-to-end deep learning framework to jointly learn facial micro-expression recognition, optical flow estimation, and feature point regression. It leverages the correlation between tasks to promote micro-expression recognition, effectively identifying the motion changes of facial micro-expressions in three-dimensional spacetime, thus realizing the construction of a facial micro-expression recognition system.

Description

Technical Field

[0001] This invention relates to a micro-expression recognition method and apparatus based on multi-task learning and global recurrent convolution, belonging to the field of computer vision technology. Background Technology

[0002] Microexpressions are a special type of facial expression. Compared to ordinary expressions, microexpressions are characterized by their short duration (usually only 1 / 25 to 1 / 3 of a second), low intensity, difficulty in detection, unconscious occurrence, and difficulty in concealing or faking them. Microexpression analysis typically requires video, while ordinary expressions can be analyzed from images. Because microexpressions arise spontaneously in an unconscious state, are difficult to conceal or fake, and are usually directly related to genuine emotions, they are relatively reliable in sentiment analysis and have broad application prospects. On the other hand, human recognition of microexpressions is difficult, training is challenging, and the success rate is low; therefore, automatic microexpression recognition by computers is necessary.

[0003] Currently, the main challenges in micro-expression recognition are twofold: First, micro-expressions are characterized by short duration, low intensity, and difficulty in feature extraction, thus requiring appropriate data preprocessing and feature extraction. Second, due to the difficulties in data collection and identification of micro-expressions, there are relatively few existing micro-expression datasets, which hinders the application of deep learning in micro-expression recognition.

[0004] Existing micro-expression recognition methods typically rely on traditional machine learning, designing handcrafted features to extract features from micro-expression fragments and classifying micro-expressions according to a framework of data preprocessing – feature extraction – feature classification. With the development of deep learning in computer vision in recent years, attempts to use deep learning methods for micro-expression recognition have gradually increased. However, existing deep learning-based micro-expression recognition methods generally have low accuracy due to the subtle and short duration of micro-expression movements, leaving room for improvement. Furthermore, the limited availability of micro-expression datasets also limits the generalization ability of existing micro-expression recognition models. Summary of the Invention

[0005] Purpose of the invention: In order to overcome the shortcomings of the existing technology, the present invention provides a micro-expression recognition method and device based on multi-task learning and global recurrent convolution. The global recurrent convolution is used to effectively extract global features of micro-expression video data. The multi-task learning method is used to simultaneously train the optical flow estimation task of facial micro-expression, the five-point feature point regression task of face, and the micro-expression classification and prediction task to improve the robustness and generalization ability of the model.

[0006] Technical Solution: Addressing the challenges and shortcomings of existing micro-expression recognition methods, this invention employs a multi-task learning approach combined with global recurrent convolution. First, given the short duration, low intensity, and difficulty in feature extraction of micro-expressions, this invention incorporates optical flow estimation into the micro-expression classification and prediction task. Since optical flow expresses image changes and contains information about target motion, adding optical flow estimation enhances the capture of facial movement changes in the temporal domain, thereby improving micro-expression recognition. Second, building upon the micro-expression classification and prediction and optical flow estimation tasks, this invention further incorporates a five-point facial feature regression task. Because the facial muscles used during micro-expressions are mainly distributed around the eyes, nose, and corners of the mouth, adding a five-point facial feature regression task enhances the capture of facial movement changes in the spatial domain, thus improving micro-expression recognition. Meanwhile, since the input requirements of each task in multi-task learning are different—for example, optical flow estimation requires the spatiotemporal information corresponding to the entire micro-expression video as input, while the five-point facial feature regression task requires the spatial information corresponding to each frame in the micro-expression video as input—this invention first uses global recurrent convolution to extract global features from each micro-expression video data, and then inputs the global features into each task in a form that meets the input requirements of each task. This invention extracts global features rather than local features because global features have good invariance and can better preserve the spatiotemporal information of the original data, which is beneficial to improving the multi-task learning effect. Furthermore, to address the problem of limited micro-expression datasets, this invention uses multiple micro-expression datasets to construct a training dataset and performs image enhancement operations such as random translation, random rotation, random scaling, random horizontal flipping, and random cropping on each data point, which can expand the size of the training dataset and improve the model's generalization ability. The specific scheme of this invention is as follows.

[0007] A micro-expression recognition method based on multi-task learning and global recurrent convolution includes the following steps:

[0008] S01: Extract raw video data from publicly available facial micro-expression datasets to form a training dataset. The raw video data includes micro-expression videos and their corresponding micro-expression classification labels.

[0009] S02: Preprocess the training dataset to obtain a fixed-length amplified frame sequence. Use the optical flow method to calculate the optical flow between adjacent frames and annotate the coordinates of the five feature points of the facial features. The five feature points of the facial features refer to the center of the left eye, the center of the right eye, the nose, the left corner of the mouth, and the right corner of the mouth.

[0010] S03: Constructing the convolutional neural network module I: Adjusting the frame size and number of channels of the augmented frame sequence;

[0011] S04: Construct a global recurrent convolution module II for global feature extraction;

[0012] S05: Using the global features obtained in step S04, the micro-expression classification labels in step S01, the inter-frame optical flow and five-point feature coordinates obtained in step S02, construct the multi-task learning module III to perform inter-frame optical flow estimation, five-point feature regression prediction of facial features and micro-expression classification prediction, thereby realizing micro-expression recognition.

[0013] S06: Train the network model, which mainly consists of convolutional neural network module I, global recurrent convolutional module II, and multi-task learning module III, using the training dataset, and update the parameters of each module using a gradient-based optimization method.

[0014] S07: Input the given video into the trained network model to predict the micro-expression classification result.

[0015] Specifically, in step S02, the training dataset is preprocessed to obtain a fixed-length augmented frame sequence. Specifically, for micro-expression video data of arbitrary length (usually 50-100 frames), 24 frames are taken at equal intervals from the first frame to the last frame, and each frame is grayscaled. The size of each frame is reduced to 128×128 using the nearest neighbor interpolation method to obtain an augmented frame sequence of 24 frames.

[0016] Specifically, in step S02, the optical flow between adjacent frames is calculated using the optical flow method, which is the TV-L1 optical flow algorithm. The TV-L1 optical flow algorithm is a two-frame estimation algorithm based on total variation, which uses two frames of images to estimate the optical flow vector of an object.

[0017] Specifically, in step S03, a convolutional neural network module I is constructed to adjust the frame size and number of channels of the augmented frame sequence. The convolutional neural network module I includes four cascaded two-dimensional convolutional layers with progressively increasing channel numbers. The input of the convolutional neural network module I is used as the input of the first two-dimensional convolutional layer, and the output of the last two-dimensional convolutional layer is used as the output of the convolutional neural network module I. Each frame in the augmented frame sequence obtained in step S02 is used sequentially as the input of the convolutional neural network module I, and the sequence formed by the outputs of the convolutional neural network module I corresponding to each frame is the output frame sequence after adjusting the frame size and number of channels. The convolutional neural network module I includes two-dimensional convolutional layer II, two-dimensional convolutional layer I-II, and two-dimensional convolutional layer I. -III. Two-dimensional convolutional layers I-IV: Within two-dimensional convolutional layer II, the entire input is convolved once, and the convolution result is used as the output of two-dimensional convolutional layer II; the output of two-dimensional convolutional layer II is used as the input of two-dimensional convolutional layer I-II, and the entire input is convolved once, and the convolution result is used as the output of two-dimensional convolutional layer I-II; the output of two-dimensional convolutional layer I-II is used as the input of two-dimensional convolutional layer I-III, the entire input is zero-paddinged and convolved once, and the convolution result is used as the output of two-dimensional convolutional layer I-III; the output of two-dimensional convolutional layer I-III is used as the input of two-dimensional convolutional layer I-IV, the entire input is convolved once, and the convolution result is used as the output of two-dimensional convolutional layer I-IV.

[0018] Specifically, in step S04, a global recurrent convolutional module II is constructed to perform global feature extraction. The global recurrent convolutional module II includes a location embedding module II-I and a recurrent convolutional module II-II. The input of the global recurrent convolutional module II is used as the input of the location embedding module II-I, the output of the location embedding module II-I is used as the input of the recurrent convolutional module II-II, and the output of the recurrent convolutional module II-II is used as the output of the global recurrent convolutional module II. Each frame in the output frame sequence obtained in step S03 is sequentially used as the input of the global recurrent convolutional module II, and the corresponding global recurrent network module II for each frame... The output sequence is the global feature map sequence of the output frame sequence; the position embedding module II-I includes a horizontal position embedding module II-II and a vertical position embedding module II-I-II. In the horizontal position embedding module II-II, the input is split into row vectors in the horizontal direction, and a position row vector of the same size representing the position information of the row vector is generated for each row vector. The position row vector is added to the original row vector to obtain the position embedding row vector. All the position embedding row vectors are merged in the horizontal direction to obtain the horizontal position embedding map; in the vertical position embedding module II-I-II, the input is split into row vectors in the horizontal direction. The system is vertically split into column vectors, and a position column vector of the same size representing the position information of each column vector is generated. This position column vector is added to the original column vector to obtain the position embedding column vector. All position embedding column vectors are then merged vertically to obtain the vertical position embedding map. The horizontal and vertical position embedding maps are the outputs of position embedding module II-I. The circular convolution module II-II includes a horizontal convolution module II-II-I and a vertical convolution module II-II-II. Within the horizontal convolution module II-II-I, the input horizontal position embedding map is first copied, and... The horizontal position embedding image is concatenated with the original horizontal position embedding image in the horizontal direction. The concatenated result is then convolved with an asymmetric convolution kernel of size 1×W to obtain a horizontal convolution feature map, where W is the width of the horizontal position embedding image. In the vertical convolution module II-II-II, the input vertical position embedding image is copied and concatenated with the original vertical position embedding image in the vertical direction. The concatenated result is then convolved with an asymmetric convolution kernel of size H×1 to obtain a vertical convolution feature map, where H is the height of the vertical position embedding image. The horizontal convolution feature map and the vertical convolution feature map are then summed, and the result is used as the output of the recurrent convolution module II-II.

[0019] Specifically, in step S05, a multi-task learning module III is constructed to perform inter-frame optical flow estimation, five-point feature point regression prediction of facial features, and micro-expression classification prediction. The multi-task learning module III includes an optical flow estimation module III-I, a facial feature point regression module III-II, and a three-dimensional convolutional neural network micro-expression classification prediction module III-III. The input of the multi-task learning module III serves as the input to all three modules simultaneously: the optical flow estimation module III-I, the facial feature point regression module III-II, and the three-dimensional convolutional neural network micro-expression classification prediction module III-III. -I, the outputs of the face feature point regression module III-II and the three-dimensional convolutional neural network micro-expression classification prediction module III-III together constitute the output of the multi-task learning module III; the global feature map sequence obtained in step S04 is used as the input of the multi-task learning module III, and the inter-frame optical flow estimation results output by the optical flow estimation module III-I, the five-point feature point regression prediction results of the face features output by the face feature point regression module III-II, and the micro-expression classification prediction results output by the three-dimensional convolutional neural network micro-expression classification prediction module III-III together constitute the output of the multi-task learning module III.

[0020] In the supervised learning process of optical flow estimation module III-I, optical flow features containing dynamic temporal information are extracted. RAFT (Recurrent All-Pairs Field Transforms) is used as the backbone network of optical flow estimation module III-I, and optical flow is estimated according to the following loss function:

[0021]

[0022] Among them: E of represents the mean squared error loss function for optical flow estimation; y represents the true optical flow vector between the reference image and the target image, i.e., the inter-frame optical flow calculated in step S02; This represents the predicted optical flow vector between the reference image and the target image, which is the inter-frame optical flow estimation result extracted by the optical flow estimation module III-I.

[0023] In the face feature point regression module III-II, the input is the shape features of the face image. The input features are flattened and passed through two consecutive fully connected layers to obtain the output, which is the predicted feature point location. Feature point regression prediction is performed using the following loss function:

[0024]

[0025] Among them: E ldm This represents the loss function for feature point regression prediction. and Let x and y represent the x and y coordinates of the i-th feature point, respectively, i = 1, 2, ..., n, n = 5, which represents the five feature points (number of feature points) on the face image; ||·||2 represents the L2 norm; d0 represents the actual distance between the pupils of both eyes; q represents the actual feature point coordinates, i.e., the coordinates of the five feature points of the facial features marked in step S02; This represents the feature point regression prediction coordinates, which are the five feature points of the face extracted by the face feature point regression module III-II.

[0026] In the supervised learning process of the micro-expression classification and prediction module III-III of the 3D convolutional neural network, facial feature vectors containing spatiotemporal information are extracted, and micro-expression prediction is performed according to the following loss function:

[0027]

[0028] Among them: E mer The cross-entropy loss function represents the micro-expression classification prediction; y (i) This represents the truth value of micro-expression classification, i.e., the micro-expression classification label obtained through step S01; This represents the micro-expression classification prediction value, which is the micro-expression classification prediction result extracted by the three-dimensional convolutional neural network micro-expression classification prediction module III-III. N represents the total number of micro-expression categories.

[0029] Specifically, in step S06, the network model is trained using the training dataset, and the parameters of each module are updated using a gradient-based optimization method. The network model is then updated according to the following loss function:

[0030] E mt =E mer +λ1E of +λ2E ldm (4)

[0031] Among them: E mt Let λ1 and λ2 represent the multi-task loss function, where λ1 and λ2 are constants.

[0032] An apparatus for micro-expression recognition based on the above-mentioned multi-task learning and global recurrent convolution includes an image acquisition unit, an input adaptation unit, a feature extraction unit, and a multi-task output unit.

[0033] The image acquisition unit is used to extract raw video data from the publicly available facial micro-expression dataset to form a training dataset, preprocess the raw video data to obtain a fixed-length amplified frame sequence, calculate the optical flow between adjacent frames using the optical flow method, and annotate the coordinates of five feature points of the facial features.

[0034] The input adaptation unit includes a convolutional neural network module I, which adjusts the frame size and number of channels of the amplified frame sequence;

[0035] The feature extraction unit includes a global recurrent convolution module II, which performs global feature extraction on the output frame sequence;

[0036] The multi-task output unit includes a multi-task learning module III, which performs multi-task learning on the results of the feature extraction unit and outputs the final inter-frame optical flow estimation, five-point feature point regression prediction of facial features, and micro-expression classification prediction results.

[0037] Beneficial Effects: The micro-expression recognition method and device based on multi-task learning and global recurrent convolution provided by this invention have the following advantages compared with the prior art: 1. It adopts an end-to-end deep learning framework to jointly learn facial micro-expression recognition, optical flow estimation, and feature point regression, and uses the correlation between tasks to promote micro-expression recognition. It can effectively identify the motion changes of facial micro-expressions in three-dimensional spatiotemporal space and realize the construction of a facial micro-expression recognition system; 2. Through global recurrent convolution, it can effectively extract global features of micro-expression video data; 3. It adopts a multi-task learning method to simultaneously train the optical flow estimation task, the five-point feature point regression task, and the micro-expression classification prediction task of facial micro-expression, which can effectively complete the micro-expression recognition task and improve the robustness and generalization ability of the model. Attached Figure Description

[0038] Figure 1 This is a schematic diagram illustrating the implementation process of the method of the present invention;

[0039] Figure 2 This is a schematic diagram of the structure of convolutional neural network module I;

[0040] Figure 3 This is a schematic diagram of the structure of the Global Recurrent Convolution Module II;

[0041] Figure 4 This is a structural diagram of the multi-task learning module III;

[0042] Figure 5 A schematic diagram of the micro-expression classification and prediction module III-III;

[0043] Figure 6 This is a flowchart illustrating the entire micro-expression recognition system. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0045] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0046] like Figure 1 The diagram shows a flowchart of a micro-expression recognition method based on multi-task learning and global recurrent convolution. The method includes the following steps:

[0047] S01: Extract raw video data from publicly available facial micro-expression datasets to form a training dataset. The raw video data includes micro-expression videos and their corresponding micro-expression classification labels.

[0048] To ensure the model has good generalization ability, the original video data was uniformly extracted from multiple publicly available facial micro-expression datasets, including the SMIC dataset, the CASME II dataset, and the SAMM dataset.

[0049] S02: Preprocess the training dataset to obtain a fixed-length amplified frame sequence. Use the optical flow method to calculate the optical flow between adjacent frames and label the coordinates of the five feature points of the facial features. The five feature points of the facial features refer to the center of the left eye, the center of the right eye, the nose, the left corner of the mouth, and the right corner of the mouth.

[0050] The training dataset is preprocessed to obtain a fixed-length augmented frame sequence. Specifically, for micro-expression video data of arbitrary length (typically 50-100 frames), 24 frames are taken at equal intervals from the first frame to the last frame. Each frame is then converted to grayscale, and the size of each frame is reduced to 128×128 using nearest neighbor interpolation, resulting in an augmented frame sequence of 24 frames. The optical flow method used is the TV-L1 optical flow algorithm, a two-frame estimation algorithm based on total variation that uses two frames to estimate the optical flow vector of an object.

[0051] S03: Constructing Convolutional Neural Network Module I: Adjusting the frame size and number of channels of the augmented frame sequence.

[0052] like Figure 2As shown, a convolutional neural network module I is constructed, and the frame size and number of channels of the frame sequence are adjusted. The convolutional neural network module I includes four cascaded two-dimensional convolutional layers with the number of channels increasing sequentially. The input of the convolutional neural network module I is used as the input of the first two-dimensional convolutional layer, and the output of the last two-dimensional convolutional layer is used as the output of the convolutional neural network module I. Each frame in the amplified frame sequence obtained in step S02 is used as the input of the convolutional neural network module I in sequence, and the sequence composed of the outputs of the convolutional neural network module I corresponding to each frame is the output frame sequence after adjusting the frame size and number of channels.

[0053] The convolutional neural network module I includes a two-dimensional convolutional layer II, two-dimensional convolutional layers I-II, two-dimensional convolutional layers I-III, and two-dimensional convolutional layers I-IV. In two-dimensional convolutional layer II, the entire input is convolved once, and the convolution result is used as the output of two-dimensional convolutional layer II. The output of two-dimensional convolutional layer II is used as the input of two-dimensional convolutional layer I-II, and the entire input is convolved once, and the convolution result is used as the output of two-dimensional convolutional layer I-II. The output of two-dimensional convolutional layer I-II is used as the input of two-dimensional convolutional layer I-III, the entire input is zero-paddinged, and a convolution is performed once, and the convolution result is used as the output of two-dimensional convolutional layer I-III. The output of two-dimensional convolutional layer I-III is used as the input of two-dimensional convolutional layer I-IV, the entire input is convolved once, and the convolution result is used as the output of two-dimensional convolutional layer I-IV.

[0054] In this example, the number of output channels for two-dimensional convolutional layer II, two-dimensional convolutional layer I-II, two-dimensional convolutional layer I-III, and two-dimensional convolutional layer I-IV are 8, 32, 64, and 128, respectively; the filter sizes are 4×4, 3×3, 2×2, and 1×1, respectively; and the stride sizes are 2×2, 2×2, 2×2, and 1×1, respectively.

[0055] S04: Construct a global recurrent convolution module II for global feature extraction.

[0056] like Figure 3 As shown, the global recurrent convolution module II includes a location embedding module II-I and a recurrent convolution module II-II. The input of the global recurrent convolution module II is used as the input of the location embedding module II-I, the output of the location embedding module II-I is used as the input of the recurrent convolution module II-II, and the output of the recurrent convolution module II-II is used as the output of the global recurrent convolution module II. Each frame in the output frame sequence obtained in step S03 is used as the input of the global recurrent convolution module II in sequence, and the sequence composed of the outputs of the global recurrent network module II corresponding to each frame is the global feature map sequence of the output frame sequence.

[0057] The position embedding module II-I includes a horizontal position embedding module II-II and a vertical position embedding module II-I-II. In the horizontal position embedding module II-II, the input is split into row vectors horizontally, and a position row vector of the same size representing the position information of each row vector is generated. This position row vector is added to the original row vector to obtain a position embedding row vector. All position embedding row vectors are then merged horizontally to obtain a horizontal position embedding map. In the vertical position embedding module II-I-II, the input is split into column vectors vertically, and a position column vector of the same size representing the position information of each column vector is generated. This position column vector is added to the original column vector to obtain a position embedding column vector. All position embedding column vectors are then merged vertically to obtain a vertical position embedding map. The horizontal and vertical position embedding maps serve as the outputs of the position embedding module II-I.

[0058] The recurrent convolution module II-II includes a horizontal convolution module II-II-I and a vertical convolution module II-II-II. In the horizontal convolution module II-II-I, the input horizontal position embedding image is copied and concatenated with the original horizontal position embedding image in the horizontal direction. The concatenated result is then convolved using an asymmetric convolution kernel of size 1×W to obtain a horizontal convolution feature map, where W is the width of the horizontal position embedding image. In the vertical convolution module II-II-II, the input vertical position embedding image is copied and concatenated with the original vertical position embedding image in the vertical direction. The concatenated result is then convolved using an asymmetric convolution kernel of size H×1 to obtain a vertical convolution feature map, where H is the height of the vertical position embedding image. The horizontal and vertical convolution feature maps are then summed, and the result is used as the output of the recurrent convolution module II-II.

[0059] In this example, the image size of each frame in the output frame sequence obtained in step S03 is 16×16, and the corresponding size of the horizontal position embedding map and the vertical position embedding map is also 16×16. Therefore, the values ​​of H and W are both 16.

[0060] S05: Using the global features obtained in step S04, the micro-expression classification labels in step S01, the inter-frame optical flow and five-point feature coordinates obtained in step S02, construct the multi-task learning module III to perform inter-frame optical flow estimation, five-point feature regression prediction of facial features and micro-expression classification prediction, thereby realizing micro-expression recognition.

[0061] like Figure 4The diagram shows the construction of a multi-task learning module III, which performs inter-frame optical flow estimation, five-point feature point regression prediction of facial features, and micro-expression classification prediction. Multi-task learning module III includes an optical flow estimation module III-I, a facial feature point regression module III-II, and a 3D convolutional neural network micro-expression classification prediction module III-III. The input of multi-task learning module III serves as the input to all three modules simultaneously. The outputs of these modules collectively constitute the output of multi-task learning module III. The global feature map sequence obtained in step S04 is used as the input to multi-task learning module III. The inter-frame optical flow estimation results output by optical flow estimation module III-I, the five-point feature point regression prediction results of facial features output by facial feature point regression module III-II, and the micro-expression classification prediction results output by 3D convolutional neural network micro-expression classification prediction module III-III collectively constitute the output of multi-task learning module III.

[0062] In the supervised learning process of optical flow estimation module III-I, optical flow features containing dynamic temporal information are extracted. RAFT (Recurrent All-Pairs Field Transforms) is used as the backbone network of optical flow estimation module III-I, and optical flow is estimated according to the following loss function:

[0063]

[0064] Among them: E of represents the mean squared error loss function for optical flow estimation; y represents the true optical flow vector between the reference image and the target image, i.e., the inter-frame optical flow calculated in step S02; This represents the predicted optical flow vector between the reference image and the target image, which is the inter-frame optical flow estimation result extracted by the optical flow estimation module III-I.

[0065] In the face feature point regression module III-II, the input is the shape features of the face image. The input features are flattened and passed through two consecutive fully connected layers to obtain the output, which is the predicted feature point location. Feature point regression prediction is performed using the following loss function:

[0066]

[0067] Among them: E ldm This represents the loss function for feature point regression prediction. and Let x and y represent the x and y coordinates of the i-th feature point, respectively, i = 1, 2, ..., n, n = 5, which represents the five feature points (number of feature points) on the face image; ||·||2 represents the L2 norm; d0 represents the actual distance between the pupils of both eyes; q represents the actual feature point coordinates, i.e., the coordinates of the five feature points of the facial features marked in step S02; This represents the feature point regression prediction coordinates, which are the five feature points of the face extracted by the face feature point regression module III-II.

[0068] like Figure 5 As shown, in the supervised learning process of the micro-expression classification and prediction module III-III of the 3D convolutional neural network, facial feature vectors containing spatiotemporal information are extracted, and micro-expression prediction is performed according to the following loss function:

[0069]

[0070] Among them: E mer The cross-entropy loss function represents the micro-expression classification prediction; y (i) This represents the truth value of micro-expression classification, i.e., the micro-expression classification label obtained through step S01; This represents the micro-expression classification prediction value, which is the micro-expression classification prediction result extracted by the three-dimensional convolutional neural network micro-expression classification prediction module III-III. N represents the total number of micro-expression categories.

[0071] S06: Train the network model, which mainly consists of Convolutional Neural Network Module I, Global Recurrent Convolutional Module II, and Multi-Task Learning Module III, using the training dataset, and update the parameters of each module using a gradient-based optimization method.

[0072] Train the entire network model using an end-to-end method (e.g.) Figure 6 As shown, each training iteration includes three tasks: optical flow estimation, feature point regression prediction, and micro-expression classification prediction. Within a multi-task learning framework, the correlation between these three tasks is utilized to improve micro-expression recognition. The network model is trained using the training dataset, and the parameters of each module are updated using a gradient-based optimization method. The network model is updated according to the following loss function:

[0073] E mt =E mer +λ1E of +λ2E ldm (4)

[0074] Among them: E mt Let λ1 and λ2 represent the multi-task loss function, where λ1 and λ2 are constants. In this example, we take λ1 = 10 and λ2 = 5.

[0075] S07: Input the given video into the trained network model to predict the micro-expression classification result.

[0076] When making predictions, you only need to input the video of the micro-expression to be tested, and you can get the results of inter-frame optical flow estimation, five-point feature point regression prediction of facial features and micro-expression classification prediction.

[0077] like Figure 6 As shown, this is an apparatus for micro-expression recognition based on the above-mentioned multi-task learning and global recurrent convolution method, including an image acquisition unit, an input adaptation unit, a feature extraction unit, and a multi-task output unit.

[0078] The image acquisition unit is used to extract raw video data from the publicly available facial micro-expression dataset to form a training dataset, preprocess the raw video data to obtain a fixed-length amplified frame sequence, calculate the optical flow between adjacent frames using the optical flow method, and annotate the coordinates of five feature points of the facial features.

[0079] The input adaptation unit includes a convolutional neural network module I, which adjusts the frame size and number of channels of the amplified frame sequence;

[0080] The feature extraction unit includes a global recurrent convolution module II, which performs global feature extraction on the output frame sequence;

[0081] The multi-task output unit includes a multi-task learning module III, which performs multi-task learning on the results of the feature extraction unit and outputs the final inter-frame optical flow estimation, five-point feature point regression prediction of facial features, and micro-expression classification prediction results.

[0082] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0083] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0084] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A micro-expression recognition method based on multi-task learning and global recurrent convolution, characterized in that: Includes the following steps: S01: Extract raw video data from publicly available facial micro-expression datasets to form a training dataset. The raw video data includes micro-expression videos and their corresponding micro-expression classification labels. S02: Preprocess the training dataset to obtain a fixed-length amplified frame sequence. Use the optical flow method to calculate the optical flow between adjacent frames and annotate the coordinates of the five feature points of the facial features. The five feature points of the facial features refer to the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth, and the right corner of the mouth. S03: Constructing the convolutional neural network module I: Adjusting the frame size and number of channels of the augmented frame sequence; S04: Construct a global recurrent convolution module II for global feature extraction; S05: Using the global features obtained in step S04, the micro-expression classification labels in step S01, and the inter-frame optical flow and five-point feature coordinates obtained in step S02, a multi-task learning module III is constructed to perform inter-frame optical flow estimation, five-point feature regression prediction of facial features, and micro-expression classification prediction, thereby achieving micro-expression recognition. The multi-task learning module III includes an optical flow estimation module III-I, a facial feature regression module III-II, and a three-dimensional convolutional neural network micro-expression classification prediction module III-III. The input of the multi-task learning module III serves as the input of the optical flow estimation module III-I, the facial feature regression module III-II, and the three-dimensional convolutional neural network micro-expression module III-III. The input of the classification prediction module III-III, the output of the optical flow estimation module III-I, the facial feature point regression module III-II, and the output of the 3D convolutional neural network micro-expression classification prediction module III-III together constitute the output of the multi-task learning module III; the global feature map sequence obtained in step S04 is used as the input of the multi-task learning module III, and the optical flow estimation results between adjacent frames output by the optical flow estimation module III-I, the five-point feature point regression prediction results of facial features output by the facial feature point regression module III-II, and the micro-expression classification prediction results output by the 3D convolutional neural network micro-expression classification prediction module III-III together constitute the output of the multi-task learning module III; S06: Train the network model consisting of Convolutional Neural Network Module I, Global Recurrent Convolutional Module II, and Multi-Task Learning Module III using the training dataset, and update the parameters of each module using a gradient-based optimization method. S07: Input the given video into the trained network model to predict the micro-expression classification result.

2. The micro-expression recognition method based on multi-task learning and global recurrent convolution as described in claim 1, characterized in that: In step S02, the training dataset is preprocessed to obtain a fixed-length augmented frame sequence. Specifically, for micro-expression video data of arbitrary length, 24 frames are taken at equal intervals from the first frame to the last frame, and each frame image is grayscaled. The size of each frame image is reduced to 128×128 by the nearest neighbor interpolation method to obtain an augmented frame sequence of 24 frames.

3. The micro-expression recognition method based on multi-task learning and global recurrent convolution as described in claim 1, characterized in that: In step S02, the optical flow between adjacent frames is calculated using the optical flow method, which is the TV-L1 optical flow algorithm. The TV-L1 optical flow algorithm is a two-frame estimation algorithm based on total variation, which uses two frames of images to estimate the optical flow vector of an object.

4. The micro-expression recognition method based on multi-task learning and global recurrent convolution as described in claim 1, characterized in that: In step S03, a convolutional neural network module I is constructed to adjust the frame size and number of channels of the augmented frame sequence. The convolutional neural network module I includes four cascaded two-dimensional convolutional layers with the number of channels increasing sequentially. The input of the convolutional neural network module I is used as the input of the first two-dimensional convolutional layer, and the output of the last two-dimensional convolutional layer is used as the output of the convolutional neural network module I. Each frame in the augmented frame sequence obtained in step S02 is used as the input of the convolutional neural network module I in sequence, and the sequence composed of the outputs of the convolutional neural network module I corresponding to each frame is the output frame sequence after adjusting the frame size and number of channels. The convolutional neural network module I includes a two-dimensional convolutional layer II, two-dimensional convolutional layers I-II, two-dimensional convolutional layers I-III, and two-dimensional convolutional layers I-IV. In two-dimensional convolutional layer II, the entire input is convolved once, and the convolution result is used as the output of two-dimensional convolutional layer II. The output of two-dimensional convolutional layer II is used as the input of two-dimensional convolutional layer I-II, and the entire input is convolved once, and the convolution result is used as the output of two-dimensional convolutional layer I-II. The output of two-dimensional convolutional layer I-II is used as the input of two-dimensional convolutional layer I-III, the entire input is zero-paddinged, and a convolution is performed once, and the convolution result is used as the output of two-dimensional convolutional layer I-III. The output of two-dimensional convolutional layer I-III is used as the input of two-dimensional convolutional layer I-IV, the entire input is convolved once, and the convolution result is used as the output of two-dimensional convolutional layer I-IV.

5. The micro-expression recognition method based on multi-task learning and global recurrent convolution as described in claim 1, characterized in that: In step S04, a global recurrent convolutional module II is constructed to extract global features. The global recurrent convolutional module II includes a location embedding module II-I and a recurrent convolutional module II-II. The input of the global recurrent convolutional module II is used as the input of the location embedding module II-I, the output of the location embedding module II-I is used as the input of the recurrent convolutional module II-II, and the output of the recurrent convolutional module II-II is used as the output of the global recurrent convolutional module II. Each frame in the output frame sequence obtained in step S03 is used as the input of the global recurrent convolutional module II in sequence. The sequence composed of the outputs of the global recurrent network module II corresponding to each frame is the global feature map sequence of the output frame sequence. The position embedding module II-I includes a horizontal position embedding module II-II and a vertical position embedding module II-I-II. In the horizontal position embedding module II-II, the input is split into row vectors in the horizontal direction, and a position row vector of the same size representing the position information of the row vector is generated for each row vector. The position row vector is added to the original row vector to obtain the position embedding row vector. All the position embedding row vectors are merged in the horizontal direction to obtain the horizontal position embedding map. Within the vertical position embedding module II-I-II, the input is split into column vectors along the vertical direction, and a position column vector of the same size representing the position information of each column vector is generated. This position column vector is added to the original column vector to obtain the position embedding column vector. All position embedding column vectors are merged sequentially along the vertical direction to obtain the vertical position embedding map. The horizontal position embedding map and the vertical position embedding map are the outputs of the position embedding module II-I. The recurrent convolution module II-II includes a horizontal convolution module II-II-I and a vertical convolution module II-II-II. In the horizontal convolution module II-II-I, the input horizontal position embedding image is copied and concatenated with the original horizontal position embedding image in the horizontal direction. The concatenated result is then convolved using an asymmetric convolution kernel of size 1×W to obtain a horizontal convolution feature map, where W is the width of the horizontal position embedding image. In the vertical convolution module II-II-II, the input vertical position embedding image is copied and concatenated with the original vertical position embedding image in the vertical direction. The concatenated result is then convolved using an asymmetric convolution kernel of size H×1 to obtain a vertical convolution feature map, where H is the height of the vertical position embedding image. The horizontal and vertical convolution feature maps are then summed, and the result is used as the output of the recurrent convolution module II-II.

6. The micro-expression recognition method based on multi-task learning and global recurrent convolution as described in claim 1, characterized in that: In step S05, optical flow features containing dynamic temporal information are extracted during the supervised learning process of optical flow estimation module III-I, and optical flow is estimated according to the following loss function: Where: E of represents the mean squared error loss function for optical flow estimation; y represents the true optical flow vector between the reference image and the target image, i.e., the inter-frame optical flow calculated in step S02; This represents the predicted optical flow vector between the reference image and the target image, which is the inter-frame optical flow estimation result extracted by the optical flow estimation module III-I. The input to the face feature point regression module III-II is the shape features of the face image, and the output is the predicted feature point locations. Feature point regression prediction is performed using the following loss function: Where: E ldm This represents the loss function for feature point regression prediction. and Let x and y represent the x and y coordinates of the i-th feature point, i = 1, 2, ..., n, and n = 5, where n is the five feature points on the face image; ||·||2 represents the L2 norm; d0 represents the actual distance between the pupils of both eyes; and q represents the actual feature point coordinates, i.e., the coordinates of the five feature points of the facial features marked in step S02. This represents the feature point regression prediction coordinates, which are the five feature points of the facial features extracted by the facial feature point regression module III-II. In the supervised learning process of the micro-expression classification and prediction module III-III of the 3D convolutional neural network, facial feature vectors containing spatiotemporal information are extracted, and micro-expression prediction is performed according to the following loss function: Where: E mer The cross-entropy loss function represents the micro-expression classification prediction; y (i) This represents the truth value of micro-expression classification, i.e., the micro-expression classification label obtained through step S01; This represents the micro-expression classification prediction value, which is the micro-expression classification prediction result extracted by the three-dimensional convolutional neural network micro-expression classification prediction module III-III. N represents the total number of micro-expression categories.

7. The micro-expression recognition method based on multi-task learning and global recurrent convolution according to claim 1, characterized in that: In step S06, the network model is trained using the training dataset, and the parameters of each module are updated using a gradient-based optimization method. The network model is updated according to the following loss function: AND mt =And mer +λ1E of +λ2E ldm (4) Where: E mt Let λ1 and λ2 represent the multi-task loss function, where λ1 and λ2 are constants.

8. An apparatus for implementing the micro-expression recognition method based on multi-task learning and global recurrent convolution as described in any one of claims 1 to 6, characterized in that: It includes an image acquisition unit, an input adaptation unit, a feature extraction unit, and a multi-task output unit; The image acquisition unit is used to extract raw video data from the publicly available facial micro-expression dataset to form a training dataset, preprocess the raw video data to obtain a fixed-length amplified frame sequence, calculate the optical flow between adjacent frames using the optical flow method, and annotate the coordinates of five feature points of the facial features. The input adaptation unit includes a convolutional neural network module I, which adjusts the frame size and number of channels of the amplified frame sequence; The feature extraction unit includes a global recurrent convolution module II, which performs global feature extraction on the output frame sequence; The multi-task output unit includes a multi-task learning module III, which performs multi-task learning on the results of the feature extraction unit and outputs the final inter-frame optical flow estimation, five-point feature point regression prediction of facial features, and micro-expression classification prediction results.

Images