A micro-expression recognition method and system based on PCANet+ and LSTM

Abstract

The present invention provides a micro-expression recognition method and system based on PCANet+ and LSTM in the field of expression recognition technology. The method includes: step S10, performing key point detection on the face of each frame of a head image in a portrait video; step S20, preprocessing the head image based on the key points to obtain a plurality of face images; step S30, normalizing the number of frames of the face images; step S40, performing optical flow calculation on each face image to obtain an optical flow image sequence; step S50, inputting the optical flow image sequence into a PCANet+ network to obtain a plurality of feature maps, and performing weighted averaging on each feature map to obtain a two-dimensional feature map; step S60, intercepting feature regions from the two-dimensional feature map based on the key points, inputting each feature region into an LSTM network to extract temporal features, and performing weighted summation to obtain scores of different categories of micro-expressions; and step S70, using a Softmax function to map the scores to probabilities of different categories of micro-expressions. The present invention has the advantage of greatly improving the recognition effect of micro-expressions.

Description

Technical Field

[0001] This invention relates to the field of facial expression recognition technology, and in particular to a micro-expression recognition method and system based on PCANet+ and LSTM. Background Technology

[0002] With the advancement of technology, artificial intelligence is also constantly developing, including facial expression recognition technology. By automatically recognizing the micro-expressions of people in videos, it is possible to quickly determine the current mood and psychological activities of the person. However, traditional deep learning-based micro-expression recognition methods often contain deep network layers, resulting in too many parameters. When recognizing micro-expressions with limited data, overfitting is likely to occur. Furthermore, two-dimensional convolutional neural networks cannot extract complete temporal information of micro-expressions, leading to poor recognition results.

[0003] Therefore, how to provide a micro-expression recognition method and system based on PCANet+ and LSTM to improve the recognition effect of micro-expressions has become an urgent technical problem to be solved. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a micro-expression recognition method and system based on PCANet+ and LSTM, so as to improve the recognition effect of micro-expressions.

[0005] In a first aspect, the present invention provides a micro-expression recognition method based on PCANet+ and LSTM, comprising the following steps:

[0006] Step S10: Acquire a portrait video and perform key point detection on the faces of the head images in each frame of the portrait video;

[0007] Step S20: Based on the key points, perform preprocessing such as alignment, cropping and scaling on each frame of head image to obtain several face images;

[0008] Step S30: Normalize the number of frames in the face image;

[0009] Step S40: Perform optical flow calculation on each of the face images to obtain an optical flow image sequence;

[0010] Step S50: Input the optical flow image sequence into the PCANet+ network for spatial feature extraction to obtain several feature maps. Then, perform a weighted average of the feature maps to obtain a two-dimensional feature map.

[0011] Step S60: Based on each of the key points, extract feature regions from the two-dimensional feature map, input each of the feature regions into the LSTM network to extract temporal features, and perform weighted summation on each of the temporal features to obtain scores for different categories of micro-expressions;

[0012] Step S70: Use the Softmax function to map the scores to the probabilities of different categories of micro-expressions in order to complete the micro-expression recognition.

[0013] Further, step S10 specifically includes:

[0014] A portrait video is acquired, and 68 key points of the face in each frame of the portrait video are detected using an active shape model.

[0015] Further, step S20 specifically includes:

[0016] The inner corners of the left and right eyes in each frame of the head image are obtained from the key points. The head images in each frame are rotated and aligned based on the line connecting the inner corners of the left and right eyes. The face regions in the head images are then cropped based on the key points. The cropped face regions are scaled to a uniform size to complete the preprocessing of the head images in each frame, resulting in several face images.

[0017] Further, step S30 specifically includes:

[0018] After grayscale processing of each frame of the face image, the frame number of the face image is normalized using a time interpolation algorithm.

[0019] Furthermore, in step S60, the feature region includes at least the eyebrow region, the eye region, the mouth region, and the nose wing region.

[0020] Secondly, the present invention provides a micro-expression recognition system based on PCANet+ and LSTM, comprising the following modules:

[0021] The key point detection module is used to acquire portrait videos and perform key point detection on the faces of each frame of the portrait video.

[0022] The head image preprocessing module is used to perform alignment, cropping, and scaling preprocessing on each frame of head image based on the key points to obtain several face images.

[0023] A frame number normalization module is used to normalize the frame number of the face image;

[0024] An optical flow calculation module is used to perform optical flow calculations on each of the face images to obtain an optical flow image sequence;

[0025] The PCANet+ feature extraction module is used to input the optical flow image sequence into the PCANet+ network for spatial feature extraction, obtain several feature maps, and perform a weighted average of the feature maps to obtain a two-dimensional feature map.

[0026] The LSTM feature extraction module is used to extract feature regions from the two-dimensional feature map based on each key point, input each feature region into the LSTM network to extract temporal features, and perform weighted summation on each temporal feature to obtain scores for different categories of micro-expressions;

[0027] The score mapping module is used to map the scores to the probabilities of different categories of micro-expressions using the Softmax function, in order to complete the micro-expression recognition.

[0028] Furthermore, the key point detection module is specifically used for:

[0029] A portrait video is acquired, and 68 key points of the face in each frame of the portrait video are detected using an active shape model.

[0030] Furthermore, the head image preprocessing module is specifically used for:

[0031] The inner corners of the left and right eyes in each frame of the head image are obtained from the key points. The head images in each frame are rotated and aligned based on the line connecting the inner corners of the left and right eyes. The face regions in the head images are then cropped based on the key points. The cropped face regions are scaled to a uniform size to complete the preprocessing of the head images in each frame, resulting in several face images.

[0032] Furthermore, the frame number normalization module is specifically used for:

[0033] After grayscale processing of each frame of the face image, the frame number of the face image is normalized using a time interpolation algorithm.

[0034] Furthermore, in the LSTM feature extraction module, the feature regions include at least the eyebrow region, eye region, mouth region, and nose wing region.

[0035] The advantages of the present invention are:

[0036] By performing keypoint detection on the faces of each frame of the acquired portrait video, and preprocessing the face images of each frame based on the keypoints through alignment, cropping, and scaling, several face images are obtained, and the frame number of the face images is normalized. Then, optical flow calculation is performed on each face image to obtain an optical flow image sequence. The optical flow image sequence is input into the PCANet+ network for spatial feature extraction and weighted averaging to obtain a two-dimensional feature map. Feature regions are extracted from the two-dimensional feature map and input into the LSTM network to extract temporal features. The temporal features are weighted and summed to obtain scores for different categories of micro-expressions. Finally, the Softmax function is used to map the scores to the probabilities of different categories of micro-expressions to complete the micro-expression recognition. That is, the PCANet+ network and the LSTM network are combined. The PCANet+ network extracts spatial features, and the LSTM network extracts temporal features, effectively extracting the spatiotemporal features of micro-expressions. Moreover, the PCANet+ network can directly calculate the network parameters of the current layer through the input of the current layer, reducing the network parameters and computational load, avoiding overfitting, and ultimately greatly improving the micro-expression recognition effect. Attached Figure Description

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0038] Figure 1 This is a flowchart of a micro-expression recognition method based on PCANet+ and LSTM according to the present invention.

[0039] Figure 2 This is a schematic diagram of the structure of a micro-expression recognition system based on PCANet+ and LSTM according to the present invention. Detailed Implementation

[0040] The overall idea of ​​the technical solution in this application embodiment is as follows: Combining PCANet+ network and LSTM network, spatial features are extracted through PCANet+ network and temporal features are extracted through LSTM network to effectively extract the spatiotemporal features of micro-expressions. PCANet+ network can directly calculate the network parameters of the current layer through the input of the current layer, reducing the network parameters and computation, avoiding overfitting, and improving the recognition effect of micro-expressions.

[0041] Please refer to Figures 1 to 2 As shown, a preferred embodiment of the micro-expression recognition method based on PCANet+ and LSTM of the present invention includes the following steps:

[0042] Step S10: Acquire a portrait video and perform key point detection on the faces of the head images in each frame of the portrait video;

[0043] Step S20: Based on the key points, perform preprocessing such as alignment, cropping and scaling on each frame of head image to obtain several face images;

[0044] Step S30: Normalize the number of frames in the face image;

[0045] Step S40: Perform optical flow calculation on each of the face images to obtain an optical flow image sequence;

[0046] Step S50: Input the optical flow image sequence into the PCANet+ network for spatial feature extraction to obtain several feature maps. Then, perform a weighted average of the feature maps to obtain a two-dimensional feature map.

[0047] Step S60: Based on each of the key points, extract feature regions from the two-dimensional feature map, input each of the feature regions into the LSTM network to extract temporal features, and perform weighted summation on each of the temporal features to obtain scores for different categories of micro-expressions;

[0048] Considering that micro-expressions only show obvious movement in certain areas of the face, not all facial regions are helpful for micro-expression classification. Therefore, feature regions are extracted from the two-dimensional feature map.

[0049] Step S70: Use the Softmax function to map the scores to the probabilities of different categories of micro-expressions in order to complete the micro-expression recognition.

[0050] Step S10 specifically involves:

[0051] A portrait video is acquired, and 68 key points of the face in each frame of the portrait video are detected using an active shape model.

[0052] Active shape model is built on the point distribution model. When detecting faces, it takes into account prior knowledge such as grayscale, size, shape and approximate position of the image. By learning from the feature points labeled in the training set, it obtains a statistical model of the feature point distribution of the training image samples. Using this as the initial position, it iterates to obtain the shape model of the target image. Finally, it applies shape constraints on the test set to search for the best matching point, thereby realizing the localization of facial feature points (key points).

[0053] Step S20 specifically involves:

[0054] The inner corners of the left and right eyes in each frame of the head image are obtained from the key points. The head images in each frame are rotated and aligned based on the line connecting the inner corners of the left and right eyes. The face regions in the head images are then cropped based on the key points. The cropped face regions are scaled to a uniform size to complete the preprocessing of the head images in each frame, resulting in several face images.

[0055] To eliminate the impact of head rotation and displacement on micro-expression recognition, face alignment of the head image is required based on the detected key points. Since the relative positions of the feature points at the inner corners of the left and right eyes are relatively stable and do not change due to facial muscle movements, this invention performs horizontal alignment based on the line connecting the inner corners of the left and right eyes. Then, the face region is cropped based on the aligned image and key points, removing areas unrelated to micro-expressions such as clothing, background, and hair from the head image. Due to variations in head rotation and distance from the camera, the cropped face image size is not entirely consistent. To meet the input size requirements of subsequent networks, the size needs to be standardized.

[0056] Step S30 specifically involves:

[0057] After grayscale processing of each frame of the face image, the frame number of the face image is normalized using a time interpolation algorithm.

[0058] Because micro-expressions vary in duration during acquisition and different camera frame rates are used in different datasets, the resulting micro-expression image sequences are inconsistent in length. Furthermore, this invention requires the use of an LSTM network for micro-expression feature extraction. To facilitate subsequent LSTM network processing, a temporal interpolation algorithm is used during data preprocessing to normalize the frame count of the micro-expression sample sequence (face image). The temporal interpolation algorithm, originally used for lip-reading, is a stream-based interpolation method that maps the entire micro-expression sequence to a curve, with each frame in the sequence corresponding to a point on the curve. The interpolated image sequence is then obtained by resampling on the curve.

[0059] In step S40, optical flow is a two-dimensional vector field on the image plane, describing the instantaneous velocity of pixel motion in two consecutive frames of a video sequence. To improve the feature learning effect of the PCANet+ network, this invention performs intensive optical flow calculation on image sequences of micro-expressions (face images) to enhance facial motion information. Optical flow calculation relies on two basic assumptions: 1. Brightness remains constant, that is, the brightness of pixels at corresponding positions remains constant in the image sequence during object movement and shooting; 2. Temporal continuity, that is, the movement of target pixels between adjacent frames cannot be too drastic, thus ensuring that pixel coordinates are continuously differentiable in time.

[0060] The optical flow calculation process is as follows:

[0061] Optical flow represents an image sequence as a three-dimensional matrix, where the brightness of a pixel in the sequence is represented as I(x,y,t), where x,y,t are its spatiotemporal coordinates. This pixel reaches the next frame after a time interval Δt and undergoes a displacement of Δx and Δy in the image. Based on the assumption that the brightness remains constant, the intensity of the pixel is constant before and after the motion, resulting in the following formula:

[0062] I(x,y,t)=I(x+Δx,y+Δy,t+Δt)--------------------------(1)

[0063] Based on the assumption of time continuity, a Taylor expansion of the right side of equation (1) yields the following equation:

[0064]

[0065] Where ε represents a higher-order infinitesimal, which can be ignored; substituting formula (2) into formula (1) and dividing by Δt, we get the following formula:

[0066]

[0067] Let u and v represent the velocity vectors of the pixel on the x-axis and y-axis, respectively, i.e., u = Δx / Δt, v ​​= Δy / Δt. Substituting these into formula (3), we get the following formula:

[0068] I x u+I y v+I t =0--------------------------------------(4)

[0069] (u,v) represents the optical flow field generated by the pixel within time Δt, which can be obtained by adding constraints. Different constraints can lead to different optical flow field calculation methods. This invention applies the TV-L1 algorithm to the calculation of the optical flow field. The TV-L1 algorithm introduces a subspace trajectory model to ensure the temporal consistency of the optical flow, while preserving the edge features in the image. For a pixel in a micro-expression sequence, the continuous optical flow field is first calculated using the optical flow estimation loss function shown in formula (5).

[0070]

[0071] Where L represents the length of the micro-expression image sequence; Represents the R-basis trajectory used to construct the trajectory space; Represents the spatial domain of an image; lin: The mapping function represents the optical flow fields u(t) and v(t) that can be mapped to a new space constructed from the R-based trajectory. The first term of Equation (5) represents the penalty term for the constant brightness constraint, the second term is used to make the derived optical flow lie on the base trajectory, and the third term represents the spatial regularization of the trajectory model coefficients based on the total variation.

[0072] Let a sequence of micro-expression images be... The first frame is set as the reference frame, and the horizontal and vertical optical flow components U of all other frames are calculated according to the optical flow calculation method described above. The calculated optical flow sequences are then stacked frame by frame and input into the PCANet+ network; for optical flow sequence U, Sampling is performed using sliding windows of size T and step size s to obtain two subsets μ and ν, defined as in formula (6):

[0073]

[0074] in, By concatenating the corresponding elements in μ and ν using formula (6), an input sequence Γ consisting of stacked optical flow components is formed:

[0075]

[0076] Here, || represents the connection of optical flow components in the channel dimension. By stacking the horizontal and vertical optical flow sequences of each video segment with T frames through multi-channel stacking operation, an optical flow image sequence with 2T channels is obtained.

[0077] Step S50 specifically involves:

[0078] The portrait video is divided into K segments, each containing T frames of face images. Optical flow calculations are performed on each segment, and the segments are stacked to obtain a set of multi-channel optical flow images Γ = {I1, I2, ..., I...}. K}, where I i Let I be the multi-channel image corresponding to the i-th video segment. Then, the optical flow image in Γ is sequentially input into a two-layer PCANet+ network. The number of filters in the first PCA convolutional layer is denoted as D1, with a size of k1×k1, and the number of filters in the second PCA convolutional layer is denoted as D2, with a size of k2×k2. Each convolutional layer is followed by a pooling layer. The first pooling layer is an average pooling layer with a fixed filter size of 3×3. The second pooling layer is a max pooling layer with a filter size of 3×3. For the multi-channel image I... i After passing through a two-layer PCANet+ network, a set of D2 two-dimensional feature images O is obtained. 2 ={O2 O 2 ,...,O 2}

[0079] Next, the feature map output from the second layer of the PCANet+ network (taking the key local facial region) is directly used as the input to the subsequent LSTM network. As can be seen from the filter learning process of PCANet+, the PCA filter in the second layer is transformed from the feature vectors corresponding to the first L2 largest eigenvalues ​​of that layer. The larger the eigenvalue of the filter, the more important the classification information contained in the feature map output after convolution; therefore, next, according to the magnitude of the corresponding eigenvalue of the filter, a weighted average is performed on the feature map output from the second layer, as shown in formula (8), to obtain a two-dimensional feature map O. i At the same time, this also helps to standardize the input size of the LSTM network, making it unaffected by the number of PCA filters.

[0080]

[0081] in, This represents the feature map of the j-th filter output in layer 2; λ j This represents the eigenvalue corresponding to the j-th filter.

[0082] In step S60, the feature region includes at least the eyebrow region, eye region, mouth region, and nose wing region.

[0083] After preprocessing, a sample in the micro-expression dataset yields a sequence of L images. During optical flow calculation, the first frame is used as the reference frame to calculate the optical flow of all remaining frames, resulting in L-1 dual-channel optical flow feature maps containing both horizontal and vertical optical flow components. These feature maps are stacked across T frames to obtain K = L-T+1 multi-channel images, where T is an odd number. Each multi-channel optical flow image undergoes feature extraction via PCANet+ and key region feature segmentation to obtain two-dimensional features near the eyebrows, eyes, mouth, and nose. These two-dimensional features are then converted into one-dimensional vectors and concatenated to provide the input to an LSTM network at a given time node. Since the LSTM network's input is temporal data containing K feature vectors, and its output is the scores for each micro-expression category, a many-to-one unfolding model is used for training.

[0084] The LSTM-based feature learning model consists of two main parts: a temporal feature extraction network composed of two LSTM layers and a classifier consisting of a fully connected layer and a softmax function. Each LSTM layer is expanded into K LSTM units, corresponding to K temporal data points generated from micro-expression samples. By adjusting the parameters of the gating units, sequential information is selectively memorized, and temporal features are extracted. Then, the fully connected layer performs a weighted summation of the extracted features to obtain scores for each micro-expression category. Finally, the softmax function maps the scores to probabilities, which is the final classification result for the micro-expressions. The model uses a cross-entropy loss function to optimize the network.

[0085]

[0086] in, y represents the predicted value of the k-th emotion category corresponding to the micro-expression sample output by the Softmax function; y represents the one-hot encoded vector of the sample's true label; y k This represents the value of the sample in the k-th emotion category. Its value is 1 when the true label of the sample is k, and 0 otherwise.

[0087] A preferred embodiment of the micro-expression recognition system based on PCANet+ and LSTM of the present invention includes the following modules:

[0088] The key point detection module is used to acquire portrait videos and perform key point detection on the faces of each frame of the portrait video.

[0089] The head image preprocessing module is used to perform alignment, cropping, and scaling preprocessing on each frame of head image based on the key points to obtain several face images.

[0090] A frame number normalization module is used to normalize the frame number of the face image;

[0091] An optical flow calculation module is used to perform optical flow calculations on each of the face images to obtain an optical flow image sequence;

[0092] The PCANet+ feature extraction module is used to input the optical flow image sequence into the PCANet+ network for spatial feature extraction, obtain several feature maps, and perform a weighted average of the feature maps to obtain a two-dimensional feature map.

[0093] The LSTM feature extraction module is used to extract feature regions from the two-dimensional feature map based on each key point, input each feature region into the LSTM network to extract temporal features, and perform weighted summation on each temporal feature to obtain scores for different categories of micro-expressions;

[0094] Considering that micro-expressions only show obvious movement in certain areas of the face, not all facial regions are helpful for micro-expression classification. Therefore, feature regions are extracted from the two-dimensional feature map.

[0095] The score mapping module is used to map the scores to the probabilities of different categories of micro-expressions using the Softmax function, in order to complete the micro-expression recognition.

[0096] The key point detection module is specifically used for:

[0097] A portrait video is acquired, and 68 key points of the face in each frame of the portrait video are detected using an active shape model.

[0098] Active shape model is built on the point distribution model. When detecting faces, it takes into account prior knowledge such as grayscale, size, shape and approximate position of the image. By learning from the feature points labeled in the training set, it obtains a statistical model of the feature point distribution of the training image samples. Using this as the initial position, it iterates to obtain the shape model of the target image. Finally, it applies shape constraints on the test set to search for the best matching point, thereby realizing the localization of facial feature points (key points).

[0099] The human head image preprocessing module is specifically used for:

[0100] The inner corners of the left and right eyes in each frame of the head image are obtained from the key points. The head images in each frame are rotated and aligned based on the line connecting the inner corners of the left and right eyes. The face regions in the head images are then cropped based on the key points. The cropped face regions are scaled to a uniform size to complete the preprocessing of the head images in each frame, resulting in several face images.

[0101] To eliminate the impact of head rotation and displacement on micro-expression recognition, face alignment of the head image is required based on the detected key points. Since the relative positions of the feature points at the inner corners of the left and right eyes are relatively stable and do not change due to facial muscle movements, this invention performs horizontal alignment based on the line connecting the inner corners of the left and right eyes. Then, the face region is cropped based on the aligned image and key points, removing areas unrelated to micro-expressions such as clothing, background, and hair from the head image. Due to variations in head rotation and distance from the camera, the cropped face image size is not entirely consistent. To meet the input size requirements of subsequent networks, the size needs to be standardized.

[0102] The frame number normalization module is specifically used for:

[0103] After grayscale processing of each frame of the face image, the frame number of the face image is normalized using a time interpolation algorithm.

[0104] Because micro-expressions vary in duration during acquisition and different camera frame rates are used in different datasets, the resulting micro-expression image sequences are inconsistent in length. Furthermore, this invention requires the use of an LSTM network for micro-expression feature extraction. To facilitate subsequent LSTM network processing, a temporal interpolation algorithm is used during data preprocessing to normalize the frame count of the micro-expression sample sequence (face image). The temporal interpolation algorithm, originally used for lip-reading, is a stream-based interpolation method that maps the entire micro-expression sequence to a curve, with each frame in the sequence corresponding to a point on the curve. The interpolated image sequence is then obtained by resampling on the curve.

[0105] In the optical flow calculation module, optical flow is a two-dimensional vector field on the image plane, describing the instantaneous velocity of pixel motion in two consecutive frames of a video sequence. To improve the feature learning effect of the PCANet+ network, this invention performs intensive optical flow calculation on image sequences of micro-expressions (face images) to enhance facial motion information. Optical flow calculation relies on two basic assumptions: 1. Brightness remains constant, that is, the brightness of pixels at corresponding positions remains constant in the image sequence during object movement and shooting; 2. Temporal continuity, that is, the movement of target pixels between adjacent frames cannot be too drastic, thus ensuring that pixel coordinates are continuously differentiable in time.

[0106] The optical flow calculation process is as follows:

[0107] Optical flow represents an image sequence as a three-dimensional matrix, where the brightness of a pixel in the sequence is represented as I(x,y,t), where x,y,t are its spatiotemporal coordinates. This pixel reaches the next frame after a time interval Δt and undergoes a displacement of Δx and Δy in the image. Based on the assumption that the brightness remains constant, the intensity of the pixel is constant before and after the motion, resulting in the following formula:

[0108] I(x,y,t)=I(x+Δx,y+Δy,t+Δt)--------------------------(1)

[0109] Based on the assumption of time continuity, a Taylor expansion of the right side of equation (1) yields the following equation:

[0110]

[0111] Where ε represents a higher-order infinitesimal, which can be ignored; substituting formula (2) into formula (1) and dividing by Δt, we get the following formula:

[0112]

[0113] Let u and v represent the velocity vectors of the pixel on the x-axis and y-axis, respectively, i.e., u = Δx / Δt, v ​​= Δy / Δt. Substituting these into formula (3), we get the following formula:

[0114] I x u+I y v+I t =0--------------------------------------(4)

[0115] (u,v) represents the optical flow field generated by the pixel within time Δt, which can be obtained by adding constraints. Different constraints can lead to different optical flow field calculation methods. This invention applies the TV-L1 algorithm to the calculation of the optical flow field. The TV-L1 algorithm introduces a subspace trajectory model to ensure the temporal consistency of the optical flow, while preserving the edge features in the image. For a pixel in a micro-expression sequence, the continuous optical flow field is first calculated using the optical flow estimation loss function shown in formula (5).

[0116]

[0117] Where L represents the length of the micro-expression image sequence; Represents the R-basis trajectory used to construct the trajectory space; Represents the spatial domain of an image; lin: The mapping function represents the optical flow fields u(t) and v(t) that can be mapped to a new space constructed from the R-based trajectory. The first term of Equation (5) represents the penalty term for the constant brightness constraint, the second term is used to make the derived optical flow lie on the base trajectory, and the third term represents the spatial regularization of the trajectory model coefficients based on the total variation.

[0118] Let a sequence of micro-expression images be... The first frame is set as the reference frame, and the horizontal and vertical optical flow components U of all other frames are calculated according to the optical flow calculation method described above. The calculated optical flow sequences are then stacked frame by frame and input into the PCANet+ network; for optical flow sequence U, Sampling is performed using sliding windows of size T and step size s to obtain two subsets μ and ν, defined as in formula (6):

[0119]

[0120] in, By concatenating the corresponding elements in μ and ν using formula (6), an input sequence Γ consisting of stacked optical flow components is formed:

[0121]

[0122] Here, || represents the connection of optical flow components in the channel dimension. By stacking the horizontal and vertical optical flow sequences of each video segment with T frames through multi-channel stacking operation, an optical flow image sequence with 2T channels is obtained.

[0123] The PCANet+ feature extraction module is specifically used for:

[0124] The portrait video is divided into K segments, each containing T frames of face images. Optical flow calculations are performed on each segment, and the segments are stacked to obtain a set of multi-channel optical flow images Γ = {I1, I2, ..., I...}. K}, where I i Let I be the multi-channel image corresponding to the i-th video segment. Then, the optical flow image in Γ is sequentially input into a two-layer PCANet+ network. The number of filters in the first PCA convolutional layer is denoted as D1, with a size of k1×k1, and the number of filters in the second PCA convolutional layer is denoted as D2, with a size of k2×k2. Each convolutional layer is followed by a pooling layer. The first pooling layer is an average pooling layer with a fixed filter size of 3×3. The second pooling layer is a max pooling layer with a filter size of 3×3. For the multi-channel image I... i After passing through a two-layer PCANet+ network, a set of D2 two-dimensional feature images O is obtained. 2 ={O 2 O 2 ,...,O 2}

[0125] Next, the feature map output from the second layer of the PCANet+ network (taking the key local facial region) is directly used as the input to the subsequent LSTM network. As can be seen from the filter learning process of PCANet+, the PCA filter in the second layer is transformed from the feature vectors corresponding to the first L2 largest eigenvalues ​​of that layer. The larger the eigenvalue of the filter, the more important the classification information contained in the feature map output after convolution; therefore, next, according to the magnitude of the corresponding eigenvalue of the filter, a weighted average is performed on the feature map output from the second layer, as shown in formula (8), to obtain a two-dimensional feature map O. i At the same time, this also helps to standardize the input size of the LSTM network, making it unaffected by the number of PCA filters.

[0126]

[0127] in, This represents the feature map of the j-th filter output in layer 2; λ j This represents the eigenvalue corresponding to the j-th filter.

[0128] In the LSTM feature extraction module, the feature regions include at least the eyebrow region, eye region, mouth region, and nose wing region.

[0129] After preprocessing, a sample in the micro-expression dataset yields a sequence of L images. During optical flow calculation, the first frame is used as the reference frame to calculate the optical flow of all remaining frames, resulting in L-1 dual-channel optical flow feature maps containing both horizontal and vertical optical flow components. These feature maps are stacked across T frames to obtain K = L-T+1 multi-channel images, where T is an odd number. Each multi-channel optical flow image undergoes feature extraction via PCANet+ and key region feature segmentation to obtain two-dimensional features near the eyebrows, eyes, mouth, and nose. These two-dimensional features are then converted into one-dimensional vectors and concatenated to provide the input to an LSTM network at a given time node. Since the LSTM network's input is temporal data containing K feature vectors, and its output is the scores for each micro-expression category, a many-to-one unfolding model is used for training.

[0130] The LSTM-based feature learning model consists of two main parts: a temporal feature extraction network composed of two LSTM layers and a classifier consisting of a fully connected layer and a softmax function. Each LSTM layer is expanded into K LSTM units, corresponding to K temporal data points generated from micro-expression samples. By adjusting the parameters of the gating units, sequential information is selectively memorized, and temporal features are extracted. Then, the fully connected layer performs a weighted summation of the extracted features to obtain scores for each micro-expression category. Finally, the softmax function maps the scores to probabilities, which is the final classification result for the micro-expressions. The model uses a cross-entropy loss function to optimize the network.

[0131]

[0132] in, y represents the predicted value of the k-th emotion category corresponding to the micro-expression sample output by the Softmax function; y represents the one-hot encoded vector of the sample's true label; y k This represents the value of the sample in the k-th emotion category. Its value is 1 when the true label of the sample is k, and 0 otherwise.

[0133] In summary, the advantages of this invention are:

[0134] By performing keypoint detection on the faces of each frame of the acquired portrait video, and preprocessing the face images of each frame based on the keypoints through alignment, cropping, and scaling, several face images are obtained, and the frame number of the face images is normalized. Then, optical flow calculation is performed on each face image to obtain an optical flow image sequence. The optical flow image sequence is input into the PCANet+ network for spatial feature extraction and weighted averaging to obtain a two-dimensional feature map. Feature regions are extracted from the two-dimensional feature map and input into the LSTM network to extract temporal features. The temporal features are weighted and summed to obtain scores for different categories of micro-expressions. Finally, the Softmax function is used to map the scores to the probabilities of different categories of micro-expressions to complete the micro-expression recognition. That is, the PCANet+ network and the LSTM network are combined. The PCANet+ network extracts spatial features, and the LSTM network extracts temporal features, effectively extracting the spatiotemporal features of micro-expressions. Moreover, the PCANet+ network can directly calculate the network parameters of the current layer through the input of the current layer, reducing the network parameters and computational load, avoiding overfitting, and ultimately greatly improving the micro-expression recognition effect.

[0135] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A micro-expression recognition method based on PCANet+ and LSTM, characterized in that: Includes the following steps: Step S10: Acquire a portrait video and perform key point detection on the faces of the head images in each frame of the portrait video; Step S20: Based on the key points, perform preprocessing such as alignment, cropping and scaling on each frame of head image to obtain several face images; Step S30: Normalize the number of frames in the face image; Step S40: Perform optical flow calculation on each of the face images to obtain an optical flow image sequence; Step S50: Input the optical flow image sequence into the PCANet+ network for spatial feature extraction to obtain several feature maps. Then, perform a weighted average of the feature maps to obtain a two-dimensional feature map. Step S60: Based on each of the key points, extract feature regions from the two-dimensional feature map, input each of the feature regions into the LSTM network to extract temporal features, and perform weighted summation on each of the temporal features to obtain scores for different categories of micro-expressions; Step S70: Use the Softmax function to map the scores to the probabilities of different categories of micro-expressions in order to complete the micro-expression recognition; Step S50 specifically involves: The portrait video is divided into K segments, each segment containing T frames of face images. Optical flow calculations are performed on each segment, and the segments are stacked to obtain a set of multi-channel optical flow images. ,in For the multi-channel image corresponding to the i-th segment; then... The optical flow image is sequentially input into a two-layer PCANet+ network. The first PCA convolutional layer has D1 filters of size k1×k1, and the second PCA convolutional layer has D2 filters of size k2×k2. Each convolutional layer is followed by a pooling layer: the first is an average pooling layer with a fixed filter size of 3×3, and the second is a max pooling layer with a filter size of 3×3. (Multi-channel image...) After passing through a two-layer PCANet+ network, we obtain D2 sets of two-dimensional feature images. ; The LSTM network performs a weighted average of the outputs of the second layer of the PCANet+ network to obtain a two-dimensional feature map. .

2. The micro-expression recognition method based on PCANet+ and LSTM as described in claim 1, characterized in that: Step S10 specifically involves: A portrait video is acquired, and 68 key points of the face in each frame of the portrait video are detected using an active shape model.

3. The micro-expression recognition method based on PCANet+ and LSTM as described in claim 1, characterized in that: Step S20 specifically involves: The inner corners of the left and right eyes in each frame of the head image are obtained from the key points. The head images in each frame are rotated and aligned based on the line connecting the inner corners of the left and right eyes. The face regions in the head images are then cropped based on the key points. The cropped face regions are scaled to a uniform size to complete the preprocessing of the head images in each frame, resulting in several face images.

4. The micro-expression recognition method based on PCANet+ and LSTM as described in claim 1, characterized in that: Step S30 specifically involves: After grayscale processing of each frame of the face image, the frame number of the face image is normalized using a time interpolation algorithm.

5. The micro-expression recognition method based on PCANet+ and LSTM as described in claim 1, characterized in that: In step S60, the feature region includes at least the eyebrow region, eye region, mouth region, and nose wing region.

6. A micro-expression recognition system based on PCANet+ and LSTM, characterized in that: Includes the following modules: The key point detection module is used to acquire portrait videos and perform key point detection on the faces of each frame of the portrait video. The head image preprocessing module is used to perform alignment, cropping, and scaling preprocessing on each frame of head image based on the key points to obtain several face images. A frame number normalization module is used to normalize the frame number of the face image; An optical flow calculation module is used to perform optical flow calculations on each of the face images to obtain an optical flow image sequence; The PCANet+ feature extraction module is used to input the optical flow image sequence into the PCANet+ network for spatial feature extraction, obtain several feature maps, and perform a weighted average of the feature maps to obtain a two-dimensional feature map. The LSTM feature extraction module is used to extract feature regions from the two-dimensional feature map based on each key point, input each feature region into the LSTM network to extract temporal features, and perform weighted summation on each temporal feature to obtain scores for different categories of micro-expressions; The score mapping module is used to map the scores to the probabilities of different categories of micro-expressions using the Softmax function, so as to complete the micro-expression recognition; The PCANet+ feature extraction module is specifically used for: The portrait video is divided into K segments, each segment containing T frames of face images. Optical flow calculations are performed on each segment, and the segments are stacked to obtain a set of multi-channel optical flow images. ,in For the multi-channel image corresponding to the i-th segment; then... The optical flow image is sequentially input into a two-layer PCANet+ network. The first PCA convolutional layer has D1 filters of size k1×k1, and the second PCA convolutional layer has D2 filters of size k2×k2. Each convolutional layer is followed by a pooling layer: the first is an average pooling layer with a fixed filter size of 3×3, and the second is a max pooling layer with a filter size of 3×3. (Multi-channel image...) After passing through a two-layer PCANet+ network, we obtain D2 sets of two-dimensional feature images. ; The LSTM network performs a weighted average of the outputs of the second layer of the PCANet+ network to obtain a two-dimensional feature map. .

7. A micro-expression recognition system based on PCANet+ and LSTM as described in claim 6, characterized in that: The key point detection module is specifically used for: A portrait video is acquired, and 68 key points of the face in each frame of the portrait video are detected using an active shape model.

8. A micro-expression recognition system based on PCANet+ and LSTM as described in claim 6, characterized in that: The human head image preprocessing module is specifically used for: The inner corners of the left and right eyes in each frame of the head image are obtained from the key points. The head images in each frame are rotated and aligned based on the line connecting the inner corners of the left and right eyes. The face regions in the head images are then cropped based on the key points. The cropped face regions are scaled to a uniform size to complete the preprocessing of the head images in each frame, resulting in several face images.

9. A micro-expression recognition system based on PCANet+ and LSTM as described in claim 6, characterized in that: The frame number normalization module is specifically used for: After grayscale processing of each frame of the face image, the frame number of the face image is normalized using a time interpolation algorithm.

10. A micro-expression recognition system based on PCANet+ and LSTM as described in claim 6, characterized in that: In the LSTM feature extraction module, the feature regions include at least the eyebrow region, eye region, mouth region, and nose wing region.

Images