A micro-expression recognition method and system based on PCB model

Abstract

The present invention provides a micro-expression recognition method and system based on a PCB model in the field of facial expression recognition technology. The method comprises the following steps: Step S10, acquiring a portrait video and performing key point detection on each frame of a head image in the portrait video; Step S20, pre-processing each head image frame by aligning, cropping, and scaling the head images based on the key points to obtain a plurality of facial images; Step S30, normalizing the number of frames of the facial images; Step S40, performing optical flow calculation on each of the facial images to obtain an optical flow image sequence; and Step S50, inputting the optical flow image sequence into the PCB model for micro-expression recognition. The present invention has the advantage of significantly improving micro-expression recognition performance.

Description

Technical Field

[0001] The present invention relates to the technical field of expression recognition, and in particular to a micro-expression recognition method and system based on a PCB model. Background Art

[0002] With technological advancements, artificial intelligence (AI) is also developing, including facial expression recognition technology. By automatically identifying micro-expressions in videos, it can quickly determine a person's mood and psychological activities. However, traditional micro-expression recognition methods based on deep learning often fail to effectively distinguish the contributions of features from different facial regions to micro-expression recognition, resulting in poor micro-expression recognition results.

[0003] Therefore, how to provide a micro-expression recognition method and system based on the PCB model to improve the micro-expression recognition performance has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a micro-expression recognition method and system based on a PCB model to improve the micro-expression recognition performance.

[0005] In a first aspect, the present invention provides a micro-expression recognition method based on a PCB model, comprising the following steps:

[0006] Step S10: Obtain a portrait video, and perform key point detection on the face of each frame of the head image in the portrait video;

[0007] Step S20: performing pre-processing of aligning, cropping, and scaling each frame of head image based on each key point to obtain a plurality of face images;

[0008] Step S30, normalizing the number of frames of the face image;

[0009] Step S40: performing optical flow calculation on each of the face images to obtain an optical flow image sequence;

[0010] Step S50: inputting the optical flow image sequence into the PCB model for micro-expression recognition.

[0011] Furthermore, the step S10 is specifically as follows:

[0012] A portrait video is obtained, and 68 key points of a face in each frame of a head image in the portrait video are detected using an active shape model.

[0013] Furthermore, the step S20 is specifically as follows:

[0014] The inner corner points of the left eye and the right eye in each frame of the head image are obtained from each of the key points, and the inner corner points of the left eye and the right eye are rotated and aligned based on the line connecting the inner corner points of the left eye and the right eye. Then, the facial area in the head image is cropped based on each of the key points, and the cropped facial areas are scaled to a uniform size to complete the preprocessing of the head image in each frame to obtain a plurality of facial images.

[0015] Furthermore, the step S30 is specifically as follows:

[0016] After grayscale processing is performed on each frame of the facial image, the number of frames of the facial image is normalized using a time interpolation algorithm.

[0017] Furthermore, in step S50, the PCB model is composed of a backbone network, a feature tensor segmentation network, and a classifier;

[0018] The backbone network is constructed based on the PCANet+ network, and is used to extract the spatial features of the optical flow image sequence to obtain a feature tensor; the feature tensor segmentation network is used to segment the feature tensor and input the corresponding classifier, and the end of each branch of the feature tensor segmentation is added to the LSTM network; the classifier is used to train the segmented feature tensor and extract micro-expression features for micro-expression recognition.

[0019] In a second aspect, the present invention provides a micro-expression recognition system based on a PCB model, comprising the following modules:

[0020] A key point detection module is used to obtain a portrait video and perform key point detection on the face of each frame of the head image in the portrait video;

[0021] A head image preprocessing module is used to preprocess each frame of head image by aligning, cropping and scaling based on each key point to obtain a plurality of face images;

[0022] An image frame number normalization module, used to normalize the frame number of the face image;

[0023] an optical flow calculation module, configured to perform optical flow calculation on each of the face images to obtain an optical flow image sequence;

[0024] The micro-expression recognition module is used to input the optical flow image sequence into the PCB model to perform micro-expression recognition.

[0025] Furthermore, the key point detection module is specifically used to:

[0026] A portrait video is obtained, and 68 key points of a face in each frame of a head image in the portrait video are detected using an active shape model.

[0027] Furthermore, the head image preprocessing module is specifically used to:

[0028] The inner corner points of the left eye and the right eye in each frame of the head image are obtained from each of the key points, and the inner corner points of the left eye and the right eye are rotated and aligned based on the line connecting the inner corner points of the left eye and the right eye. Then, the facial area in the head image is cropped based on each of the key points, and the cropped facial areas are scaled to a uniform size to complete the preprocessing of the head image in each frame to obtain a plurality of facial images.

[0029] Furthermore, the image frame number normalization module is specifically used to:

[0030] After grayscale processing is performed on each frame of the facial image, the number of frames of the facial image is normalized using a time interpolation algorithm.

[0031] Furthermore, in the micro-expression recognition module, the PCB model consists of a backbone network, a feature tensor segmentation network, and a classifier;

[0032] The backbone network is constructed based on the PCANet+ network, and is used to extract the spatial features of the optical flow image sequence to obtain a feature tensor; the feature tensor segmentation network is used to segment the feature tensor and input the corresponding classifier, and the end of each branch of the feature tensor segmentation is added to the LSTM network; the classifier is used to train the segmented feature tensor and extract micro-expression features for micro-expression recognition.

[0033] The advantages of the present invention are:

[0034] By detecting key points of the face in each frame of the head image in the acquired portrait video, and pre-processing the head image frame by alignment, cropping and scaling based on the key points, several face images are obtained; then the number of frames of the face image is normalized and optical flow is calculated to obtain an optical flow image sequence, which is then input into the PCB model for micro-expression recognition; because the PCB model combines the PCANet+ network and the LSTM network, spatial features are extracted through the PCANet+ network, and temporal features are extracted through the LSTM network. On the basis of extracting the spatiotemporal features of micro-expressions, the classifier corresponding to each local spatiotemporal feature can be used to learn the classification information of micro-expressions in a more refined manner, ultimately greatly improving the performance of micro-expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0036] Figure 1 The present invention is a flowchart of a micro-expression recognition method based on a PCB model.

[0037] Figure 2 It is a structural schematic diagram of a micro-expression recognition system based on a PCB model of the present invention.

[0038] Figure 3 Schematic diagram of the micro-expression recognition process of the present invention. DETAILED DESCRIPTION

[0039] The technical solution in the embodiments of the present application has the following overall idea: micro-expression recognition is performed on the input optical flow image sequence through the PCB model, and the PCB model combines the PCANet+ network for extracting spatial features and the LSTM network for extracting temporal features. On the basis of extracting the spatiotemporal features of micro-expressions, the classifier corresponding to each local spatiotemporal feature can be used to learn the classification information of micro-expressions in a more refined manner to improve the micro-expression recognition performance.

[0040] Please refer to Figures 1 to 3 As shown, a preferred embodiment of the micro-expression recognition method based on the PCB model of the present invention includes the following steps:

[0041] Step S10: Obtain a portrait video, and perform key point detection on the face of each frame of the head image in the portrait video;

[0042] Step S20: performing pre-processing of aligning, cropping, and scaling each frame of head image based on each key point to obtain a plurality of face images;

[0043] Step S30, normalizing the number of frames of the face image;

[0044] Step S40: performing optical flow calculation on each of the face images to obtain an optical flow image sequence;

[0045] Step S50: inputting the optical flow image sequence into the PCB model for micro-expression recognition.

[0046] The step S10 is specifically as follows:

[0047] A portrait video is obtained, and 68 key points of a face in each frame of a head image in the portrait video are detected using an active shape model.

[0048] The active shape model is established on the basis of the point distribution model. When detecting faces, it comprehensively considers the prior knowledge of the image such as grayscale, size, shape and approximate position. By learning the feature points annotated in the training set, it obtains the statistical model of the distribution of the feature points of the training image samples, and uses this as the initial position. Through continuous iteration, the shape model of the target image is obtained. Finally, the shape constraint is applied to the test set to search for the best matching point, thereby realizing the positioning of the facial feature points (key points).

[0049] The step S20 is specifically as follows:

[0050] The inner corner points of the left eye and the right eye in each frame of the head image are obtained from each of the key points, and the inner corner points of the left eye and the right eye are rotated and aligned based on the line connecting the inner corner points of the left eye and the right eye. Then, the facial area in the head image is cropped based on each of the key points, and the cropped facial areas are scaled to a uniform size to complete the preprocessing of the head image in each frame to obtain a plurality of facial images.

[0051] In order to eliminate the impact of head rotation and offset on micro-expression recognition, it is necessary to perform face alignment on the head image based on the detected key points; since the relative positions of the feature points of the inner corners of the left eye and the right eye in the face are relatively stable and will not change due to facial muscle movement, the present invention performs horizontal alignment based on the line connecting the inner corner points of the left eye and the inner corner points of the right eye, and then crops the face area based on the aligned image and the key points, removing areas such as clothes, background and hair in the head image that are not related to micro-expressions; due to changes in factors such as the rotation of the head in the head image and the distance from the camera, the size of the cropped face image is not completely consistent. In order to adapt to the subsequent network's requirements for input size, the size needs to be unified.

[0052] The step S30 is specifically as follows:

[0053] After grayscale processing is performed on each frame of the facial image, the number of frames of the facial image is normalized using a time interpolation algorithm.

[0054] Because micro-expressions last for different periods of time during the acquisition process and different camera frame rates are used in different datasets, the resulting micro-expression image sequences are of varying lengths. Furthermore, the present invention requires the use of an LSTM network to extract micro-expression features. To facilitate subsequent LSTM network processing, a temporal interpolation algorithm is used during data preprocessing to normalize the number of frames in the micro-expression sample sequence (facial image). This algorithm, first used for lip reading recognition, is a flow-based interpolation method that maps the entire micro-expression sequence into a curve, with each frame in the sequence corresponding to a point on the curve. The curve is then resampled to obtain the interpolated image sequence.

[0055] In step S40, optical flow is a two-dimensional vector field on the image plane that describes the instantaneous speed of pixel motion between two consecutive frames in a video sequence. To improve the effectiveness of PCANet+ network feature learning, the present invention performs dense optical flow calculations on image sequences of micro-expressions (facial images) to enhance facial motion information. Optical flow calculations rely on two basic assumptions: 1. Luminance remains constant, meaning that the brightness of pixels at corresponding positions in the image sequence remains constant during the object's motion and capture; and 2. Temporal continuity, meaning that the motion of target pixels between adjacent frames cannot be excessively dramatic, ensuring that pixel coordinates are continuously differentiable in time.

[0056] The optical flow calculation process is as follows:

[0057] The optical flow method represents an image sequence as a three-dimensional matrix. The brightness of a pixel in the sequence is represented as I(x, y, t), where x, y, and t are their spatiotemporal coordinates. The pixel arrives at the next frame after Δt and completes a displacement of Δx and Δy in the image. Based on the assumption that the brightness remains constant, the intensity of the pixel before and after the movement is constant, resulting in the following formula:

[0058] I(x,y,t)=I(x+Δx,y+Δy,t+Δt)--------------------------(1)

[0059] Based on the assumption of time continuity, Taylor expansion is performed on the right side of formula (1) to obtain the following formula:

[0060]

[0061] Where ε represents a high-order infinitesimal and can be ignored. Substituting formula (2) into formula (1) and dividing by Δt, we obtain the following formula:

[0062]

[0063] Let u and v represent the velocity vector of the pixel on the x-axis and y-axis respectively, that is, u = Δx / Δt, v = Δy / Δt, and substitute them into formula (3) to obtain the following formula:

[0064] I x u+I y v+I t =0--------------------------------------(4)

[0065] (u, v) is the optical flow field generated by the pixel in the time Δt, which can be solved by adding constraints. Adding different constraints can obtain different optical flow field calculation methods. This paper applies the TV-L1 algorithm to the calculation of the optical flow field. The TV-L1 algorithm introduces a subspace trajectory model to ensure the temporal consistency of the optical flow while retaining the edge features in the image. For a pixel point in the micro-expression sequence, the optical flow estimation loss function shown in formula (5) is first used to calculate the continuous optical flow field:

[0066]

[0067] Where L represents the length of the micro-expression image sequence; represents the R-basis trajectory used to construct the trajectory space; Represents the spatial domain of the image; lin: represents a mapping function that can map the optical flow field u(t), v(t) to a new space constructed by the R basis trajectory; the first term of formula (5) represents the penalty term for the brightness constrained constrain, the second term is used to make the derived optical flow located on the basis trajectory, and the third term represents the spatial regularization of the trajectory model coefficient based on the total variation.

[0068] Suppose a micro-expression image sequence is The first frame is set as the reference frame, and the horizontal and vertical optical flow components U of all the remaining frames are calculated according to the above optical flow calculation method. Then the calculated optical flow sequence is stacked in frame units and input into the PCANet+ network; for the optical flow sequence U, Use a sliding window of size T and step size s to sample it, and get two subsequence sets μ and ν, which are defined as formula (6):

[0069]

[0070] in, The elements at corresponding positions in μ and ν are connected by formula (6) to form an input sequence Γ consisting of stacked optical flow components:

[0071]

[0072] Among them, || represents the connection of optical flow components in the channel dimension. Through the multi-channel stacking operation, the horizontal and vertical optical flow sequences of each video clip are stacked in T frames to obtain an optical flow image sequence with a channel number of 2T.

[0073] In step S50, the PCB model is composed of a backbone network, a feature tensor segmentation network, and a classifier;

[0074] The backbone network is constructed based on the PCANet+ network, and is used to extract the spatial features of the optical flow image sequence to obtain a feature tensor, that is, to obtain information that is conducive to classification or prediction; the feature tensor segmentation network is used to segment the feature tensor and input the corresponding classifier, and the end of each branch of the feature tensor segmentation is added to the LSTM network; the classifier is used to train the segmented feature tensor and extract micro-expression features for micro-expression recognition.

[0075] like Figure 3 As shown, each head image (micro-expression image sequence) undergoes preprocessing and optical flow calculation to generate an optical flow image sequence. This optical flow image sequence is then stacked and fed into a backbone network consisting of PCA convolutional layers for feature extraction. The resulting output is a three-dimensional feature tensor T, whose number of channels is equal to the number of filters in the PCANet + last convolutional layer. During the training phase of the local feature-based classifier, the original PCB model evenly partitions the three-dimensional feature tensor T into multiple sub-blocks along the channel axis and then trains a classifier based on the features of each sub-block. Due to the sparse nature of micro-expression facial motion, features with classification information are primarily concentrated in key facial regions, such as the eyebrows, eyes, nose, and mouth. To leverage this characteristic, during the feature segmentation phase, p multi-channel features are generated from the output feature tensor T based on key facial regions. These multi-channel features are then weighted and summed to obtain the corresponding two-dimensional feature matrix. Each 2D feature matrix is then expanded into a feature vector gi, which serves as the input to the subsequent local feature-based classifier.

[0076] To better adapt the PCB model to video classification tasks, each branch's classifier consists of a two-layer LSTM network structure, a fully connected layer, and a Softmax function. Each classifier is trained based on its corresponding local features. Finally, the inputs of each branch's fully connected layer are concatenated into a complete vector and fed into a global classifier consisting of two fully connected layers and a Softmax function to complete micro-expression classification. To rationally utilize the influence of different regions on micro-expression recognition, weights are added to the features of each branch. The weight calculation process is as follows:

[0077] 1. Divide the training set into training subset and validation subset;

[0078] 2. Use the training subset to train the PCB model until convergence;

[0079] 3. Use the validation subset to test the classification effect of each branch; at the same time, calculate the weight of the features output by each branch in micro-expression recognition based on the accuracy of the classifier or the prediction score of each sample. The calculation formulas for the two weights are as follows:

[0080]

[0081]

[0082] Among them, w i and w i 'represents the weight of the i-th branch calculated according to the accuracy and score; p represents the number of branches; accuracy(i) represents the accuracy of the validation subset in the i-th branch; n represents the number of samples in the validation subset; c represents the total number of micro-expression categories in the dataset; score j (k) represents the score of the classifier predicting that the jth sample belongs to the kth category, and (k) represents the true category corresponding to the jth sample. In the final classification, the feature vectors output by each part-level classifier are multiplied by the corresponding weights and concatenated before being input into the classification network. This allows the classification network to better utilize the refined information to classify micro-expressions.

[0083] A preferred embodiment of the micro-expression recognition system based on a PCB model of the present invention includes the following modules:

[0084] A key point detection module is used to obtain a portrait video and perform key point detection on the face of each frame of the head image in the portrait video;

[0085] A head image preprocessing module is used to preprocess each frame of head image by aligning, cropping and scaling based on each key point to obtain a plurality of face images;

[0086] An image frame number normalization module, used to normalize the frame number of the face image;

[0087] an optical flow calculation module, configured to perform optical flow calculation on each of the face images to obtain an optical flow image sequence;

[0088] The micro-expression recognition module is used to input the optical flow image sequence into the PCB model to perform micro-expression recognition.

[0089] The key point detection module is specifically used for:

[0090] A portrait video is obtained, and 68 key points of a face in each frame of a head image in the portrait video are detected using an active shape model.

[0091] The active shape model is established on the basis of the point distribution model. When detecting faces, it comprehensively considers the prior knowledge of the image such as grayscale, size, shape and approximate position. By learning the feature points annotated in the training set, it obtains the statistical model of the distribution of the feature points of the training image samples, and uses this as the initial position. Through continuous iteration, the shape model of the target image is obtained. Finally, the shape constraint is applied to the test set to search for the best matching point, thereby realizing the positioning of the facial feature points (key points).

[0092] The head image preprocessing module is specifically used for:

[0093] The inner corner points of the left eye and the right eye in each frame of the head image are obtained from each of the key points, and the inner corner points of the left eye and the right eye are rotated and aligned based on the line connecting the inner corner points of the left eye and the right eye. Then, the facial area in the head image is cropped based on each of the key points, and the cropped facial areas are scaled to a uniform size to complete the preprocessing of the head image in each frame to obtain a plurality of facial images.

[0094] In order to eliminate the impact of head rotation and offset on micro-expression recognition, it is necessary to perform face alignment on the head image based on the detected key points; since the relative positions of the feature points of the inner corners of the left eye and the right eye in the face are relatively stable and will not change due to facial muscle movement, the present invention performs horizontal alignment based on the line connecting the inner corner points of the left eye and the inner corner points of the right eye, and then crops the face area based on the aligned image and the key points, removing areas such as clothes, background and hair in the head image that are not related to micro-expressions; due to changes in factors such as the rotation of the head in the head image and the distance from the camera, the size of the cropped face image is not completely consistent. In order to adapt to the subsequent network's requirements for input size, the size needs to be unified.

[0095] The image frame number normalization module is specifically used for:

[0096] After grayscale processing is performed on each frame of the facial image, the number of frames of the facial image is normalized using a time interpolation algorithm.

[0097] Because micro-expressions last for different periods of time during the acquisition process and different camera frame rates are used in different datasets, the resulting micro-expression image sequences are of varying lengths. Furthermore, the present invention requires the use of an LSTM network to extract micro-expression features. To facilitate subsequent LSTM network processing, a temporal interpolation algorithm is used during data preprocessing to normalize the number of frames in the micro-expression sample sequence (facial image). This algorithm, first used for lip reading recognition, is a flow-based interpolation method that maps the entire micro-expression sequence into a curve, with each frame in the sequence corresponding to a point on the curve. The curve is then resampled to obtain the interpolated image sequence.

[0098] In the optical flow calculation module, optical flow is a two-dimensional vector field on the image plane that describes the instantaneous speed of pixel motion between two consecutive frames in a video sequence. To improve the effectiveness of PCANet+ network feature learning, the present invention performs dense optical flow calculation on image sequences of micro-expressions (face images) to enhance facial motion information. Optical flow calculation relies on two basic assumptions: 1. The brightness remains constant, that is, the brightness of the corresponding pixels in the image sequence remains unchanged during the object's motion and shooting process; 2. Temporal continuity, that is, the movement of the target pixel between adjacent frames cannot be too drastic, so as to ensure that the pixel coordinates are continuously differentiable in time.

[0099] The optical flow calculation process is as follows:

[0100] The optical flow method represents an image sequence as a three-dimensional matrix. The brightness of a pixel in the sequence is represented as I(x, y, t), where x, y, and t are their spatiotemporal coordinates. The pixel arrives at the next frame after Δt and completes a displacement of Δx and Δy in the image. Based on the assumption that the brightness remains constant, the intensity of the pixel before and after the movement is constant, resulting in the following formula:

[0101] I(x,y,t)=I(x+Δx,y+Δy,t+Δt)--------------------------(1)

[0102] Based on the assumption of time continuity, Taylor expansion is performed on the right side of formula (1) to obtain the following formula:

[0103]

[0104] Where ε represents a high-order infinitesimal and can be ignored. Substituting formula (2) into formula (1) and dividing by Δt, we obtain the following formula:

[0105]

[0106] Let u and v represent the velocity vector of the pixel on the x-axis and y-axis respectively, that is, u = Δx / Δt, v = Δy / Δt, and substitute them into formula (3) to obtain the following formula:

[0107] I x u+I y v+I t =0--------------------------------------(4)

[0108] (u, v) is the optical flow field generated by the pixel in the time Δt, which can be solved by adding constraints. Adding different constraints can obtain different optical flow field calculation methods. This paper applies the TV-L1 algorithm to the calculation of the optical flow field. The TV-L1 algorithm introduces a subspace trajectory model to ensure the temporal consistency of the optical flow while retaining the edge features in the image. For a pixel point in the micro-expression sequence, the optical flow estimation loss function shown in formula (5) is first used to calculate the continuous optical flow field:

[0109]

[0110] Where L represents the length of the micro-expression image sequence; represents the R-basis trajectory used to construct the trajectory space; Represents the spatial domain of the image; lin: represents a mapping function that can map the optical flow field u(t), v(t) to a new space constructed by the R basis trajectory; the first term of formula (5) represents the penalty term for the brightness constrained constrain, the second term is used to make the derived optical flow located on the basis trajectory, and the third term represents the spatial regularization of the trajectory model coefficient based on the total variation.

[0111] Suppose a micro-expression image sequence is The first frame is set as the reference frame, and the horizontal and vertical optical flow components U of all the remaining frames are calculated according to the above optical flow calculation method. Then the calculated optical flow sequence is stacked in frame units and input into the PCANet+ network; for the optical flow sequence U, Use a sliding window of size T and step size s to sample it, and get two subsequence sets μ and ν, which are defined as formula (6):

[0112]

[0113] in, The elements at corresponding positions in μ and ν are connected by formula (6) to form an input sequence Γ consisting of stacked optical flow components:

[0114]

[0115] Among them, || represents the connection of optical flow components in the channel dimension. Through the multi-channel stacking operation, the horizontal and vertical optical flow sequences of each video clip are stacked in T frames to obtain an optical flow image sequence with a channel number of 2T.

[0116] In the micro-expression recognition module, the PCB model consists of a backbone network, a feature tensor segmentation network, and a classifier;

[0117] The backbone network is constructed based on the PCANet+ network, and is used to extract the spatial features of the optical flow image sequence to obtain a feature tensor, that is, to obtain information that is conducive to classification or prediction; the feature tensor segmentation network is used to segment the feature tensor and input the corresponding classifier, and the end of each branch of the feature tensor segmentation is added to the LSTM network; the classifier is used to train the segmented feature tensor and extract micro-expression features for micro-expression recognition.

[0118] like Figure 3 As shown, each head image (micro-expression image sequence) undergoes preprocessing and optical flow calculation to generate an optical flow image sequence. This optical flow image sequence is then stacked and fed into a backbone network consisting of PCA convolutional layers for feature extraction. The resulting output is a three-dimensional feature tensor T, whose number of channels is equal to the number of filters in the PCANet + last convolutional layer. During the training phase of the local feature-based classifier, the original PCB model evenly partitions the three-dimensional feature tensor T into multiple sub-blocks along the channel axis and then trains a classifier based on the features of each sub-block. Due to the sparse nature of micro-expression facial motion, features with classification information are primarily concentrated in key facial regions, such as the eyebrows, eyes, nose, and mouth. To leverage this characteristic, during the feature segmentation phase, p multi-channel features are generated from the output feature tensor T based on key facial regions. These multi-channel features are then weighted and summed to obtain the corresponding two-dimensional feature matrix. Each 2D feature matrix is then expanded into a feature vector gi, which serves as the input to the subsequent local feature-based classifier.

[0119] To better adapt the PCB model to video classification tasks, each branch's classifier consists of a two-layer LSTM network structure, a fully connected layer, and a Softmax function. Each classifier is trained based on its corresponding local features. Finally, the inputs of each branch's fully connected layer are concatenated into a complete vector and fed into a global classifier consisting of two fully connected layers and a Softmax function to complete micro-expression classification. To rationally utilize the influence of different regions on micro-expression recognition, weights are added to the features of each branch. The weight calculation process is as follows:

[0120] 1. Divide the training set into training subset and validation subset;

[0121] 2. Use the training subset to train the PCB model until convergence;

[0122] 3. Use the validation subset to test the classification effect of each branch; at the same time, calculate the weight of the features output by each branch in micro-expression recognition based on the accuracy of the classifier or the prediction score of each sample. The calculation formulas for the two weights are as follows:

[0123]

[0124]

[0125] Among them, w i and w i 'represents the weight of the i-th branch calculated according to the accuracy and score; p represents the number of branches; accuracy(i) represents the accuracy of the validation subset in the i-th branch; n represents the number of samples in the validation subset; c represents the total number of micro-expression categories in the dataset; score j (k) represents the score of the classifier predicting that the jth sample belongs to the kth category, and (k) represents the true category corresponding to the jth sample. In the final classification, the feature vectors output by each part-level classifier are multiplied by the corresponding weights and concatenated before being input into the classification network. This allows the classification network to better utilize the refined information to classify micro-expressions.

[0126] In summary, the advantages of the present invention are:

[0127] By detecting key points of the face in each frame of the head image in the acquired portrait video, and pre-processing the head image frame by alignment, cropping and scaling based on the key points, several face images are obtained; then the number of frames of the face image is normalized and optical flow is calculated to obtain an optical flow image sequence, which is then input into the PCB model for micro-expression recognition; because the PCB model combines the PCANet+ network and the LSTM network, spatial features are extracted through the PCANet+ network, and temporal features are extracted through the LSTM network. On the basis of extracting the spatiotemporal features of micro-expressions, the classifier corresponding to each local spatiotemporal feature can be used to learn the classification information of micro-expressions in a more refined manner, ultimately greatly improving the performance of micro-expression recognition.

[0128] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A micro-expression recognition method based on a PCB model, characterized by: The steps include: Step S10: Obtain a portrait video, and perform key point detection on the face of each frame of the head image in the portrait video; Step S20: performing pre-processing of aligning, cropping, and scaling each frame of head image based on each key point to obtain a plurality of face images; Step S30, normalizing the number of frames of the face image; Step S40: performing optical flow calculation on each of the face images to obtain an optical flow image sequence; Step S50: inputting the optical flow image sequence into a PCB model for micro-expression recognition; The PCB model consists of a backbone network, a feature tensor segmentation network, and a classifier; The backbone network is constructed based on the PCANet+ network and is used to extract the spatial features of the optical flow image sequence to obtain a feature tensor; The feature tensor segmentation network is used to segment the feature tensor and input the corresponding classifier, and each branch end of the feature tensor segmentation is added to the LSTM network; the classifier is used to train the segmented feature tensor and extract micro-expression features to perform micro-expression recognition; Each frame of the head image is preprocessed and subjected to optical flow calculation to obtain an optical flow image sequence. The optical flow image sequence is stacked and input into a backbone network composed of PCA convolutional layers to extract features, outputting a three-dimensional feature tensor T, whose number of channels is the number of filters in the last convolutional layer of the PCANet+ network. In the feature segmentation stage, p multi-channel features are divided from the output feature tensor T according to the key facial areas, and then the multi-channel features are weighted and summed to obtain the corresponding two-dimensional feature matrix. Each two-dimensional feature matrix is then expanded into a feature vector gi, which serves as the input of the subsequent local feature-based classifier. The classifier of each branch of the PCB model consists of a two-layer LSTM network structure, a fully connected layer and a Softmax function. Each classifier is trained with its corresponding local features. Finally, the input of the fully connected layer of each branch is connected into a complete vector and input into a global classifier consisting of two fully connected layers and a Softmax function to complete the final micro-expression classification.

2. A micro-expression recognition method based on a PCB model as claimed in claim 1, characterized in that: The step S10 is specifically as follows: A portrait video is obtained, and 68 key points of a face in each frame of a head image in the portrait video are detected using an active shape model.

3. A micro-expression recognition method based on a PCB model as claimed in claim 1, characterized in that: The step S20 is specifically as follows: The inner corner points of the left eye and the right eye in each frame of the head image are obtained from each of the key points, and the inner corner points of the left eye and the right eye are rotated and aligned based on the line connecting the inner corner points of the left eye and the right eye. Then, the facial area in the head image is cropped based on each of the key points, and the cropped facial areas are scaled to a uniform size to complete the preprocessing of the head image in each frame to obtain a plurality of facial images.

4. A micro-expression recognition method based on a PCB model as claimed in claim 1, characterized in that: The step S30 is specifically as follows: After grayscale processing is performed on each frame of the facial image, the number of frames of the facial image is normalized using a time interpolation algorithm.

5. A micro-expression recognition system based on a PCB model, characterized by: Includes the following modules: A key point detection module is used to obtain a portrait video and perform key point detection on the face of each frame of the head image in the portrait video; A head image preprocessing module is used to preprocess each frame of head image by aligning, cropping and scaling based on each key point to obtain a plurality of face images; An image frame number normalization module, used to normalize the frame number of the face image; an optical flow calculation module, configured to perform optical flow calculation on each of the face images to obtain an optical flow image sequence; A micro-expression recognition module, configured to input the optical flow image sequence into a PCB model for micro-expression recognition; The PCB model consists of a backbone network, a feature tensor segmentation network, and a classifier; The backbone network is constructed based on the PCANet+ network and is used to extract the spatial features of the optical flow image sequence to obtain a feature tensor; The feature tensor segmentation network is used to segment the feature tensor and input the corresponding classifier, and each branch end of the feature tensor segmentation is added to the LSTM network; the classifier is used to train the segmented feature tensor and extract micro-expression features to perform micro-expression recognition; Each frame of the head image is preprocessed and subjected to optical flow calculation to obtain an optical flow image sequence. The optical flow image sequence is stacked and input into a backbone network composed of PCA convolutional layers to extract features, outputting a three-dimensional feature tensor T, whose number of channels is the number of filters in the last convolutional layer of the PCANet+ network. In the feature segmentation stage, p multi-channel features are divided from the output feature tensor T according to the key facial areas, and then the multi-channel features are weighted and summed to obtain the corresponding two-dimensional feature matrix. Each two-dimensional feature matrix is then expanded into a feature vector gi, which serves as the input of the subsequent local feature-based classifier. The classifier of each branch of the PCB model consists of a two-layer LSTM network structure, a fully connected layer and a Softmax function. Each classifier is trained with its corresponding local features. Finally, the input of the fully connected layer of each branch is connected into a complete vector and input into a global classifier consisting of two fully connected layers and a Softmax function to complete the final micro-expression classification.

6. The micro-expression recognition system based on a PCB model as claimed in claim 5, characterized in that: The key point detection module is specifically used for: A portrait video is obtained, and 68 key points of a face in each frame of a head image in the portrait video are detected using an active shape model.

7. The micro-expression recognition system based on a PCB model as claimed in claim 5, characterized in that: The head image preprocessing module is specifically used for: The inner corner points of the left eye and the right eye in each frame of the head image are obtained from each of the key points, and the inner corner points of the left eye and the right eye are rotated and aligned based on the line connecting the inner corner points of the left eye and the right eye. Then, the facial area in the head image is cropped based on each of the key points, and the cropped facial areas are scaled to a uniform size to complete the preprocessing of the head image in each frame to obtain a plurality of facial images.

8. The micro-expression recognition system based on a PCB model as claimed in claim 5, characterized in that: The image frame number normalization module is specifically used for: After grayscale processing is performed on each frame of the facial image, the number of frames of the facial image is normalized using a time interpolation algorithm.

Images