A micro-expression recognition method based on frequency domain features

Abstract

The application discloses a micro-expression recognition method based on frequency domain features, comprising the following steps: extracting a starting frame and a vertex frame in a micro-expression sequence, and taking the vertex frame as a selected frame; calculating a dense optical flow sequence according to the starting frame and the selected frame of the micro-expression sequence; pre-processing optical flow images in the dense optical flow sequence; taking the pre-processed optical flow images as input, training a PDTNet model; inputting a to-be-tested image into the trained PDTNet model to obtain a micro-expression recognition result. The application fuses phase features and amplitude features in the frequency domain features to obtain fusion features with global information, which are used for PDTNet model training, and further improve the accuracy of the PDTNet model recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer image processing technology, specifically relating to a micro-expression recognition method based on frequency domain features. Background Technology

[0002] As early as 1966, Haggard and Isaacs first discovered the existence of microexpressions. Then, Ekman and Friesen named them microexpressions based on the painful expression hidden under the patient's smile captured in a video of a psychiatrist talking to a patient with depression in 1969. The duration of microexpressions ranges from 1 / 25 to 1 / 5 of a second. They are very short and difficult to be captured by the human eye, but they are spontaneous and uncontrollable, and can more accurately reflect a person's true emotions, with potential applications.

[0003] Therefore, research on micro-expression recognition is necessary. Currently, micro-expressions have wide applications in educational psychology, criminal investigation, clinical diagnosis, and sentiment analysis. However, micro-expression recognition remains a challenge. With the development of artificial intelligence networks, researchers have used deep learning convolutional neural networks for micro-expression recognition. However, deep learning convolutional neural network operators have the problem of limited local receptive fields. To obtain global information, multiple layers need to be stacked, but as the number of layers increases, the amount of information will deplete, and the extracted features will focus on certain areas, causing some features to lose global information, resulting in reduced recognition accuracy. Summary of the Invention

[0004] To address the problem of lost global information in existing technologies, this invention proposes a micro-expression recognition method based on frequency domain features to impart global information to the features and improve the recognition accuracy of the model.

[0005] The technical solution adopted in this invention is: a micro-expression recognition method based on frequency domain features, comprising the following steps:

[0006] Step S1: Extract the start frame and vertex frame from the micro-expression sequence, and denote the vertex frame as the selected frame;

[0007] Step S2: Calculate the dense optical flow sequence based on the starting frame and selected frame of the micro-expression sequence;

[0008] Step S3: Preprocess the optical flow images in the dense optical flow sequence;

[0009] Step S4: Use the preprocessed optical flow image as input to train the PDTNet model;

[0010] Step S5: Input the image to be tested into the trained PDTNet model to obtain the micro-expression recognition results.

[0011] Preferably, step S1 further includes the step of: extracting image frames within a preset number of frames on both sides of the vertex frame in the micro-expression sequence, and denoting them as selected frames.

[0012] Preferably, in step S2, the calculation of the optical flow sequence includes the following steps: using the Gunnar Farneback method to calculate and obtain a global dense optical flow sequence.

[0013] Preferably, step S3, the preprocessing of the optical flow image in the dense optical flow sequence, includes the following steps:

[0014] Standardize the size of optical flow images;

[0015] Perform random radian rotation on the optical flow image;

[0016] Randomly flip the image horizontally;

[0017] Adjust the size of the optical flow image and train the optical flow image according to the preset network.

[0018] Preferably, in step S4, training the PDTNet model using the preprocessed optical flow image as input includes the following steps:

[0019] Step S4.1: Construct and initialize the PDTNet model;

[0020] Step S4.2: Divide the optical flow image into n tokens, denoted as optical flow image X = [x1, x2, ..., x...]. n ], x n For the nth token;

[0021] Step S4.3: Extract the amplitude feature |X| and phase feature θ of the optical flow image;

[0022] Step S4.4: Combine the amplitude feature |X| and the phase feature θ to obtain the combined feature Z;

[0023] Step S4.5: Add the fused feature Z to the original image of the optical flow image to obtain a new fused feature. Use the PDT module to further extract features to obtain classification features, and classify the classification features to obtain the corresponding predicted labels.

[0024] Step S4.6: Perform multiple rounds of classification training on the optical flow image to obtain the trained PDTNet model.

[0025] Preferably, in step S4.3, the extraction of the phase feature θ of the optical flow image includes the steps of: passing the optical flow image X through the parameter matrix W. q W k and W v Convolution yields the corresponding three vectors q, k, and v;

[0026] The result of multiplying q by k is transposed using a transpose function and then combined with sine and cosine position codes to obtain the similarity score a_s;

[0027] The similarity score a_s is calculated using softmax to obtain the normalized similarity ratio a_r; the similarity ratio a_r is multiplied by v to obtain the phase feature θ.

[0028] Preferably, in step S4.4, obtaining the fused feature Z by fusing the amplitude feature |X| and the phase feature θ includes the following steps:

[0029] The fusion feature z of the token in the optical flow image is calculated based on the preset fusion feature formula. j ;

[0030] The fusion feature z of each token j Z constitutes the fusion feature of phase and amplitude in the optical flow image.

[0031] Preferably, the preset fusion feature formula is:

[0032] z j =|x j |⊙cosθ j +|x j |⊙i sinθ j j∈(1,2,...,n.),

[0033] Where i is the imaginary unit, |·| is the absolute value operation, ⊙ is element-wise multiplication, and |x j | and θ j respectively

[0034] This represents the amplitude and phase characteristics of the j-th token.

[0035] Preferably, in step S4.5, obtaining new fusion features, further extracting features using the PDT module to obtain classification features, and classifying the classification features to obtain corresponding predicted labels includes the following steps:

[0036] The new fused features are sequentially convolved and pooled through the first ResNet module, the second ResNet module, and the ViTTransformer module to obtain classification features;

[0037] Input the classification features into the MLP layer module for classification and obtain the predicted labels of the classification features.

[0038] The beneficial effects of this invention are as follows: This invention extracts frequency domain information from optical flow images to obtain amplitude and phase features, fuses the amplitude and phase features of the optical flow images to obtain fused features, which are used to preserve global information. Furthermore, the fused frequency domain features are combined with the original image to obtain more informative fused features, and these fused features are added to the PDTNet model for training, improving the recognition accuracy of the PDTNet model. Attached Figure Description

[0039] Figure 1 is a flowchart of a micro-expression recognition method based on frequency domain features according to the present invention.

[0040] Figure 2 is a flowchart of the process of using a preprocessed optical flow image as input to train a PDTNet model in a micro-expression recognition method based on frequency domain features according to the present invention.

[0041] Figure 3 is a flowchart of a micro-expression recognition method based on frequency domain features in this invention for extracting phase features θ from optical flow images.

[0042] Figure 4 is a flowchart of the process of fusing amplitude feature |X| and phase feature θ to obtain fused feature Z in a micro-expression recognition method based on frequency domain features in this invention.

[0043] Figure 5 is a structural diagram of the classification module—PDT module—in the PDTNet model of the present invention;

[0044] Figure 6 is a structural diagram of the ResNet module in the PDTNet model of the present invention;

[0045] Figure 7 is a structural diagram of the ViT Transformer module in the PDTNet model of the present invention;

[0046] Figure 8 is a structural diagram of the Dense layer in the PDTNet model of this invention. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more explicit definition of the scope of protection of the present invention.

[0048] Referring to Figure 1, this embodiment uses the deep learning model PDTNet proposed in this invention as the image feature classification model. By fusing the amplitude and phase features of the optical flow image, a fused feature is obtained and used for PDTNet model classification training to improve the recognition accuracy of the PDTNet model.

[0049] The technical solution adopted in this embodiment includes the following steps:

[0050] A micro-expression recognition method based on frequency domain features includes the following steps:

[0051] Step S1: Extract the start frame and vertex frame from the micro-expression sequence, and denote the vertex frame as the selected frame.

[0052] Furthermore, image frames within a preset number of frames on both sides of the vertex frame in the micro-expression sequence are extracted and denoted as selected frames.

[0053] Existing micro-expression datasets still suffer from insufficient data and uneven data distribution. To address the data insufficiency issue, this embodiment adds the two frames before and after the vertex frame to the dataset to expand the data. To make training more stable, this embodiment balances the sample distribution of the training set by controlling the number of expanded samples. Specifically, let n... sur ,n neg ,n pos These represent the number of additional samples needed in the surprise, negative, and positive categories, respectively, wn. sur wn neg wn pos These represent the total number of samples in the surprise, negative, and positive classes, respectively. Since the surprise class has the fewest samples, all four additional frames of it are added to the training set by default, therefore n... sur =4, then:

[0054] n neg =a*wn sur ÷wn neg -1

[0055] n pos =a*wn sur ÷wn pos -1

[0056] Where a is 5, because the maximum difference in the number of samples in different categories is 5 times.

[0057] Step S2: Calculate the dense optical flow sequence based on the starting frame and selected frame of the micro-expression sequence;

[0058] In step S2, the calculation of the dense optical flow sequence includes the following steps: using the Gunnar Farneback method to calculate and obtain the global dense optical flow sequence;

[0059] The Gunnar Farneback optical flow is calculated between the selected frame and the starting frame. Specifically, the vertex frame refers to the frame in a micro-expression video sequence where the micro-expression is most intense; the starting frame refers to the frame where the micro-expression begins to appear. The Gunnar Farneback optical flow method calculates a global dense optical flow sequence, which is a motion estimation algorithm based on all pixels in two frames, and its performance is better than the sparse optical flow algorithm.

[0060] Step S3: Preprocess the optical flow images in the dense optical flow sequence;

[0061] In step S3, the preprocessing of the optical flow images in the dense optical flow sequence includes the following steps: unifying the size of the optical flow images. Since the original input micro-expression sequence images are of inconsistent sizes, the calculated optical flow images are also of inconsistent sizes. In this embodiment, their sizes are normalized to the same size of 234×240.

[0062] The optical flow image is rotated by random radians to increase image diversity;

[0063] To prevent overfitting, the image is randomly horizontally flipped.

[0064] Adjust the size of the optical flow image to a uniform size of 224×224, and train the optical flow image according to the preset network.

[0065] Step S4: Use the preprocessed optical flow image as input to train the PDTNet model;

[0066] Referring to Figure 2, step S4, using the preprocessed optical flow image as input, trains the PDTNet model, including the following steps:

[0067] Step S4.1: Construct and initialize the PDTNet model;

[0068] The PDTNet network model includes an amplitude feature extractor (which uses two fully connected layers (FC) to extract amplitude features), a phase feature extractor (G module), a feature fusion module (Merging module), and a deep learning network classifier (PDT module). The parameters W of the G module are respectively... q W k W v Perform initialization;

[0069] The PDTNet network employs an end-to-end training approach, using the CrossEntropyLoss function and the Dense Relative Localization Loss function as a joint loss function.

[0070] In the cross-entropy function, m represents the classification state of the current label, K represents the total number of classifications, y represents the label value, and p represents the classification probability of the current label value. The formula for the cross-entropy function is as follows:

[0071]

[0072] Dense Relative Localization Loss(L drloc ) is a loss function proposed for few-shot Transformers, where B is a mini-batch of n input images, and e b,c and e p,h These are multiple pairs of randomly selected tokens, (g u ,g r ) T It is a two-dimensional normalized target transformation bias, representing the actual distance between two tokens in a pair of tokens, (d u ,d r ) T This represents the predicted transformation bias, i.e., the distance between a predicted pair of tokens. The prediction network is an MLP network. The formula for Dense Relative Localization Loss is as follows, where x represents each sample in B, E represents the expected value, and the |·|1 operation represents the L1 norm:

[0073]

[0074] The last layer of the deep learning network classifier PDT module uses the Sigmoid function as its activation function, where t represents the feature, and its formula is as follows:

[0075]

[0076] The activation function used in the other layers is the ReLU function, where t represents the feature, and its formula is as follows:

[0077] ReLU(t) = max(0,t)

[0078] The network uses the Adam optimizer, with a batch size of 32, a learning rate of 0.0001, a learning rate decrease rate of 0.999, and 100 epochs.

[0079] Step S4.2: Divide the optical flow image into n tokens, denoted as optical flow image X = [x1, x2, ..., x...]. n ], x n For the nth token;

[0080] First, the image is divided into n tokens, denoted as X = [x1, x2, ..., xn]. nEach token is 7×7 in size. The implementation uses a 7×7 kernel sliding window with a stride of 4, padding of 4, and 64 output channels. After convolution, the BatchNorma2d function is used for data pooling. Setting the stride to 4, which is smaller than the token size, allows for information overlap between tokens, increasing the model's sensitivity to global information.

[0081] Step S4.3: Extract the amplitude feature |X| and phase feature θ of the optical flow image;

[0082] Referring to Figure 3, the phase feature extraction step of module G, specifically step S4.3, involves extracting the phase feature θ of the optical flow image, which includes the following steps:

[0083] The optical flow image X is processed by the parameter matrix W. q W k and W v Convolution yields the corresponding three vectors q, k, and v;

[0084] The result of multiplying q by k is transposed using a transpose function and then combined with sine and cosine positional codes to obtain the similarity score a_s; the similarity score a_s is then processed by softmax to obtain the normalized similarity ratio a_r;

[0085] Multiplying the similarity ratio a_r by v yields the phase characteristic θ.

[0086] Amplitude information is obtained through two fully connected layers. Specifically, X is passed through a convolution operation with 64 output channels and a kernel size of 1×1 to obtain the amplitude feature |X|.

[0087] Referring to Figure 4, the Merging module is used to fuse the amplitude feature |X| and the phase feature θ, that is, to achieve step S4.4: fusing the amplitude feature |X| and the phase feature θ to obtain the fused feature Z;

[0088] In step S4.4, fusing the amplitude feature |X| and the phase feature θ to obtain the fused feature Z includes the following steps:

[0089] The fusion feature z of the token in the optical flow image is calculated based on the preset fusion feature formula. j ;

[0090] Let z be the fusion feature of the j-th token. j The formula is expressed as:

[0091]

[0092] Clearly, the extracted features reside in the frequency domain. To map them to the proposed network, we expand them using Euler's formula: The fusion feature formula expressed in trigonometric functions is as follows:

[0093]

[0094] Where i is the imaginary unit, and |·| is the absolute value operation. It is element-wise multiplication, |x j | and θ j These represent the amplitude and phase characteristics of the j-th token, respectively.

[0095] The fusion feature z of each token j Z constitutes the fusion feature of phase and amplitude in the optical flow image.

[0096] The fused feature Z is then passed through an output channel 64, a 1×1 kernel convolution operation, and a Dropout operation to prevent the network from overfitting.

[0097] Step S4.5: Add the fused feature Z to the original image of the optical flow image to obtain a new fused feature. Use the PDT module to further extract features to obtain classification features, and classify the classification features to obtain the corresponding predicted labels.

[0098] Since the output channel and input channel are the same in the above steps, and 1×1 convolution operations are used, the size of the resulting feature map is the same as the size of the original input image. By adding the obtained joint features to the original input image, the new fused features contain both spatial and frequency domain features, making the information contained richer.

[0099] The new fused features are sequentially convolved and pooled through the first ResNet module, the second ResNet module, and the ViTTransformer module to obtain classification features;

[0100] Input the classification features into the MLP layer module for classification and obtain the predicted labels of the classification features.

[0101] Referring to Figures 5 to 7, the first ResNet module is structured as follows:

[0102] In the first ResNet module, the input is the new fused feature. First, it passes through a convolutional layer with 64 input channels, 18 output channels, a 3×3 kernel, and padding of 1. Then, it passes through a BatchNorm2d function (batch normalization) and a ReLU activation function with 18 output channels. Next, it passes through another convolutional layer with 18 output channels, a 3×3 kernel, and padding of 1, followed by another BatchNorm2d function and a ReLU activation function with 18 output channels. The shorted portion of the ResNet module passes the initial input Z through a convolutional layer with 64 input channels, 18 output channels, and a 1×1 kernel. The sum of the above two parts then passes through a max-pooling layer with a 2×2 kernel and a stride of 2.

[0103] The output of the first ResNet module is used as the input of the second ResNet module. The second ResNet module has the same structure as the first ResNet module, except that the number of input channels for the convolution operation is 18 and the number of output channels is 28, and the number of output channels for the pooling operation is also 28.

[0104] The output of the second ResNet module serves as the input to the ViT Transformer module. The ViT Transformer module splits the input into two outputs, which are then summed. Specifically, the input is first passed through a convolutional layer with 28 input channels, 8 output channels, and a 1×1 kernel; simultaneously, it is passed through a multi-head self-attention layer with 28 input channels and 8 output channels. After summing the two outputs, the result is passed through a max-pooling layer with a 2×2 kernel and a stride of 2 to obtain the final output.

[0105] The above output is used as input to the MLP layer to obtain the predicted labels. Specifically, the feature map of the final output of the ViT Transformer module is first flattened into a vector, at which point the feature vector has a dimension of 28×28×8. Referring to Figure 8, the feature vector passes through three sets of Linear layers, ReLU activation functions, and Dropout layers in sequence to obtain a feature vector with an output dimension of 64. The output dimensions of the three Linear layers are 1024, 64, and 64, respectively. The feature vector with an output dimension of 64 is then input into a Linear layer with an input dimension of 64 and an output dimension of 3 to obtain the final predicted labels, namely the three classification results of surprised, negative, and positive micro-expressions.

[0106] Step S4.6: Perform multiple rounds of classification training on the optical flow image to obtain the trained PDTNet model. Then, combine the initialization loss function (the CrossEntropyLoss function and the Dense Relative Localization Loss function are added together as a joint loss function) for backpropagation and network training.

[0107] Step S5: Input the image to be tested into the trained PDTNet model to obtain the micro-expression recognition results.

[0108] On the other hand, to verify the effectiveness of the present invention, a test set was selected to test the proposed PDTNet network. The Leave One Subject Out (LOSO) cross-validation method was used to determine the segmentation of the training and test sets and to ensure independent evaluation by the subjects. Since the distribution of micro-expression classes remains imbalanced even after merging the databases, to reduce potential bias, the evaluation criteria of the present invention use two balanced metrics: unweighted F1 score (UF1) and unweighted average recall (UAR). Table 1 shows the classification performance using the trained PDTNet model and a comparison with other classification methods.

[0109]

[0110] Table 1. Comparison of the recognition performance of the present invention with that of other methods

[0111] Experimental results, as shown in Table 1, demonstrate that this invention exhibits superior performance on multiple datasets compared to other methods. References to existing technologies are as follows:

[0112] Literature[1]Zhao,G.,Pietikainen,M.:Dynamic texture recognition using localbinary patterns with an application to facial expressions.IEEE transactionson pattern analysis and machine intelligence 29(6),915–928(2007)

[0113] Reference [2] Gan, Y.S., Liong, S.-T., Yau, W.-C., Huang, Y.-C., Tan, L.-K.: Offapexnet on micro-expression recognition system. Signal Processing: Image Communication 74, 129–139 (2019)

[0114] Reference [3] Liong, S.-T., See, J., Wong, K., Phan, R.C.-W.: Less is more: Microexpression recognition from video using apex frame. Signal Processing: Image Communication 62, 82–92 (2018)

[0115] Reference [4] Zhou, L., Mao, Q., Xue, L.: Dual-inception network for cross-database micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face&Gesture Recognition (FG2019), pp. 1–5 (2019). IEEE

[0116] Reference [5] Liu, Y., Du, H., Zheng, L., Gedeon, T.: A neural micro-expression recognizer. In: 2019 14th IEEE International Conference on Automatic Face&Gesture Recognition (FG 2019), pp. 1–4 (2019). IEEE

[0117] Reference [6] Nie, X., Takalkar, M.A., Duan, M., Zhang, H., Xu, M.: Geme: Dual-stream multi-task gender-based micro-expression recognition. Neurocomputing 427, 13–28 (2021)

[0118] Literature [7] Chen, B., Liu, K.-H., Xu, Y., Wu, Q.-Q., Yao, J.-F.: Block divisionconvolutional network with implicit deep features augmentation for micro-expression recognition. IEEE Transactions on Multimedia (2022)

[0119] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope of the present invention.

Claims

1. A micro-expression recognition method based on frequency domain features, characterized in that, Including the following steps: Step S1: Extract the start frame and vertex frame from the micro-expression sequence, and denote the vertex frame as the selected frame; Step S2: Calculate the dense optical flow sequence based on the starting frame and selected frame of the micro-expression sequence; Step S3: Preprocess the optical flow images in the dense optical flow sequence; Step S4: Use the preprocessed optical flow image as input to train the PDTNet model; Step S5: Input the image to be tested into the trained PDTNet model to obtain the micro-expression recognition results; In step S4, the step of using the preprocessed optical flow image as input to train the PDTNet model includes the following steps: Step S4.1: Construct and initialize the PDTNet model; Step S4.2: Divide the optical flow image into n tokens, denoted as optical flow image X = [x1, x2, ..., x...]. n ], x n For the nth token; Step S4.3: Extract the amplitude feature |X| and phase feature θ of the optical flow image; Step S4.4: Combine the amplitude feature |X| and the phase feature θ to obtain the combined feature Z; Step S4.5: Add the fused feature Z to the original image of the optical flow image to obtain a new fused feature. Use the PDT module to further extract features to obtain classification features, and classify the classification features to obtain the corresponding predicted labels. Step S4.6: Perform multiple rounds of classification training on the optical flow image to obtain the trained PDTNet model.

2. The micro-expression recognition method based on frequency domain features as described in claim 1, characterized in that, Step S1 also includes the step of: extracting image frames within a preset number of frames on both sides of the vertex frame in the micro-expression sequence, and recording them as selected frames.

3. The micro-expression recognition method based on frequency domain features as described in claim 1, characterized in that, In step S2, the calculation of the optical flow sequence includes the following steps: using the Gunnar Farneback method to calculate and obtain the global dense optical flow sequence.

4. The micro-expression recognition method based on frequency domain features as described in claim 1, characterized in that, In step S3, the preprocessing of the optical flow image in the dense optical flow sequence includes the following steps: Standardize the size of optical flow images; Perform random radian rotation on the optical flow image; Randomly flip the image horizontally; Adjust the size of the optical flow image and train the optical flow image according to the preset network.

5. The micro-expression recognition method based on frequency domain features as described in claim 1, characterized in that, In step S4.3, extracting the phase feature θ of the optical flow image includes the following steps: The optical flow image X is processed by the parameter matrix W. q W k and W v Convolution yields the corresponding three vectors q, k, and v; The result of multiplying q by k is transposed using a transpose function and then combined with sine and cosine position codes to obtain the similarity score a_s; The similarity score a_s is calculated using softmax to obtain the normalized similarity ratio a_r; Multiplying the similarity ratio a_r by v yields the phase characteristic θ.

6. The micro-expression recognition method based on frequency domain features as described in claim 1, characterized in that, In step S4.4, obtaining the fused feature Z by fusing the amplitude feature |X| and the phase feature θ includes the following steps: The fusion feature z of the token in the optical flow image is calculated based on the preset fusion feature formula. j ; The fusion feature z of each token j Z constitutes the fusion feature of phase and amplitude in the optical flow image.

7. The micro-expression recognition method based on frequency domain features as described in claim 6, characterized in that, The preset fusion feature formula is: z j =|x j |⊙cosθ j +|x j |⊙isinθ j ,j∈(1,2,...,n.), Where i is the imaginary unit, |·| is the absolute value operation, ⊙ is element-wise multiplication, and |x j | and θ j These represent the amplitude and phase characteristics of the j-th token, respectively.

8. The micro-expression recognition method based on frequency domain features as described in claim 1, characterized in that, In step S4.5, obtaining new fusion features, further extracting features using the PDT module to obtain classification features, and classifying the classification features to obtain corresponding predicted labels includes the following steps: The new fused features are sequentially convolved and pooled through the first ResNet module, the second ResNet module, and the ViT Transformer module to obtain classification features; Input the classification features into the MLP layer module for classification and obtain the predicted labels of the classification features.

Images