A micro-expression recognition method and system based on ACNN and Bi-LSTM

Through the combination of ACNN and Bi-LSTM, the local and global features of micro-expressions are extracted, and combined with image sequence information, the problem of insufficient recognition accuracy in the prior art is solved, and a higher recognition accuracy of micro-expressions is achieved.

CN114724210BActive Publication Date: 2024-12-03BEIJING CHEAP BUY NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210299804.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-25
Publication Date
2024-12-03
Estimated Expiration
2042-03-25

AI Technical Summary

Technical Problem

The existing micro-expression recognition methods cannot effectively combine local and global features, resulting in insufficient recognition accuracy and failure to make full use of image sequence correlation.

Method used

Using the micro-expression recognition method based on ACNN and Bi-LSTM, expression features are extracted through initial feature vector extraction network, local recognition block attention convolution neural network and global attention convolution neural network, and combined with bidirectional long and short-term memory network, Expression category recognition.

Benefits of technology

Accurate recognition of micro-expressions is achieved, recognition accuracy is improved, local and global features can be effectively combined, and image sequence information is used to enhance the recognition effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114724210B_ABST
    Figure CN114724210B_ABST
Patent Text Reader

Abstract

The present application relates to a method and system for micro-expression recognition based on ACNN and Bi-LSTM. The method comprises: obtaining a continuous multi-frame expression image of a person to be detected; inputting the multi-frame expression image into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs; wherein the micro-expression recognition model comprises an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network. The use of this method can improve the accuracy of micro-expression recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a micro-expression recognition method and system based on ACNN and Bi-LSTM. Background Art

[0002] Compared with ordinary expressions, micro-expressions are facial expressions with short duration and small movement amplitude. As a spontaneous expression, micro-expressions are produced when trying to cover up inner emotions. They are expressions that can neither be faked nor suppressed. In 1966, Haggard et al. discovered a facial expression that is fast and difficult to be detected by the human eye and first proposed the concept of micro-expressions. At first, this tiny and short-lived facial change did not attract the attention of other peer researchers. It was not until 1969 that Ekman found that when studying a video of a patient with depression, a patient with a smiling expression would have a short-term painful expression. The patient used positive expressions such as smiles to cover up his inner anxiety. Unlike macro-expressions, micro-expressions last only 1 / 25 to 1 / 5 seconds. Therefore, it is not enough to accurately identify micro-expressions by human eyes alone. It is very necessary to use modern artificial intelligence to identify them.

[0003] The existing spatial attention model SAIN_GRAY is trained by the original grayscale image, but the sequence correlation between images is not considered, or the ATNet method is used to extract the micro-expression spatial information from the vertex frame using ResNet10, and the micro-expression temporal information from the adjacent frames using LSTM and recognize it after feature fusion. However, the existing micro-expression recognition method does not have the ability to pay attention to the overall situation while taking into account the details. Summary of the invention

[0004] Based on this, it is necessary to provide a micro-expression recognition method and system based on ACNN and Bi-LSTM that can improve the accuracy of micro-expression recognition in response to the above technical problems.

[0005] A micro-expression recognition method based on ACNN and Bi-LSTM, the method comprising:

[0006] Acquire multiple frames of facial expression images of the person to be detected over a continuous period of time;

[0007] The multi-frame expression images are input into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs; wherein the micro-expression recognition model includes an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network.

[0008] In one embodiment, the step of inputting the multiple frames of expression images into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs includes:

[0009] Inputting the multiple frames of expression images into an initial feature vector extraction network and subjecting the network to transfer learning processing to obtain an initial feature vector of a first preset scale;

[0010] Input the initial feature vector into the local recognition block attention convolutional neural network, extract multiple recognition blocks, and calculate the weighted feature vector of each recognition block;

[0011] Input the initial feature vector into the global attention convolutional neural network to obtain a weighted global feature vector;

[0012] The weighted feature vectors of all recognition blocks and the weighted global feature vectors are respectively input into a bidirectional long short-term memory network, and processed through three fully connected layers to obtain the category to which the expression of the person to be detected belongs.

[0013] In one embodiment, the step of inputting the multiple frames of expression images into an initial feature vector extraction network and processing them through transfer learning to obtain an initial feature vector of a first preset scale comprises: inputting the multiple frames of expression images into a VGG16 network and processing them through transfer learning to obtain a feature vector of a scale of 512×28×28.

[0014] In one of the embodiments, the initial feature vector is input into a local recognition block attention convolutional neural network, multiple recognition blocks are extracted, and a weighted feature vector of each recognition block is calculated, including: extracting facial key points according to the initial feature vector; extracting micro-expression recognition points in the eyebrow, eye, nose, and mouth areas according to the facial key points; extracting a recognition block of a second preset scale from the initial feature vector with the micro-expression recognition point as the center; and processing the recognition blocks through a fully connected layer and a normal attention network with the output as a weight scalar, respectively, to obtain a weighted feature vector for each recognition block.

[0015] In one embodiment, the inputting the initial feature vector into the global attention convolutional neural network to obtain a weighted global feature vector includes: processing the initial feature vector through the conv4_2 layer to the conv5_2 layer of the VGG16 network to obtain a feature vector with an output scale of 512×14×14; processing the feature vector with a scale of 512×14×14 through a fully connected layer and a normal attention network whose output is a weight scalar to obtain a weighted global feature vector.

[0016] In one of the embodiments, the micro-expression recognition method based on ACNN and Bi-LSTM further includes: obtaining multi-frame expression images of multiple collectors in continuous time, and setting a micro-expression label for each collector to construct a model training data set; inputting the model training data set into an initial micro-expression recognition model, training the initial micro-expression recognition model, and obtaining a trained micro-expression recognition model; wherein the trained micro-expression recognition model includes an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network.

[0017] In one embodiment, the bidirectional long short-term memory network further includes a hidden layer, a fully connected layer, and a Dropout layer between the hidden layer and the fully connected layer.

[0018] A micro-expression recognition system based on ACNN and Bi-LSTM, the system comprising:

[0019] The detection image acquisition module is used to acquire multiple frames of facial expression images of the person to be detected over a continuous period of time;

[0020] An expression category recognition module is used to input the multiple frames of expression images into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs; wherein the micro-expression recognition model includes an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network.

[0021] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0022] Acquire multiple frames of facial expression images of the person to be detected over a continuous period of time;

[0023] The multi-frame expression images are input into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs; wherein the micro-expression recognition model includes an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network.

[0024] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0025] Acquire multiple frames of facial expression images of the person to be detected over a continuous period of time;

[0026] The multi-frame expression images are input into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs; wherein the micro-expression recognition model includes an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network.

[0027] The above-mentioned micro-expression recognition method, system, computer device and storage medium based on ACNN and Bi-LSTM adopt a micro-expression recognition model including a local recognition block attention convolutional neural network, a global attention convolutional neural network and a bidirectional long short-term memory network, which can perform local expression feature recognition and global feature recognition on multiple frames of expression images of the person to be detected in continuous time, and combine the expression changes in continuous time to accurately identify the category to which the expression of the person to be detected belongs. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 1 is a flow chart of a micro-expression recognition method based on ACNN and Bi-LSTM in one embodiment;

[0029] Figure 2 A schematic diagram of the structure of a micro-expression recognition model in one embodiment;

[0030] Figure 3 A schematic diagram of extracting 68 facial key points using the Dlib method in one embodiment;

[0031] Figure 4 A schematic diagram of a bidirectional recurrent neural network structure in one embodiment;

[0032] Figure 5 This is a schematic diagram of the LSTM unit network structure in one embodiment;

[0033] Figure 6 is a structural block diagram of a micro-expression recognition system based on ACNN and Bi-LSTM in one embodiment;

[0034] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] In one embodiment, Figure 1 As shown, a micro-expression recognition method based on ACNN and Bi-LSTM is provided, including the following steps:

[0037] S110, obtaining multiple frames of facial expression images of the person to be detected over a continuous period of time.

[0038] The continuous-time multi-frame expression image is a video sequence with a start frame, a vertex frame and an end frame, which is obtained by continuously shooting the face of the same person to be detected with an industrial camera.

[0039] S120, input the multiple frames of expression images into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs; wherein the micro-expression recognition model includes an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network.

[0040] Among them, the category to which the expression belongs can be negative, positive, surprised, etc. The initial feature vector extraction network is used to extract feature vectors that are suitable for the input scale of the local recognition block attention convolutional neural network and the global attention convolutional neural network. For example, the initial feature vector extraction network processes the input multi-frame expression images into feature vectors with a scale of 512×28×28. The local recognition block attention convolutional neural network (BR-ACNN) is a network for micro-expression detail recognition based on the improvement of ACNN. Since the micro-expression facial changes are small and difficult to be effectively recognized, this application adopts the recognition method of facial key area block attention, that is, the recognition of the key representative area of ​​micro-expression is added to the facial features to be recognized and the corresponding attention network is adopted. The global attention convolutional neural network (GR-ACNN) has the ability to learn subtle changes in facial features of human faces. However, the completion of micro-expression movements sometimes involves the cooperation of various parts of the face, so it is necessary not only to extract the features of facial blocks but also global features. Integrating global features into feature recognition is expected to improve the recognition effect of micro-expressions. The bidirectional long short-term memory network (Bi-LSTM) adds a back propagation layer to the unidirectional LSTM so that the LSTM model can not only utilize historical sequence information but also future information. The bidirectional long short-term memory network (Bi-LSTM) used in this application can better extract the feature information and sequence information in micro-expressions.

[0041] In the above-mentioned micro-expression recognition method based on ACNN and Bi-LSTM, a micro-expression recognition model including a local recognition block attention convolutional neural network, a global attention convolutional neural network and a bidirectional long short-term memory network is adopted, which can perform local expression feature recognition and global feature recognition on multiple frames of expression images of the person to be detected in continuous time, and combine the expression changes in continuous time to accurately identify the category of the expression of the person to be detected.

[0042] In one embodiment, the step of inputting the multiple frames of expression images into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs includes: inputting the multiple frames of expression images into an initial feature vector extraction network and subjecting them to transfer learning processing to obtain an initial feature vector of a first preset scale; inputting the initial feature vector into a local recognition block attention convolutional neural network to extract multiple recognition blocks, and calculating a weighted feature vector for each recognition block; inputting the initial feature vector into a global attention convolutional neural network to obtain a weighted global feature vector; respectively inputting the weighted feature vectors of all recognition blocks and the weighted global feature vector into a bidirectional long short-term memory network, and processing them through three fully connected layers to obtain the category to which the expression of the person to be detected belongs.

[0043] The first preset scale may be 512×28×28, where 512 represents the number of channels of the feature vector and 28×28 represents the pixel size of a single channel feature vector. After inputting the initial feature vector, the local recognition block attention convolutional neural network divides the micro-expression recognition area according to the facial area, extracts multiple recognition blocks, and calculates the weighted feature vector for each recognition block through the attention network.

[0044] In one embodiment, the step of inputting the multiple frames of expression images into an initial feature vector extraction network and processing them through transfer learning to obtain an initial feature vector of a first preset scale comprises: inputting the multiple frames of expression images into a VGG16 network and processing them through transfer learning to obtain a feature vector of a scale of 512×28×28.

[0045] In one of the embodiments, the initial feature vector is input into a local recognition block attention convolutional neural network, multiple recognition blocks are extracted, and a weighted feature vector of each recognition block is calculated, including: extracting facial key points according to the initial feature vector; extracting micro-expression recognition points in the eyebrow, eye, nose, and mouth areas according to the facial key points; extracting a recognition block of a second preset scale from the initial feature vector with the micro-expression recognition point as the center; and processing the recognition blocks through a fully connected layer and a normal attention network with the output as a weight scalar, respectively, to obtain a weighted feature vector for each recognition block.

[0046] Specifically, facial key points can be extracted using the Dlib [Xia Z, Hong X, Gao X, et al. Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-expressions [J]. IEEE Transactions on Multimedia, 2019] method to obtain 68 facial key points. Micro-expression recognition points are selected from facial key points, but changes in micro-expressions can be perceived through changes in the features of micro-expression recognition points. Specifically, 68 facial key points (such as Figure 3 As shown in the figure), then based on the 68 facial key points, they are converted into 24 micro-expression recognition points, and the recognition point positions cover the cheeks, mouth, nose, eyes, and eyebrows; the conversion process is as follows: first, 16 recognition points covering the mouth, nose, eyes, and eyebrows are selected from the 68 facial key points, and the 68 facial key points (as shown in the figure) are converted into 24 micro-expression recognition points. The recognition point positions cover the cheeks, mouth, nose, eyes, and eyebrows. Figure 3 The extraction sequence numbers in the figure are as follows: 19, 22, 23, 26, 39, 37, 44, 46, 28, 30, 49, 51, 53, 55, 59, 57, and the micro-expression recognition point sequences generated are as follows: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16; Secondly, since there is no direct key point coverage for the eyes, eyebrows and cheeks, the midpoint coordinates of the key points are used to generate them in this embodiment. For example, the left eye, left eyebrow and left cheek are generated from 68 facial key points (as shown in FIG. Figure 3 As shown in the figure, the midpoint coordinates of the point pairs (20, 38), (41, 42), and (18, 59) are selected as micro-expression recognition points, and the micro-expression recognition points are numbered: 17, 19, and 18; the midpoint coordinates of the point pairs (25, 45), (47, 48), and (27, 57) are selected from the 68 facial key points for the right eye, right eyebrow, and right cheek as micro-expression recognition points, and the micro-expression recognition points are numbered: 20, 21, and 22; finally, for the left and right corners of the mouth, this embodiment selects key point 49 and key point 55 from the 68 facial key points, and then selects the relative offset points of the two corners of the mouth according to the coordinates of the two points, and finally obtains the micro-expression recognition points numbered: 23 and 24.

[0047] The calculation formula for the coordinates of micro-expression recognition points numbered 23 and 24 is as follows:

[0048] (x 23 ,y 23 )=(x left -16,y left -16) (1)

[0049] (x 24 ,y 24 )=(x right +16,y right +16) (2)

[0050] Among them, x left ,y left is the coordinate of key point 49, x right ,y right is the coordinate of key point 55, x 23 ,y 23 is the coordinate of micro-expression recognition point number 23, x 24 ,y 24 is the coordinate of micro-expression recognition point number 24. Thus, 24 micro-expression recognition points in the eyebrow, eye, nose, and mouth regions are finally extracted based on the facial key points. With the 24 micro-expression recognition points as the center, 24 recognition regions with a scale of 6×6 are generated, where the scale of the recognition region can be set according to the accuracy of the model. Among them, in order to improve the robustness of the model and increase the receptive field of the recognition block, the recognition region will act on the 512×28×28 feature vector after VGG16 transfer learning processing to obtain 24 recognition blocks, each of which is 512×6×6 in size, that is, the second preset scale can be 512×6×6.

[0051] The local recognition block convolutional neural network with attention mechanism (BR-ACNN) is as follows Figure 2 The upper part of the dotted box is shown. In BR-ACNN, the 24 local recognition blocks obtained after cropping are respectively passed through a fully connected layer and a normal attention network with the output as a weight scalar, and finally 24 weighted feature vectors are obtained. In the local recognition block convolutional neural network (BR-ACNN), the attention weight p of the i-th recognition block i , unweighted features The characteristic α of the attention weight i The calculation formula is as follows:

[0052]

[0053]

[0054]

[0055] Among them, c i represents the feature vector of the i-th recognition block, is the operation in the attention network, and τ(·) represents the feature learning of the input feature vector. The weighted feature vectors of all recognition blocks can be calculated through equations (1), (2) and (3).

[0056] In one embodiment, the inputting the initial feature vector into the global attention convolutional neural network to obtain a weighted global feature vector includes: processing the initial feature vector through the conv4_2 layer to the conv5_2 layer of the VGG16 network to obtain a feature vector with an output scale of 512×14×14; processing the feature vector with a scale of 512×14×14 through a fully connected layer and a normal attention network whose output is a weight scalar to obtain a weighted global feature vector.

[0057] Among them, the detailed structure of the global attention convolutional neural network (GR-ACNN) is as follows Figure 2 As shown in the lower half of the dotted box. The scale of the input feature vector of the global attention convolutional neural network (GR-ACNN) is 512×28×28. First, in this embodiment, the input global feature vector passes through the conv4_2 to conv5_2 layers of the VGG16 network to obtain a feature vector with an output size of 512×14×14; secondly, the feature vector of size 512×14×14 passes through a fully connected layer and an ordinary attention network with an output as a weight scalar, and finally outputs a weighted global feature vector.

[0058] In one of the embodiments, the micro-expression recognition method based on ACNN and Bi-LSTM also includes: obtaining multi-frame expression images of multiple collectors in continuous time, and setting a micro-expression label for each collector to construct a model training data set; inputting the model training data set into an initial micro-expression recognition model, training the initial micro-expression recognition model, and obtaining a trained micro-expression recognition model; wherein the trained micro-expression recognition model includes an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network.

[0059] Among them, the multi-frame expression images of the collector in continuous time can use the CSAME II dataset established by Fu Xiaolan's team at the Institute of Psychology, Chinese Academy of Sciences. The CSAME II dataset uses a 200 frames / second high-speed camera, and each frame size is 280×340 pixels. The CSAME II dataset has 255 samples, the average age of the participants in the dataset is 22 years old, and the total number of subjects collected is 24. The dataset includes the micro-expression labels corresponding to each sample of the subject and has the start frame, apex frame, and end frame video sequences. The labels include repression, disgust, happiness, surprise, fear, sadness, and other categories (repression, disgust, happiness, surprise, fear, sadness, others). In this embodiment, the CSAME II data set is divided into three categories: negative, positive, and surprised, and micro-expression labels are set. The temporal interpolation model (TIM) is used to process the data set into a fixed 10-frame input sequence and the video frame is normalized to 224×224 pixels by bilinear interpolation. Then, the Dlib method is used to crop, align, and extract 68 facial key points. After analyzing and processing the facial key points, this embodiment selects 24 micro-expression recognition points, and generates 24 recognition blocks of 6×6 in size from the 24 micro-expression recognition points. Finally, the pre-processed video frames and the recognition blocks of each frame are put into the model training.

[0060] In one embodiment, the bidirectional long short-term memory network further includes a hidden layer, a fully connected layer, and a Dropout layer between the hidden layer and the fully connected layer.

[0061] Among them, the long short-term memory network (LSTM) is a new structure designed to overcome the long-term dependency problem that has always existed in traditional recurrent neural networks (RNN). The bidirectional long short-term memory network (Bi-LSTM) adds a reverse layer on the basis of the long short-term memory network (LSTM), so that the new network structure can not only use the historical information that has been input, but also has the ability to capture future available information. The structure of the bidirectional recurrent neural network is as follows Figure 4 As shown in the figure, the Bi-LSTM structure replaces each node of the bidirectional recurrent neural network with an LSTM unit. Define the input feature sequence of the Bi-LSTM model as X = (x1, ..., x T ), the sequence of hidden layer variables in the previous propagation is And the sequence of hidden layer variables in back propagation is Define the Bi-LSTM model output sequence as y = (y1, ..., y T ), the following formula is obtained:

[0062]

[0063]

[0064]

[0065] Among them, S(x) is the activation function, Respectively represent the elements in the characteristic sequence X in the calculation The weight of Represents the sequence of hidden layer variables in the previous propagation The weight of the elements in , Represents the sequence of hidden layer variables in back propagation The weight of the elements in , b0 represents the calculation y t When the bias term, x t is the tth element in the input feature sequence X, is the sequence of hidden layer variables in the previous propagation The tth element in is the sequence of hidden layer variables in back propagation The tth element in is the sequence of hidden layer variables in the previous propagation The t-1th element in is the sequence of hidden layer variables in back propagation The t-1th element in y t The tth element of the Bi-LSTM model output sequence, for The corresponding weight, for The corresponding weights. Each part is calculated using LSTM units. LSTM units are as follows Figure 5 As shown. The input of LSTM is the spatial features extracted from the frame sequence by the BR-ACNN and GR-ACNN models and spliced. This embodiment adopts a single-layer bidirectional LSTM structure, which includes a hidden layer of 256 nodes. In order to increase the robustness of the model network nodes and reduce the complex co-adaptation relationship between neurons, this embodiment adds a Dropout layer between the bidirectional LSTM hidden layer and the fully connected layer to randomly shield neurons with a certain probability.

[0066] In one embodiment, the unweighted F1-score (UF1), unweighted average recall (UAR) and accuracy (Acc) are used as evaluation indicators of the micro-expression recognition model. The calculation of the unweighted F1-score (UF1) is shown in formula (9), where TP i , FP i 、FN i are the number of true positives, false positives, and false negatives in the i-th category, respectively, and C is the number of categories. The calculation of the unweighted average recall (UAR) is shown in formula (10), where TP i is the number of correct predictions in the i-th category, N i is the number of samples in the ith category. The accuracy (Acc) is given by formula (11), where TP is the number of true positive examples in all categories and FP is the number of false positive examples in all categories. The formula is as follows:

[0067]

[0068]

[0069]

[0070] The Adam optimizer is used for training, the learning rate is 0.00016, the number of iterations epoch is set to 100, and the training batch_size is set to 16. Because micro-expressions are small data sets that are prone to overfitting, in order to improve the robustness and generalization ability of the model, this embodiment takes the regularized L2 norm of the micro-expression recognition model parameters and adds λ times the L2 parameter norm in the loss function. After multiple experiments, it is shown that the model effect is best when λ is set to 0.01. The results of the 5-fold cross-validation experiment show that the average accuracy of the 5-fold cross-validation is 69%, UF1 is 0.6382, and UAR is 0.6750. The improved micro-expression recognition model in this embodiment has a relative improvement in accuracy compared to previous recognition algorithms, among which UF1 is improved by 0.0281 and UAR is improved by 0.0969 compared to the OFFApexNet model; UF1 is improved by 0.0072 and UAR is improved by 0.032 compared to the ATNet model.

[0071] The above-mentioned micro-expression recognition method based on ACNN and Bi-LSTM, considering that the micro-expression data set is small and prone to overfitting problems, first extracts the feature set of the preprocessed micro-expression video frame through VGG16; then extracts the local and global features of the extracted feature set through the local recognition block attention convolutional neural network (BR-ACNN) and the global attention convolutional neural network (GR-ACNN); then extracts the sequence information of each frame through the bidirectional long short-term memory network (Bi-LSTM); and finally classifies through three fully connected layers. Experiments show that the recognition method based on the recognition block attention network and the bidirectional long short-term memory network can effectively extract useful information from video frames with small movement amplitudes, and shows a high accuracy rate in the experiment.

[0072] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0073] In one embodiment, Figure 6 As shown, a micro-expression recognition system based on ACNN and Bi-LSTM is provided, including: an image acquisition module 210 to be detected and an expression category recognition module 220, wherein:

[0074] The detection image acquisition module 210 is used to acquire multiple frames of facial expression images of the person to be detected over a continuous period of time;

[0075] The expression category recognition module 220 is used to input the multiple frames of expression images into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs; wherein the micro-expression recognition model includes an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network.

[0076] In one embodiment, the expression category recognition module 220 includes:

[0077] A transfer learning processing unit, used for inputting the multiple frames of expression images into an initial feature vector extraction network for transfer learning processing to obtain an initial feature vector of a first preset scale;

[0078] A recognition block extraction unit, used for inputting the initial feature vector into the local recognition block attention convolutional neural network, extracting multiple recognition blocks, and calculating the weighted feature vector of each recognition block;

[0079] A global feature vector extraction unit, used to input the initial feature vector into the global attention convolutional neural network to obtain a weighted global feature vector;

[0080] The expression category detection unit is used to input the weighted feature vectors of all recognition blocks and the weighted global feature vectors into a bidirectional long short-term memory network respectively, and process them through three fully connected layers to obtain the category to which the expression of the person to be detected belongs.

[0081] In one of the embodiments, the transfer learning processing unit is further used to input the multiple frames of expression images into the VGG16 network for transfer learning processing to obtain a feature vector with a scale of 512×28×28.

[0082] In one of the embodiments, the recognition block extraction unit includes: a facial key point extraction subunit, which is used to extract facial key points based on the initial feature vector; a micro-expression recognition point extraction subunit, which is used to extract micro-expression recognition points in the eyebrow, eye, nose and mouth areas based on the facial key points; a recognition block extraction subunit, which is used to extract a recognition block of a second preset scale from the initial feature vector with the micro-expression recognition point as the center; and a recognition block weighted feature vector calculation subunit, which is used to process the recognition block through a fully connected layer and a normal attention network with the output as a weight scalar, respectively, to obtain a weighted feature vector for each recognition block.

[0083] In one embodiment, the global feature vector extraction unit includes: a convolution processing subunit, used to process the initial feature vector through the conv4_2 layer to the conv5_2 layer of the VGG16 network to obtain a feature vector with an output scale of 512×14×14; a weighted global feature vector calculation subunit, used to process the feature vector with a scale of 512×14×14 through a fully connected layer and a normal attention network whose output is a weight scalar to obtain a weighted global feature vector.

[0084] In one embodiment, the micro-expression recognition system based on ACNN and Bi-LSTM also includes: a model training data set construction module, which is used to obtain multi-frame expression images of multiple collectors in continuous time, and set the micro-expression label of each collector to construct a model training data set; a training module, which is used to input the model training data set into an initial micro-expression recognition model, train the initial micro-expression recognition model, and obtain a trained micro-expression recognition model; wherein the trained micro-expression recognition model includes an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network.

[0085] In one embodiment, the bidirectional long short-term memory network further includes a hidden layer, a fully connected layer, and a Dropout layer between the hidden layer and the fully connected layer.

[0086] For the specific definition of the micro-expression recognition system based on ACNN and Bi-LSTM, please refer to the definition of the micro-expression recognition method based on ACNN and Bi-LSTM above, which will not be repeated here. Each module in the above-mentioned micro-expression recognition system based on ACNN and Bi-LSTM can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0087] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store micro-expression recognition model data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a micro-expression recognition method based on ACNN and Bi-LSTM is implemented.

[0088] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0089] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0090] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0091] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0092] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A micro-expression recognition method based on ACNN and Bi-LSTM, characterized in that: The method comprises: Acquire multiple frames of facial expression images of the person to be detected over a continuous period of time; The multi-frame expression images are input into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs; wherein the micro-expression recognition model includes an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network; wherein the structure of the bidirectional long short-term memory network is to replace each node of the bidirectional recurrent neural network with an LSTM unit, and define the input feature sequence of the bidirectional long short-term memory network model as X=(x1, ..., x T ), the sequence of hidden layer variables in the previous propagation is And the sequence of hidden layer variables in back propagation is Define the output sequence of the bidirectional long short-term memory network model as y = (y1, ..., y T ), the following formula is obtained: Among them, S(x) is the activation function, Respectively represent the elements in the characteristic sequence X in the calculation The weight of Represents the sequence of hidden layer variables in the previous propagation The weight of the elements in , Represents the sequence of hidden layer variables in back propagation The weight of the elements in , b0 represents the calculation y t When the bias term, x t is the tth element in the input feature sequence X, is the sequence of hidden layer variables in the previous propagation The tth element in is the sequence of hidden layer variables in back propagation The tth element in is the sequence of hidden layer variables in the previous propagation The t-1th element in is the sequence of hidden layer variables in back propagation The t-1th element in y t is the tth element of the output sequence of the bidirectional long short-term memory network model, for The corresponding weights, for The corresponding weights; The step of inputting the multiple frames of expression images into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs includes: Inputting the multiple frames of expression images into an initial feature vector extraction network and subjecting the network to transfer learning processing to obtain an initial feature vector of a first preset scale; Input the initial feature vector into the local recognition block attention convolutional neural network, extract multiple recognition blocks, and calculate the weighted feature vector of each recognition block; Input the initial feature vector into the global attention convolutional neural network to obtain a weighted global feature vector; The weighted feature vectors of all recognition blocks and the weighted global feature vectors are respectively input into a bidirectional long short-term memory network, and processed through three fully connected layers to obtain the category to which the expression of the person to be detected belongs; The step of inputting the initial feature vector into the local recognition block attention convolutional neural network, extracting multiple recognition blocks, and calculating the weighted feature vector of each recognition block includes: Extract facial key points based on the initial feature vector; Extract micro-expression recognition points in the eyebrow, eye, nose, and mouth areas based on facial key points; Taking the micro-expression recognition point as the center, extracting a recognition block of a second preset scale from the initial feature vector; Processing the recognition blocks through a fully connected layer and a common attention network with the output as a weight scalar, respectively, to obtain a weighted feature vector for each recognition block; Among them, in the local recognition block attention convolutional neural network, the attention weight p of the i-th recognition block i , unweighted features The characteristic α of the attention weight i The calculation formula is as follows: Among them, c i represents the feature vector of the i-th recognition block, is the operation in the attention network, τ(·) represents the feature learning of the input feature vector; The step of inputting the initial feature vector into the global attention convolutional neural network to obtain a weighted global feature vector includes: The initial feature vector is processed through the conv4_2 layer to the conv5_2 layer of the VGG16 network to obtain a feature vector with an output scale of 512×14×14; The feature vector of scale 512×14×14 is processed by a fully connected layer and a normal attention network with output as weight scalar to obtain a weighted global feature vector.

2. The method according to claim 1, characterized in that The step of inputting the multiple frames of expression images into the initial feature vector extraction network and performing transfer learning processing to obtain an initial feature vector of a first preset scale comprises: The multi-frame expression images are input into the VGG16 network and processed by transfer learning to obtain a feature vector with a scale of 512×28×28.

3. The method according to claim 1, characterized in that Also includes: Obtain multiple frames of facial expression images of multiple collectors in a continuous period of time, set micro-expression labels for each collector, and construct a model training dataset; Inputting the model training data set into an initial micro-expression recognition model, training the initial micro-expression recognition model, and obtaining a trained micro-expression recognition model; Among them, the trained micro-expression recognition model includes an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network.

4. The method according to any one of claims 1 to 3, characterized in that: The bidirectional long short-term memory network also includes a middle hidden layer, a fully connected layer, and a Dropout layer between the hidden layer and the fully connected layer.

5. A micro-expression recognition system based on ACNN and Bi-LSTM, characterized in that: The system comprises: The detection image acquisition module is used to acquire multiple frames of facial expression images of the person to be detected over a continuous period of time; The expression category recognition module is used to input the multi-frame expression images into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs; wherein the micro-expression recognition model includes an initial feature vector extraction network, a local recognition block attention convolutional neural network connected to the initial feature vector extraction network, a global attention convolutional neural network connected to the initial feature vector extraction network, and a bidirectional long short-term memory network connected to the local recognition block attention convolutional neural network and the global attention convolutional neural network; wherein the structure of the bidirectional long short-term memory network is to replace each node of the bidirectional recurrent neural network with an LSTM unit, and define the input feature sequence of the bidirectional long short-term memory network model as X=(x1, ..., x T ), the sequence of hidden layer variables in the previous propagation is And the sequence of hidden layer variables in back propagation is Define the output sequence of the bidirectional long short-term memory network model as y = (y1, ..., y T ), the following formula is obtained: Among them, S(x) is the activation function, Respectively represent the elements in the characteristic sequence X in the calculation The weight of Represents the sequence of hidden layer variables in the previous propagation The weight of the elements in , Represents the sequence of hidden layer variables in back propagation The weight of the elements in , b0 represents the calculation y t When the bias term, x t is the tth element in the input feature sequence X, is the sequence of hidden layer variables in the previous propagation The tth element in is the sequence of hidden layer variables in back propagation The tth element in is the sequence of hidden layer variables in the previous propagation The t-1th element in is the sequence of hidden layer variables in back propagation The t-1th element in y t is the tth element of the output sequence of the bidirectional long short-term memory network model, for The corresponding weights, for The corresponding weights; The step of inputting the multiple frames of expression images into a micro-expression recognition model to obtain the category to which the expression of the person to be detected belongs includes: Inputting the multiple frames of expression images into an initial feature vector extraction network and subjecting the network to transfer learning processing to obtain an initial feature vector of a first preset scale; Input the initial feature vector into the local recognition block attention convolutional neural network, extract multiple recognition blocks, and calculate the weighted feature vector of each recognition block; Input the initial feature vector into the global attention convolutional neural network to obtain a weighted global feature vector; The weighted feature vectors of all recognition blocks and the weighted global feature vectors are respectively input into a bidirectional long short-term memory network, and processed through three fully connected layers to obtain the category to which the expression of the person to be detected belongs; The step of inputting the initial feature vector into the local recognition block attention convolutional neural network, extracting multiple recognition blocks, and calculating the weighted feature vector of each recognition block includes: Extract facial key points based on the initial feature vector; Extract micro-expression recognition points in the eyebrow, eye, nose, and mouth areas based on facial key points; Taking the micro-expression recognition point as the center, extracting a recognition block of a second preset scale from the initial feature vector; Processing the recognition blocks through a fully connected layer and a common attention network with the output as a weight scalar, respectively, to obtain a weighted feature vector for each recognition block; Among them, in the local recognition block attention convolutional neural network, the attention weight p of the i-th recognition block i , unweighted features The characteristic α of the attention weight i The calculation formula is as follows: Among them, c i represents the feature vector of the i-th recognition block, is the operation in the attention network, τ(·) represents the feature learning of the input feature vector; The step of inputting the initial feature vector into the global attention convolutional neural network to obtain a weighted global feature vector includes: The initial feature vector is processed through the conv4_2 layer to the conv5_2 layer of the VGG16 network to obtain a feature vector with an output scale of 512×14×14; The feature vector of scale 512×14×14 is processed by a fully connected layer and a normal attention network with output as weight scalar to obtain a weighted global feature vector.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.