A method and device for recognizing micro-expressions of human face

Abstract

The present application proposes a method and device for recognizing micro-expressions of human faces, which includes: obtaining a sequence of micro-expression images of human faces to be recognized; respectively obtaining the HLTP features of each pixel of the sequence of micro-expression images of human faces; wherein the HLTP features are image features obtained by combining LTP features with Haar features; based on the HLTP features of each pixel, determining the feature histogram of the sequence of micro-expression images of human faces on each plane of the three-dimensional plane of time and space; classifying the sequence of micro-expression images of human faces according to the feature histogram of the sequence of micro-expression images of human faces on each plane of the three-dimensional plane of time and space, and determining the expression category corresponding to the sequence of micro-expression images of human faces. The above method can more accurately recognize micro-expressions of human faces.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for recognizing facial micro-expressions. Background Art

[0002] Facial expressions play a very important role in the process of communication between people. However, some facial expressions are short-lived and fleeting, and are not easy to detect. In 1969, Ekman and Friesen defined a new facial expression, called micro-expression. Ekman and others conducted a series of studies on facial micro-expressions. The results showed that micro-expressions play a vital role in identifying lies and can be widely used in national security, justice, clinical and other fields. Micro-expression research has attracted widespread attention from social media and the scientific community.

[0003] Existing facial micro-expression recognition is usually based on LTP features. The recognition effect of this recognition method is poor and it is often unable to accurately recognize facial micro-expressions. Summary of the invention

[0004] Based on the above content, the present application proposes a method and device for recognizing facial micro-expressions, which can more accurately recognize facial micro-expressions.

[0005] A method for recognizing facial micro-expressions, comprising:

[0006] Obtain a sequence of facial micro-expression images to be recognized;

[0007] Respectively obtain the HLTP features of each pixel of the facial micro-expression image sequence; wherein the HLTP feature is an image feature obtained by combining the LTP feature with the Haar feature;

[0008] Determine a feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane based on the HLTP feature of each pixel;

[0009] According to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, the facial micro-expression image sequence is classified to determine the expression category corresponding to the facial micro-expression image sequence.

[0010] Optionally, after obtaining the facial micro-expression image sequence to be recognized, the method further includes:

[0011] The facial micro-expression image sequence is enlarged, denoised and normalized.

[0012] Optionally, respectively acquiring the HLTP features of each pixel of the facial micro-expression image sequence includes:

[0013] Corresponding to each pixel of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, the following processing is performed respectively:

[0014] The feature calculation threshold of the pixel is calculated by performing convolution calculation on the HLTP coding model and the feature calculation area, wherein the feature calculation area refers to an image area of ​​a set size centered on the pixel on the plane where the pixel is located;

[0015] The HLTP feature value of the pixel is calculated and determined according to the convolution result of the HLTP coding model and the feature calculation area and the feature calculation threshold.

[0016] Optionally, determining a feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane based on the HLTP feature of each pixel includes:

[0017] For each plane of the three-dimensional space-time plane, the following processing is performed respectively:

[0018] The HLTP features of each pixel of the facial micro-expression image sequence on the plane are counted to determine a feature histogram of the facial micro-expression image sequence on the plane.

[0019] Optionally, classifying the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane to determine the expression category corresponding to the facial micro-expression image sequence includes:

[0020] Determining a feature histogram of the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane;

[0021] Inputting the feature histogram of the facial micro-expression image sequence into a pre-trained expression classification model to determine the expression category corresponding to the facial micro-expression image sequence;

[0022] The expression classification model is obtained by setting weights for the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane, and performing expression classification training on the facial micro-expression image sequence samples based on the set weights and the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane.

[0023] Optionally, the training process of the expression classification model includes:

[0024] For each preset expression category, the corresponding SVM classifier is trained by the following processing:

[0025] Obtaining a feature histogram of facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane;

[0026] Setting weights for the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane, and obtaining weighted feature histograms of the facial micro-expression image sequence samples;

[0027] Using the SVM classifier to classify facial micro-expression image sequence samples based on the weighted feature histogram of the facial micro-expression image sequence samples;

[0028] Based on the expression classification results and the expression category labels corresponding to the facial micro-expression image sequence samples, the parameters of the SVM classifier are calibrated;

[0029] Repeat the above process until the SVM classifier can accurately classify the facial micro-expression image sequence samples;

[0030] The trained SVM classifiers corresponding to each preset expression category are combined to obtain an expression classification model.

[0031] A facial micro-expression recognition device, comprising:

[0032] A data acquisition unit, used for acquiring a sequence of facial micro-expression images to be recognized;

[0033] A pixel feature extraction unit, used to respectively obtain the HLTP feature of each pixel of the facial micro-expression image sequence; wherein the HLTP feature is an image feature obtained by combining the LTP feature with the Haar feature;

[0034] A plane feature extraction unit, used for determining a feature histogram of each plane of the facial micro-expression image sequence on a spatiotemporal three-dimensional plane based on the HLTP feature of each pixel;

[0035] The expression classification unit is used to classify the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, and determine the expression category corresponding to the facial micro-expression image sequence.

[0036] Optionally, the data acquisition unit is further used for:

[0037] The facial micro-expression image sequence is enlarged, denoised and normalized.

[0038] Optionally, respectively acquiring the HLTP features of each pixel of the facial micro-expression image sequence includes:

[0039] Corresponding to each pixel of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, the following processing is performed respectively:

[0040] The feature calculation threshold of the pixel is calculated by performing convolution calculation on the HLTP coding model and the feature calculation area, wherein the feature calculation area refers to an image area of ​​a set size centered on the pixel on the plane where the pixel is located;

[0041] The HLTP feature value of the pixel is calculated and determined according to the convolution result of the HLTP coding model and the feature calculation area and the feature calculation threshold.

[0042] Optionally, determining a feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane based on the HLTP feature of each pixel includes:

[0043] For each plane of the three-dimensional space-time plane, the following processing is performed respectively:

[0044] The HLTP features of each pixel of the facial micro-expression image sequence on the plane are counted to determine a feature histogram of the facial micro-expression image sequence on the plane.

[0045] Optionally, classifying the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane to determine the expression category corresponding to the facial micro-expression image sequence includes:

[0046] Determining a feature histogram of the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane;

[0047] Inputting the feature histogram of the facial micro-expression image sequence into a pre-trained expression classification model to determine the expression category corresponding to the facial micro-expression image sequence;

[0048] The expression classification model is obtained by setting weights for the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane, and performing expression classification training on the facial micro-expression image sequence samples based on the set weights and the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane.

[0049] The facial micro-expression recognition method proposed in the embodiment of the present application is used to classify the facial micro-expression image sequence into micro-expressions by extracting the HLTP features of each pixel of the facial micro-expression image sequence and constructing the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane using the HLTP features of each pixel, thereby determining the expression category corresponding to the facial micro-expression image sequence. The above-mentioned HLTP features can capture more detailed features of pixel points, so that the method can capture the dynamic spatiotemporal texture features in the image sequence more comprehensively and meticulously. Therefore, the classification of facial micro-expressions based on the features can improve the accuracy of micro-expression classification and enhance the effect of micro-expression recognition.

[0050] Furthermore, the facial micro-expression recognition method proposed in the embodiment of the present application can also set adaptive thresholds for each pixel and set weights for the feature histograms of each plane, so as to more accurately extract the dynamic spatiotemporal texture features of the image sequence, and thus more accurately identify facial micro-expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0052] Figure 1 It is a flowchart of a method for recognizing facial micro-expressions provided in an embodiment of the present application;

[0053] Figure 2 It is a schematic diagram of the spatiotemporal three-dimensional plane of the facial micro-expression image sequence and the feature histograms on each plane provided by the embodiment of the present application;

[0054] Figure 3 is a schematic diagram of a coding model provided in an embodiment of the present application;

[0055] Figure 4 It is a flowchart of another method for recognizing facial micro-expressions provided in an embodiment of the present application;

[0056] Figure 5 It is a structural schematic diagram of a facial micro-expression recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solution of the embodiment of the present application is applied to the application scenario of recognizing facial micro-expressions in videos. By adopting the technical solution of the embodiment of the present application, facial micro-expressions in videos can be recognized more accurately.

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] This application embodiment proposes a method for recognizing micro-expressions of human faces, see Figure 1 As shown, the method includes:

[0060] S101, obtaining a sequence of facial micro-expression images to be recognized.

[0061] Specifically, the above-mentioned facial micro-expression image sequence is captured from a surveillance video and includes an image sequence of the facial micro-expression process.

[0062] The facial micro-expression image sequence may be obtained by detecting facial micro-expression from a video image sequence by detecting individual image frames or image regions containing facial micro-expressions.

[0063] Furthermore, after obtaining the above-mentioned facial micro-expression image sequence to be identified, in order to facilitate the classification and identification of facial micro-expressions in the facial micro-expression image sequence, thereby determining the type of facial micro-expressions, the embodiment of the present application further amplifies, denoises and normalizes the facial micro-expression image sequence.

[0064] Specifically, the above-mentioned facial micro-expression image sequence to be identified is subjected to Euler magnification processing, and then the facial micro-expression image sequence after Euler magnification is denoised using a median filtering algorithm. Finally, the Adaboost algorithm is used to detect faces in the facial micro-expression image sequence, and the faces are cropped, and then the image size is normalized using a bilinear interpolation algorithm.

[0065] S102, respectively obtaining HLTP features of each pixel of the facial micro-expression image sequence.

[0066] The HLTP feature is constructed based on the Haar feature and the LTP feature.

[0067] Specifically, the above-mentioned HLTP feature is an image feature type obtained by combining the LTP feature of the local three-value pattern on the basis of the Haar feature, which is a feature obtained by extracting the three-value feature of the image using the Haar feature extraction method. Compared with the binary feature, the three-value feature has better feature expression, so it can more accurately grasp the subtle features of the image and use it to recognize the micro-expressions of the face in the image.

[0068] For example Figure 2 As shown, the facial micro-expression image sequence is a three-dimensional image sequence, in which the XY plane contains two dimensions, and the time axis T is the third dimension. The facial micro-expression image sequence can be regarded as an image sequence composed of pixel points on three planes: XY, XT, and YT.

[0069] For each pixel point on each plane in the above facial expression image sequence, its HLTP feature is calculated respectively.

[0070] S103, determining a feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane based on the HLTP feature of each pixel.

[0071] Specifically, after obtaining the HLTP feature of each pixel in the facial micro-expression image sequence, according to the HLTP feature of each pixel, such as Figure 2 As shown, a feature histogram of each plane of the three-dimensional space-time planes XY, XT, and YT is constructed.

[0072] The above-mentioned feature histogram can represent the distribution of HLTP feature values ​​on each three-dimensional plane of the facial micro-expression image sequence, which can also become a dynamic spatiotemporal texture feature.

[0073] S104, classifying the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, and determining the expression category corresponding to the facial micro-expression image sequence.

[0074] Specifically, the facial micro-expression image sequence is classified into micro-expression categories based on the feature histogram of the facial micro-expression image sequence on each plane of the three-dimensional space-time plane XY, XT, and YT, so as to determine the expression category corresponding to the facial micro-expression image sequence.

[0075] From the above introduction, it can be seen that the facial micro-expression recognition method proposed in the embodiment of the present application extracts the HLTP features of each pixel of the facial micro-expression image sequence, and uses the HLTP features of each pixel to construct the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, which is used to classify the facial micro-expression image sequence, thereby determining the expression category corresponding to the facial micro-expression image sequence. The above-mentioned HLTP features can capture more detailed features of pixel points, so that the method can capture the dynamic spatiotemporal texture features in the image sequence more comprehensively and meticulously. Therefore, facial micro-expression classification based on this feature can improve the accuracy of micro-expression classification and enhance the micro-expression recognition effect.

[0076] Below, the specific processing content of each step of the facial micro-expression recognition method proposed in this application is exemplarily introduced.

[0077] Exemplarily, the above-mentioned respectively obtaining the HLTP features of each pixel of the facial micro-expression image sequence includes:

[0078] Corresponding to each pixel of the above-mentioned facial micro-expression image sequence on each plane of the three-dimensional plane of space and time, decibel performs the following processing of A1 and A2:

[0079] A1. By performing convolution calculation on the HLTP coding model and the feature calculation area, a feature calculation threshold of the pixel is calculated.

[0080] The feature calculation area refers to an image area of ​​a set size centered on the pixel on the plane where the pixel is located.

[0081] Specifically, the HLTP feature is based on 8 groups of HLTP coding models, as shown below: 1 -M 8 That is, 8 sets of coding models:

[0082]

[0083] In the above 8 groups of coding models, the weights of the 5 pixel values ​​in the outermost layer of each group of coding models are 1, the weights of the 5 pixel values ​​in the second layer are -1, and the other weights are 0.

[0084] like Figure 3 The figure shows the 8-group coding model of HLTP. It can be seen from the figure that the 2 groups of dark and light areas running through the center rotate clockwise in 8 directions and then return to the original state. The difference between the light and dark areas on the same side of the center point can effectively reflect the image feature texture change information.

[0085] The above-mentioned feature calculation area refers to the image area of ​​a set size centered on the pixel on the plane where the pixel is located when calculating the HLTP feature of the pixel. The following image area W(x,y) represents the image area centered on the pixel point P 0 The characteristic calculation area centered is:

[0086]

[0087] The texture change storing the Haar feature is the center point P 0 , that is, pixel P 0 is the pixel point for which the HLTP feature is to be calculated, and the surrounding pixel points P i (i=1, 2, ..., 24) are 24 neighborhood pixels.

[0088] For any pixel point I(x, y), a feature calculation area W(x, y) can be constructed. Then, according to the following formula, by performing convolution calculation on the above HLTP coding model and the feature calculation area of ​​the pixel point, the feature calculation threshold t of the pixel point I(x, y) is calculated:

[0089]

[0090] Among them, M k W(x,y) is the feature calculation area W(x,y) and the encoding model M k The decimal number obtained after convolution.

[0091] A2. Calculate and determine the HLTP feature value of the pixel according to the convolution result of the HLTP coding model and the feature calculation area and the feature calculation threshold.

[0092] Specifically, for the pixel point I (x, y), according to the above-mentioned HLTP coding model and the convolution result of the feature calculation area, and the feature calculation threshold t of the pixel point calculated in the above-mentioned step A1, the HLTP feature value of the pixel point I (x, y) is obtained according to the following calculation formula:

[0093]

[0094] Wherein, T(x) is the threshold comparison function of the HLTP value, and t is the threshold. Specifically, the value of T(x) is determined according to the following formula:

[0095]

[0096] From the above introduction, it can be seen that the embodiment of the present application adopts an adaptive threshold method when calculating the HLTP feature of each pixel, that is, the feature calculation threshold is determined for each pixel, and then the HLTP feature value of the pixel is calculated based on the feature calculation threshold of the pixel. Compared with the technical solution of unified feature calculation threshold, this feature calculation method can get rid of the limitation and influence of artificially set feature calculation threshold on feature extraction, thereby improving feature accuracy.

[0097] According to the above method, after respectively obtaining the HLTP feature value of each pixel point on each plane of the spatiotemporal three-dimensional plane of the facial micro-expression image sequence, the above HLTP feature based on each pixel is used to determine the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, which can be specifically achieved by the following method:

[0098] For each plane of the three-dimensional space-time plane, the following processing is performed respectively:

[0099] The HLTP features of each pixel of the facial micro-expression image sequence on the plane are counted to determine a feature histogram of the facial micro-expression image sequence on the plane.

[0100] Specifically, the HLTP features of each pixel point calculated above are counted on the XY, XT, and YT planes, and the HLTP histograms of the three planes are calculated respectively by the following calculation formula:

[0101]

[0102] Among them, the value of j is 0, 1, and 2, representing the XY, XT, and YT planes respectively, i represents the HLTP eigenvalue, and n j For the maximum decimal HLTP eigenvalue generated in the jth plane, the binary function H(x) is as follows:

[0103]

[0104] The HLTP feature histograms of the three planes XY, XT, and YT are finally calculated as follows:

[0105] H j =[H HLTP,j ]j=0,1,2

[0106] As an alternative embodiment, see Figure 4 As shown, the above-mentioned classification of the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane to determine the expression category corresponding to the facial micro-expression image sequence includes:

[0107] S404: Determine a feature histogram of the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane.

[0108] Specifically, the feature histogram of each plane is standardized according to the following formula:

[0109]

[0110] In the above formula, the denominator is the sum of the feature histograms of the three planes XY, XT, and YT, and the numerator is the feature histogram of each plane. After obtaining the standardized feature histogram of each plane, the histograms are concatenated according to the following formula to obtain the HLTP-TOP feature histogram H of the above-mentioned facial micro-expression image sequence:

[0111]

[0112] S405 , inputting the feature histogram of the facial micro-expression image sequence into a pre-trained expression classification model to determine the expression category corresponding to the facial micro-expression image sequence.

[0113] The expression classification model is obtained by setting weights for the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane, and performing expression classification training on the facial micro-expression image sequence samples based on the set weights and the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane.

[0114] Specifically, the embodiment of the present application pre-trains an expression classification model for performing expression classification on a sequence of facial micro-expression images according to a feature histogram of the sequence of facial micro-expression images.

[0115] The above-mentioned expression classification model can set weights for the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane, and perform expression classification on the facial micro-expression image sequence samples according to the set weights and the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane.

[0116] It can be understood that since the importance of the micro-expression texture information presented by each plane is different, the embodiment of the present application assigns different weights to the feature histograms of the three planes, and finally connects the weighted feature histograms of the three planes in series, which can highlight the important micro-expression information and weaken the feature weights of the planes with less micro-expression information or relatively unimportant planes, thereby better identifying micro-expressions.

[0117] Figure 4 Steps S401 to S403 in the facial micro-expression recognition method shown in FIG. Figure 1 For details of steps S101 to S103 in the method embodiment shown in FIG. Figure 1 The contents of the method embodiment shown will not be repeated here.

[0118] Exemplarily, the above expression classification model can be trained through the following process:

[0119] First, for each preset expression category, such as happiness, sadness, fear, anger, disgust, surprise, contempt, etc., the corresponding SVM classifier is trained by the following processing:

[0120] Obtain the feature histogram of facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane.

[0121] Specifically, the feature histograms of the facial micro-expression image sequence samples in the XY, XT, and YT planes may be obtained referring to the description of the above embodiments of the present application.

[0122] Weights are set for the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane, and weighted feature histograms of the facial micro-expression image sequence samples are obtained.

[0123] Specifically, the micro-expression information expressed by each plane of XY, XT, and YT is different, and the importance is also different. The embodiment of the present application proposes to assign different weights to each plane, and the sum of the weights of the three planes of XY, XT, and YT is 1, forming the HLTP-WTOP algorithm, which can highlight the plane containing rich micro-expression information and weaken the plane influence of the plane with less micro-expression information, and finally improve the micro-expression recognition rate. Here, the weight of the XY plane is a, the weight of XT is b, and the weight of the YT plane is (1-ab), and the weight vector W is:

[0124] W=(ab 1-ab)

[0125] The weighted HLTP-TOP, that is, the weighted feature histogram HLTP-WTOP of the facial micro-expression image sequence samples is as follows:

[0126] HLTP-WTOP=W·H T =[aH 0 B H 1 (1-ab)H 2 ]

[0127] The SVM classifier is used to classify facial micro-expression image sequence samples based on weighted feature histograms of the facial micro-expression image sequence samples; and the SVM classifier is used to calibrate parameters based on the expression classification results and expression category labels corresponding to the facial micro-expression image sequence samples.

[0128] Repeat the above process until the SVM classifier can accurately classify the facial micro-expression image sequence samples.

[0129] Specifically, according to the above method, a training set and a test set are constructed, and specifically, the ratio of the training set to the test set is 3: 1. Then, the SVM classifier is repeatedly trained using the training set until, during the test process, the SVM classifier can accurately classify the facial micro-expression image sequence samples.

[0130] Finally, the trained SVM classifiers corresponding to each preset expression category are combined to obtain an expression classification model.

[0131] The combined expression classification model can classify and identify each of the above-mentioned preset expression categories.

[0132] The SVM classifier is a C-SVC classifier, and the kernel function is an RBF kernel function.

[0133] Corresponding to the above-mentioned facial micro-expression recognition method, the present application embodiment also proposes a facial micro-expression recognition device, see Figure 5 As shown, the device comprises:

[0134] The data acquisition unit 100 is used to acquire a sequence of facial micro-expression images to be recognized;

[0135] The pixel feature extraction unit 110 is used to respectively obtain the HLTP feature of each pixel of the facial micro-expression image sequence; wherein the HLTP feature is an image feature obtained by combining the LTP feature with the Haar feature;

[0136] A plane feature extraction unit 120, for determining a feature histogram of each plane of the facial micro-expression image sequence on a spatiotemporal three-dimensional plane based on the HLTP feature of each pixel;

[0137] The expression classification unit 130 is used to classify the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, and determine the expression category corresponding to the facial micro-expression image sequence.

[0138] Optionally, the data acquisition unit is further used for:

[0139] The facial micro-expression image sequence is enlarged, denoised and normalized.

[0140] Optionally, respectively acquiring the HLTP features of each pixel of the facial micro-expression image sequence includes:

[0141] Corresponding to each pixel of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, the following processing is performed respectively:

[0142] The feature calculation threshold of the pixel is calculated by performing convolution calculation on the HLTP coding model and the feature calculation area, wherein the feature calculation area refers to an image area of ​​a set size centered on the pixel on the plane where the pixel is located;

[0143] The HLTP feature value of the pixel is calculated and determined according to the convolution result of the HLTP coding model and the feature calculation area and the feature calculation threshold.

[0144] Optionally, determining a feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane based on the HLTP feature of each pixel includes:

[0145] For each plane of the three-dimensional space-time plane, the following processing is performed respectively:

[0146] The HLTP features of each pixel of the facial micro-expression image sequence on the plane are counted to determine a feature histogram of the facial micro-expression image sequence on the plane.

[0147] Optionally, classifying the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane to determine the expression category corresponding to the facial micro-expression image sequence includes:

[0148] Determining a feature histogram of the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane;

[0149] Inputting the feature histogram of the facial micro-expression image sequence into a pre-trained expression classification model to determine the expression category corresponding to the facial micro-expression image sequence;

[0150] The expression classification model is obtained by setting weights for the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane, and performing expression classification training on the facial micro-expression image sequence samples based on the set weights and the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane.

[0151] Optionally, the training process of the expression classification model includes:

[0152] For each preset expression category, the corresponding SVM classifier is trained by the following processing:

[0153] Obtaining a feature histogram of facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane;

[0154] Setting weights for the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane, and obtaining weighted feature histograms of the facial micro-expression image sequence samples;

[0155] Using the SVM classifier to classify facial micro-expression image sequence samples based on the weighted feature histogram of the facial micro-expression image sequence samples;

[0156] Based on the expression classification results and the expression category labels corresponding to the facial micro-expression image sequence samples, the parameters of the SVM classifier are calibrated;

[0157] Repeat the above process until the SVM classifier can accurately classify the facial micro-expression image sequence samples;

[0158] The trained SVM classifiers corresponding to each preset expression category are combined to obtain an expression classification model.

[0159] Specifically, for the specific working contents of each unit of the above-mentioned facial micro-expression recognition device, please refer to the contents of the above-mentioned method embodiment, which will not be repeated here.

[0160] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0161] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0162] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0163] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0164] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0165] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0167] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0168] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0169] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0170] It can be understood that the facial micro-expression images disclosed in the various embodiments of the present disclosure are based on relevant laws and regulations, and are used in an appropriate manner to inform users of the type, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure and obtain the user's authorization, and the facial micro-expression images disclosed in the various embodiments of the present disclosure are legally used.

[0171] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for recognizing facial micro-expressions, characterized in that: include: Obtain a sequence of facial micro-expression images to be recognized; Respectively obtain the HLTP features of each pixel of the facial micro-expression image sequence; wherein the HLTP feature is an image feature obtained by combining the LTP feature with the Haar feature; Determine a feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane based on the HLTP feature of each pixel; Classifying the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, and determining the expression category corresponding to the facial micro-expression image sequence; Wherein, the step of respectively obtaining the HLTP features of each pixel of the facial micro-expression image sequence includes: Corresponding to each pixel of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, the following processing is performed respectively: The feature calculation threshold of the pixel is calculated by performing convolution calculation on the HLTP coding model and the feature calculation area, wherein the feature calculation area refers to an image area of ​​a set size centered on the pixel on the plane where the pixel is located; The HLTP feature value of the pixel is calculated and determined according to the convolution result of the HLTP coding model and the feature calculation area and the feature calculation threshold.

2. The method according to claim 1, characterized in that After obtaining the facial micro-expression image sequence to be recognized, the method further includes: The facial micro-expression image sequence is enlarged, denoised and normalized.

3. The method according to claim 1, characterized in that The method of determining a feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane based on the HLTP feature of each pixel comprises: For each plane of the three-dimensional space-time plane, the following processing is performed respectively: The HLTP features of each pixel of the facial micro-expression image sequence on the plane are counted to determine a feature histogram of the facial micro-expression image sequence on the plane.

4. The method according to claim 1, characterized in that The method of classifying the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane to determine the expression category corresponding to the facial micro-expression image sequence includes: Determining a feature histogram of the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane; Inputting the feature histogram of the facial micro-expression image sequence into a pre-trained expression classification model to determine the expression category corresponding to the facial micro-expression image sequence; The expression classification model is obtained by setting weights for the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane, and performing expression classification training on the facial micro-expression image sequence samples based on the set weights and the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane.

5. The method according to claim 4, characterized in that The training process of the expression classification model includes: For each preset expression category, the corresponding SVM classifier is trained by the following processing: Obtaining a feature histogram of facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane; Setting weights for the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane, and obtaining weighted feature histograms of the facial micro-expression image sequence samples; Using the SVM classifier to classify facial micro-expression image sequence samples based on the weighted feature histogram of the facial micro-expression image sequence samples; Based on the expression classification results and the expression category labels corresponding to the facial micro-expression image sequence samples, the parameters of the SVM classifier are calibrated; Repeat the above process until the SVM classifier can accurately classify the facial micro-expression image sequence samples; The trained SVM classifiers corresponding to each preset expression category are combined to obtain an expression classification model.

6. A facial micro-expression recognition device, characterized in that: include: A data acquisition unit, used for acquiring a sequence of facial micro-expression images to be recognized; A pixel feature extraction unit, used to respectively obtain the HLTP feature of each pixel of the facial micro-expression image sequence; wherein the HLTP feature is an image feature obtained by combining the LTP feature with the Haar feature; A plane feature extraction unit, used for determining a feature histogram of each plane of the facial micro-expression image sequence on a spatiotemporal three-dimensional plane based on the HLTP feature of each pixel; An expression classification unit, used to classify the facial micro-expression image sequence according to a feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, and determine the expression category corresponding to the facial micro-expression image sequence; Wherein, the step of respectively obtaining the HLTP features of each pixel of the facial micro-expression image sequence includes: Corresponding to each pixel of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane, the following processing is performed respectively: The feature calculation threshold of the pixel is calculated by performing convolution calculation on the HLTP coding model and the feature calculation area, wherein the feature calculation area refers to an image area of ​​a set size centered on the pixel on the plane where the pixel is located; The HLTP feature value of the pixel is calculated and determined according to the convolution result of the HLTP coding model and the feature calculation area and the feature calculation threshold.

7. The device according to claim 6, characterized in that The data acquisition unit is also used for: The facial micro-expression image sequence is enlarged, denoised and normalized.

8. The device according to claim 6, characterized in that The method of determining a feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane based on the HLTP feature of each pixel comprises: For each plane of the three-dimensional space-time plane, the following processing is performed respectively: The HLTP features of each pixel of the facial micro-expression image sequence on the plane are counted to determine a feature histogram of the facial micro-expression image sequence on the plane.

9. The device according to claim 6, characterized in that The method of classifying the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane to determine the expression category corresponding to the facial micro-expression image sequence includes: Determining a feature histogram of the facial micro-expression image sequence according to the feature histogram of the facial micro-expression image sequence on each plane of the spatiotemporal three-dimensional plane; Inputting the feature histogram of the facial micro-expression image sequence into a pre-trained expression classification model to determine the expression category corresponding to the facial micro-expression image sequence; The expression classification model is obtained by setting weights for the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane, and performing expression classification training on the facial micro-expression image sequence samples based on the set weights and the feature histograms of the facial micro-expression image sequence samples on each plane of the spatiotemporal three-dimensional plane.

Images