Face micro-expression recognition method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of image processing, and provides a face micro-expression recognition method and device, electronic equipment and a storage medium, the method comprising the following steps: acquiring a face micro-expression image sequence to be recognized; pre-processing the face micro-expression image sequence; extracting HCTP features of three orthogonal planes of the pre-processed face micro-expression image sequence; determining feature histograms in three dimensions based on the HCTP features; the HCTP features are image features obtained by combining CTP features on the basis of Haar features; the feature histograms in the three dimensions are standardized; the three feature histograms after the standardization are concatenated into a histogram vector; the histogram vector is input into a trained neural network model based on deep learning for classification; and a corresponding expression category is obtained. In this way, by considering the face micro-expression feature information of the three orthogonal planes and the central pixel information, comprehensive features are extracted, and the accuracy of face micro-expression recognition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a facial micro-expression recognition method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Facial expressions play a very important role in the process of human communication, but facial micro-expression is different from ordinary facial expressions, which is a facial expression with very short duration and very fast speed. Therefore, by analyzing facial micro-expression, the real emotions that people try to hide can be found, which has many potential application values in the fields of clinical, law, national security and review industry, and plays a crucial role in identifying lies.

[0003] In the prior art, the local binary patterns (LBP) features of facial micro-expression can be extracted, feature selection is performed, and then the extracted features are input into a micro-expression recognition model to classify and recognize facial micro-expression.

[0004] However, the feature extraction method in the above facial micro-expression recognition method is not comprehensive, which leads to inaccurate recognition efficiency. SUMMARY

[0005] The present application provides a facial micro-expression recognition method, device, electronic equipment and storage medium, which can extract more comprehensive feature data and improve the accuracy of facial micro-expression recognition.

[0006] In a first aspect, the present application provides a facial micro-expression recognition method, which comprises:

[0007] Obtaining a facial micro-expression image sequence to be recognized, and pre-processing the facial micro-expression image sequence to obtain a pre-processed facial micro-expression image sequence;

[0008] Extracting HCTP (Haar Center Local Tri-Value Pattern) features from three orthogonal planes of the pre-processed facial micro-expression image sequence, and determining feature histograms in three dimensions based on the HCTP features; the HCTP features are image features obtained by combining CTP (Center Local Tri-Value Pattern) features on the basis of Haar features;

[0009] Standardizing the feature histograms in the three dimensions, and concatenating the three standardized feature histograms into a histogram vector;

[0010] Inputting the histogram vector into a trained neural network model based on deep learning for classification to obtain an expression category corresponding to the facial micro-expression image sequence.

[0011] Optionally, the human face micro-expression image sequence is preprocessed to obtain a preprocessed human face micro-expression image sequence, including:

[0012] A key point of a first human face micro-expression image in the human face micro-expression image sequence is acquired by an active shape model algorithm to obtain a human face key point.

[0013] The human face micro-expression image sequence is differentially processed by taking the human face key point as a reference value by using a local weighted average algorithm to obtain a first human face micro-expression image sequence.

[0014] The first human face micro-expression image sequence is Euler amplified, and the first human face micro-expression image sequence after Euler amplification is denoised by using a Gaussian filtering algorithm to obtain a second human face micro-expression image sequence.

[0015] The second human face micro-expression image sequence is subjected to gray scale normalization processing to obtain a third human face micro-expression image sequence.

[0016] Optionally, a Haar center local ternary pattern (HCTP) feature is extracted from three orthogonal planes of the preprocessed human face micro-expression image sequence, including:

[0017] For each pixel on each orthogonal plane of the third human face micro-expression image sequence, a feature calculation threshold of the pixel is obtained by convolution calculation using an encoding model, a feature calculation region, and a center pixel region; wherein the feature calculation region is a predefined size image region centered on the pixel; and the center pixel region is a predefined size pixel region with a value corresponding to the pixel.

[0018] The HCTP feature of the pixel is calculated according to the encoding model, the convolution result of the feature calculation region, the convolution result of the center pixel region, and the feature calculation threshold.

[0019] The three orthogonal planes include an XY plane, an XT plane, and a YT plane; the XY plane is used to represent the spatial information of the human face micro-expression image sequence; the XT plane is used to represent the motion texture information of the human face micro-expression image sequence in the horizontal direction along the time axis; and the YT plane is used to represent the motion texture information of the human face micro-expression image sequence in the vertical direction along the time axis.

[0020] Optionally, the three feature histograms after normalization processing are concatenated into a histogram vector, including:

[0021] Weight values corresponding to the three orthogonal planes are obtained, and the three feature histograms after normalization processing are processed based on the weight values to obtain corresponding weighted feature histograms.

[0022] concatenate the weighted feature histograms to obtain a histogram vector.

[0023] Optionally, the method further comprises:

[0024] obtaining a training data set, the training data set comprising histogram vectors corresponding to facial micro-expression image sequences and expression categories corresponding to the histogram vectors;

[0025] training a neural network model based on deep learning according to the training data set;

[0026] The expression categories include happiness, sadness, fear, anger, disgust, surprise, and contempt.

[0027] Optionally, training the neural network model based on deep learning according to the training data set comprises:

[0028] randomly dividing the histogram vectors in the training data set and the expression categories corresponding to the histogram vectors into a training set and a test set;

[0029] training the neural network model based on deep learning using the training set and testing the trained neural network model based on deep learning using the test set to determine whether the trained neural network model based on deep learning meets a preset condition;

[0030] If yes, a trained neural network model based on deep learning is obtained.

[0031] If no, updating the weight values corresponding to the three orthogonal planes to obtain updated histogram vectors, and retraining the neural network model based on deep learning.

[0032] In a second aspect, the present application provides a facial micro-expression recognition device, the device comprising:

[0033] an acquisition module configured to acquire a facial micro-expression image sequence to be recognized, and pre-process the facial micro-expression image sequence to obtain a pre-processed facial micro-expression image sequence;

[0034] an extraction module configured to extract HCTP (Haar Center Local Tri-Value Pattern) features from three orthogonal planes of the pre-processed facial micro-expression image sequence, and determine feature histograms in three dimensions based on the HCTP features; the HCTP features are image features obtained by combining CTP (Center Local Tri-Value Pattern) features with Haar features;

[0035] a processing module configured to standardize the feature histograms in the three dimensions, and concatenate the three standardized feature histograms into a histogram vector;

[0036] The classification module is used to input the histogram vector into a trained deep learning-based neural network model for classification, so as to obtain the expression category corresponding to the facial micro-expression image sequence.

[0037] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0038] The memory stores computer-executed instructions;

[0039] The processor executes computer execution instructions stored in the memory to implement the method as described in any one of the first aspects.

[0040] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as described in any one of the first aspects.

[0041] Fifthly, this application provides a computer program product, including program code, which, when a computer runs the computer program, performs the method as described in any one of the first aspects.

[0042] In summary, this application provides a method, apparatus, electronic device, and storage medium for facial micro-expression recognition. It acquires a sequence of facial micro-expression images to be recognized and preprocesses the sequence to obtain a preprocessed sequence. Further, it extracts HCTP features from three orthogonal planes of the preprocessed sequence and determines feature histograms in three dimensions based on these HCTP features. Further, it standardizes the feature histograms in the three dimensions and concatenates them into a histogram vector. This histogram vector is then input into a trained deep learning-based neural network model for classification, yielding the expression category corresponding to the facial micro-expression image sequence. The HCTP features are image features obtained by combining Haar features with Central Local Triplet Pattern (CTP) features. Thus, by extracting the central pixel information rich in micro-expression information and the facial micro-expression feature information from three orthogonal planes, a more comprehensive set of features is obtained. Furthermore, by utilizing a deep learning-based neural network model for facial micro-expression recognition, the accuracy of facial micro-expression recognition is significantly improved. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] Figure 1An application scenario schematic diagram provided by an embodiment of the present application;

[0045] Figure 2 A flowchart of a face micro-expression recognition method provided by an embodiment of the present application;

[0046] Figure 3 A structure schematic diagram of an encoding model provided by an embodiment of the present application;

[0047] Figure 4 A structure schematic diagram of a face micro-expression recognition device provided by an embodiment of the present application;

[0048] Figure 5 A structure schematic diagram of an electronic device provided by an embodiment of the present application.

[0049] The specific embodiments of the present application have been shown through the above-described drawings, and will be described in more detail hereinafter. These drawings and the written description are not intended to restrict the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0050] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same or similar components. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0051] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the terms "first", "second", etc. are used to distinguish the same or similar items or items with basically the same function and role. For example, the first device and the second device are only used to distinguish different devices, and do not limit the order. Those skilled in the art can understand that the terms "first", "second", etc. do not limit the number and execution order, and the terms "first", "second", etc. do not necessarily mean different.

[0052] It should be noted that in the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of the words "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0053] In the present application, "at least one" means one or more, and "multiple" means two or more. The association relationship of "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0054] Facial expressions play a very important role in the process of human communication, but some facial expressions are short in duration and fleeting, and are not easy to detect. In 1969, Ekman and Friesen defined it as a new facial expression, called micro-expression. Ekman et al. conducted a series of studies on facial micro-expression, and the results showed that micro-expression plays a crucial role in identifying lies, and has many potential applications in clinical, legal, national security and review industries, etc. Therefore, the facial micro-expression recognition method has become the focus of current research.

[0055] The present application will be described below with reference to the accompanying drawings, Figure 1 An application scenario diagram is provided for the embodiments of the present application, and the facial micro-expression recognition method provided by the present application can be applied to the application scenario as shown in Figure 1 The application scenario includes a user 101, a camera device 102, an image processing platform 103, and a display device 104 corresponding to the image processing platform 103. Specifically, the camera device 102 can capture facial images of the user 101 within a period of time, and the facial images are continuous. Further, the camera device 102 sends the captured facial images to the image processing platform 103 for processing. The image processing platform 103 can extract a facial micro-expression image sequence from the facial images, and identify an expression category corresponding to the user 101 based on the facial micro-expression image sequence. Further, the image processing platform 103 can display the identified expression category on the corresponding display device 104 for viewing by other users.

[0056] It should be noted that the camera device 102 can be a separate camera device such as a camera, or a camera device installed on a terminal device such as a camera of a smart phone. The embodiments of the present application do not make specific limitations on this, and further, the image processing platform 103 can also send the identified expression category to the terminal device of the user for viewing.

[0057] The terminal device can be a wireless terminal or a wired terminal. The wireless terminal can be a device that provides voice and / or other service data connectivity to users, a handheld device with wireless connection capability, or other processing devices connected to a wireless modem. The wireless terminal can communicate with one or more core network devices via a radio access network (RAN). The wireless terminal can be a mobile terminal, such as a mobile telephone (also known as a "cellular" telephone) and a computer with a mobile termination, for example, which can be portable, pocket, hand-held, computer-included, or vehicle-mounted, and exchanges language and / or data with a radio access network. For another example, the wireless terminal can also be a personal communication service (PCS) phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), and the like. The wireless terminal can also be referred to as a system, a subscriber unit, a subscriber station, a mobile station, a mobile, a remote station, a remote terminal, an access terminal, a user terminal, a user agent, a user device, or user equipment, without limitation. Optionally, the terminal device can also be a tablet computer or the like.

[0058] In a possible implementation, the LBP features of the facial micro-expression are extracted, feature selection is performed, and then the extracted features are input into a micro-expression recognition model to classify and recognize the facial micro-expression.

[0059] However, when the LBP algorithm is used to extract features, the center pixel information and the facial micro-expression feature information in multiple planes are not considered, so that the extracted features are not comprehensive, resulting in inaccurate recognition efficiency.

[0060] It can be understood that, since the micro-expression texture information presented by the three orthogonal planes XY, XT and YT corresponding to the face micro-expression image is different in importance, by considering the face micro-expression statistical feature information of the three orthogonal planes XY, XT and YT and the center pixel information with rich micro-expression information, the important micro-expression information can be highlighted, and thus the accuracy of recognizing the face micro-expression rate can be improved.

[0061] To solve the above problems, the present application provides a face micro-expression recognition method, which extracts feature data by using a Haar center local ternary pattern (HCTP) algorithm, that is, an algorithm innovation of adding a center local ternary pattern (CTP) on the basis of Haar features; specifically, the HCTP algorithm is used to extract face micro-expression feature information of three orthogonal planes XY, XT and YT, and the center pixel information is added to the features, and then rich feature data is obtained, and further, the feature data is input into a trained neural network model based on deep learning to recognize the face micro-expression, and the expression category of the face micro-expression is obtained; in this way, by considering the center pixel information with rich micro-expression information and the face micro-expression feature information of the three orthogonal planes XY, XT and YT, the features of the face micro-expression are enriched, and the trained neural network model is used for expression detection, which greatly improves the accuracy of face micro-expression recognition.

[0062] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0063] Figure 2 A flowchart of a face micro-expression recognition method provided by an embodiment of the present application is shown in FIG. 1, which includes the following steps: Figure 2

[0064] S201, a face micro-expression image sequence to be recognized is acquired, and the face micro-expression image sequence is preprocessed to obtain a preprocessed face micro-expression image sequence.

[0065] In an embodiment of the present application, the face micro-expression image sequence can refer to an image sequence containing a face micro-expression process, which is a high-dimensional data containing various complex features such as time or space, and can be obtained by intercepting from a monitoring video or by directly shooting several frames of images, which is not limited in the present application.

[0066] ​In this step, the face micro-expression image sequence is preprocessed to remove the differences of the face in the face micro-expression image sequence in the expressionless state and redundant data, that is, by enlarging, denoising and normalizing the face micro-expression image sequence, a preprocessed face micro-expression image sequence is obtained, and the specific processing method can be determined according to the actual situation, which is not limited in the embodiments of the present application.

[0067] S202, extracting a Haar center local ternary pattern (HCTP) feature of three orthogonal planes of the preprocessed face micro-expression image sequence, and determining a feature histogram in three dimensions based on the HCTP feature; the HCTP feature is an image feature obtained by combining a center local ternary pattern (CTP) feature on the basis of a Haar feature.

[0068] In the embodiments of the present application, since the face micro-expression image sequence changes over time and space, the time feature and the space feature of the face micro-expression image sequence can be extracted, such as generating a time axis according to different time granularity, and extracting the time feature and the space feature according to the changes of the micro-expression at different time points; the time feature, the space feature and the center pixel feature constitute the HCTP feature; the space feature refers to the relationship of the pixel points or the target in the image with other pixel points in space, including the relative spatial position relationship and the absolute spatial position relationship, which is not limited in the embodiments of the present application.

[0069] The HCTP feature is an image feature type obtained by combining the CTP feature of the center local ternary pattern on the basis of the Haar feature, which is a feature obtained by extracting the ternary feature of the image and the center pixel feature using the Haar feature, and the ternary feature has better feature performance than the binary feature, and the center pixel feature can obtain more accurate image subtle features.

[0070] In this step, the HCTP feature of each pixel point on each plane in the preprocessed face micro-expression image sequence is calculated, and further, the HCTP feature of each pixel on each orthogonal plane of the face micro-expression image sequence is counted, and then the feature histogram in three dimensions is determined.

[0071] S203, standardizing the feature histogram in three dimensions, and concatenating the three feature histograms after the standardization into a histogram vector.

[0072] In the embodiment of the present application, the feature histogram can be used to represent the HCTP feature value distribution of the face micro-expression image sequence on each three-dimensional plane, which can be understood as a dynamic space-time texture feature, including the spatial feature of the face micro-expression image sequence, the motion texture feature in the horizontal direction along the time axis of the face micro-expression image sequence, and the motion texture feature in the vertical direction along the time axis.

[0073] In this step, the feature histograms in three dimensions can be standardized according to a predefined formula, and then the three standardized feature histograms are concatenated into a histogram vector. The predefined formula is not specifically limited in the embodiment of the present application.

[0074] S204, inputting the histogram vector into the trained deep learning-based neural network model for classification to obtain the expression category corresponding to the face micro-expression image sequence.

[0075] In the embodiment of the present application, the neural network model can refer to a neural network learning algorithm, which includes an input layer, an intermediate layer and an output layer. The intermediate layer can be expanded to multiple layers. The neurons between adjacent layers are fully connected, and there is no connection between the neurons in each layer. Each neuron generates a connection weight (Weight) after obtaining an input response, and then corrects each connection weight layer by layer from the output layer through each intermediate layer in the direction of reducing the error between the desired output and the actual output, and returns to the input layer. The deep learning-based neural network model refers to the above process repeatedly alternating until the global error of the network tends to a given minimum value.

[0076] In this step, the histogram vector is input into the trained deep learning-based neural network model for classification, that is, the feature histograms of the three planes are taken as the input layer and input into the trained deep learning-based neural network model for classification. The neural network model contains 3 input layers, 1 hidden layer and 1 output layer. Further, the expression category corresponding to the face micro-expression image sequence is obtained. The expression category can refer to the facial micro-expression expressing emotion, including happy, sad, afraid, angry, disgusted, surprised, contemptuous, etc. The types of expressions included in the expression category are not specifically limited in the embodiment of the present application.

[0077] Therefore, the present application provides a facial micro-expression recognition method, which can obtain a facial micro-expression image sequence to be recognized, and pre-process the facial micro-expression image sequence to obtain a pre-processed facial micro-expression image sequence. Further, HCTP features are extracted from three orthogonal planes of the pre-processed facial micro-expression image sequence, and feature histograms in three dimensions are determined based on the HCTP features. Further, the feature histograms in three dimensions are standardized, and the three standardized feature histograms are concatenated into a histogram vector. Then, the histogram vector is input into a trained neural network model based on deep learning for classification to obtain an expression category corresponding to the facial micro-expression image sequence. The HCTP feature is an image feature obtained by combining a center local ternary pattern (CTP) feature with a Haar feature. In this way, the center pixel information and the facial micro-expression feature information of three orthogonal planes are extracted to obtain comprehensive features. The facial micro-expression is recognized by using the neural network model based on deep learning, which greatly improves the accuracy of facial micro-expression recognition.

[0078] Optionally, the facial micro-expression image sequence is pre-processed to obtain a pre-processed facial micro-expression image sequence, including:

[0079] The first facial micro-expression image of the facial micro-expression image sequence is subjected to key point acquisition by an active shape model algorithm to obtain facial key points.

[0080] The facial micro-expression image sequence is subjected to differential processing by using a local weighted average algorithm with the facial key points as reference values to obtain a first facial micro-expression image sequence.

[0081] The first facial micro-expression image sequence is subjected to Euler magnification, and the first facial micro-expression image sequence subjected to Euler magnification is subjected to noise reduction processing by using a Gaussian filtering algorithm to obtain a second facial micro-expression image sequence.

[0082] The second facial micro-expression image sequence is subjected to gray scale normalization processing to obtain a third facial micro-expression image sequence.

[0083] In the present application, the active shape model (ASM) algorithm is a key point extraction algorithm for a facial model, which usually needs to extract 68 points, which are respectively fixedly distributed on each position on the facial image. It can be understood that 68 key points of the facial image are extracted. Further, the 68 extracted key points are used as reference values, and the other facial micro-expression images after the first facial micro-expression image of the facial micro-expression image sequence are subjected to differential processing by using a local weighted average algorithm, which is used to eliminate the differences between different facial images and different sequences in the expressionless state.

[0084] The local average weighting algorithm can refer to an algorithm that takes the distance between samples and a training sample set as the weight of the corresponding training sample, and acts on the samples in the training sample set after the first sample to eliminate the differences under different samples, and the training sample weight is defined as the smaller the distance, the greater the weight.

[0085] Further, the first facial micro-expression image sequence is subjected to Euler amplification processing, and the first facial micro-expression image sequence subjected to Euler amplification is denoised by using a Gaussian filtering algorithm to obtain a second facial micro-expression image sequence, and then the second facial micro-expression image sequence is subjected to gray scale normalization processing to obtain a third facial micro-expression image sequence.

[0086] Optionally, the second facial micro-expression image sequence can also be subjected to gray scale normalization and scale normalization processing. The gray scale and scale of the processed facial micro-expression image sequence can meet the subsequent processing requirements through gray scale normalization and scale normalization. Redundant data of the first facial micro-expression image sequence can also be removed through a time domain interpolation model or a median filtering algorithm. The face in the facial micro-expression image sequence can also be detected by using an Adaboost algorithm, and the face is cropped, and then the image size is normalized by using a bilinear difference algorithm. The embodiments of the present application do not make specific limitations in this regard.

[0087] Therefore, in order to identify the facial micro-expression in the facial micro-expression image sequence, a series of preprocessing operations are performed, which can reduce the influence of non-expression images and interference images on feature extraction, is conducive to improving the accuracy of micro-expression recognition, and can also facilitate subsequent data processing and speed up the processing rate.

[0088] Optionally, the HCTP features of the three orthogonal planes of the preprocessed facial micro-expression image sequence are extracted, including:

[0089] For each pixel on each orthogonal plane of the third facial micro-expression image sequence, convolution calculation is performed by using an encoding model, a feature calculation region, and a center pixel region to obtain a feature calculation threshold of the pixel; wherein the feature calculation region is a predefined size image region centered on the pixel; and the center pixel region is a predefined size pixel region with the value corresponding to the pixel.

[0090] The HCTP feature of the pixel is calculated according to the convolution result of the encoding model, the convolution result of the feature calculation region, the convolution result of the center pixel region, and the feature calculation threshold.

[0091] The three orthogonal planes include the XY plane, the XT plane, and the YT plane; the XY plane is used to represent the spatial information of the facial micro-expression image sequence, the XT plane is used to represent the motion texture information of the facial micro-expression image sequence along the horizontal direction of the time axis, and the YT plane is used to represent the motion texture information of the facial micro-expression image sequence along the vertical direction of the time axis.

[0092] In this embodiment of the application, the encoding model may refer to eight pre-defined encoding models, denoted by M. k This is represented as shown in the following formula:

[0093]

[0094]

[0095]

[0096] In the above 8 coding models, taking M2 as an example, the weight of the outermost 5 pixel values ​​is 1, the weight of the second layer 5 pixel values ​​is -1, and the weight of other layers is 0. The definitions of M1-M8 are similar and will not be repeated here.

[0097] For example, Figure 3 This is a schematic diagram of the structure of an encoding model provided in an embodiment of this application, such as... Figure 3 As shown, Figure 3 The diagram shows the 8-group coding model of HCTP, consisting of... Figure 3 It can be seen that the two groups of dark and light areas that run through the center return to their original state after rotating clockwise in eight directions. The difference between the light and dark areas on the same side of the center point can effectively reflect the changes in image feature texture.

[0098] The feature calculation region can refer to a predefined image region centered on the pixel on the plane where the pixel is located when calculating the HCTP feature of the pixel. As shown in the formula W(x,y), it is a 5x5 window centered on pixel P0, which represents the feature calculation region. Pixel P0 is the center point of the texture changes storing Haar features and is the pixel whose HCTP feature is to be calculated. i (i = 1, 2, ..., 24) are 24 neighboring pixels.

[0099]

[0100] The center pixel region can refer to a pixel region of a predefined size set with the value corresponding to the pixel on the plane where the pixel is located when calculating the HCTP feature of the pixel, as shown in the following formula C(x, y), which is a 5*5 small window composed of the pixel point P0, i.e. the center pixel region.

[0101]

[0102] Therefore, it can be known that any pixel point I(x, y) in the image has a small window W(x, y) around it, and there is also a C(x, y) formed by the center point P0. Since the center pixel point itself has rich feature texture information, the influence of the center pixel point is added, and then, according to the following formula, the feature calculation threshold t of the pixel point I(x, y) is calculated by convolution calculation on the HCTP encoding model, the feature calculation region of the pixel point, and the center pixel region of the pixel point. The feature calculation threshold t is an adaptive threshold:

[0103]

[0104] Wherein, M k is an encoding model, t is a feature calculation threshold, M k ·W(x, y) is a small window W(x, y) and an encoding model M k The decimal number obtained after convolution, M k ·C(x, y) is a small window C(x, y) and an encoding model M k The decimal number obtained after convolution.

[0105] Further, for the pixel point I(x, y), according to the convolution results of the HCTP encoding model and the feature calculation region, the convolution results of the center pixel region, and the feature calculation threshold t of the pixel point calculated above, the HCTP feature value of the pixel point I(x, y) can be obtained according to the following calculation formula:

[0106]

[0107] Wherein, T(x) is a threshold comparison function of the HCTP value, and the value of T(x) is determined according to the following formula:

[0108]

[0109] Therefore, in the embodiment of the present application, an adaptive threshold method is adopted when calculating the HCTP feature of each pixel point, that is, the feature calculation threshold of each pixel point is determined respectively, and then the HCTP feature value of the pixel point is calculated based on the feature calculation threshold of the pixel point. Compared with the technical solution of a unified feature calculation threshold, the feature extraction is less limited and affected by the artificially set feature calculation threshold, thereby improving the accuracy of the extracted feature.

[0110] Optionally, the three feature histograms subjected to the normalization processing are concatenated into a histogram vector, including:

[0111] The weight values corresponding to the three orthogonal planes are obtained, and the three feature histograms subjected to the normalization processing are processed based on the weight values respectively to obtain corresponding weighted feature histograms;

[0112] The weighted feature histograms are concatenated to obtain a histogram vector.

[0113] In the embodiment of the present application, the weight values corresponding to the three orthogonal planes can refer to the optimal weight of each plane obtained by training a neural network model based on deep learning, which is used to highlight the important micro-expression information of different orthogonal planes, or refer to the weight values artificially set based on a large number of experiments, which are not limited in the embodiment of the present application.

[0114] Optionally, the neural network model based on deep learning can set the weight of the feature histogram of the face micro-expression image sequence sample on each plane of the three orthogonal planes, and perform expression classification on the face micro-expression image sequence sample according to the set weight and the feature histogram of the face micro-expression image sequence sample subjected to the normalization processing on each plane of the three orthogonal planes.

[0115] Specifically, after obtaining the HCTP feature value of each pixel point on each plane of the three orthogonal planes of the face micro-expression image sequence respectively, the HCTP feature of each pixel point obtained on each plane of the three orthogonal planes is counted on the XY, XT and YT planes respectively, and the HCTP histogram of the three orthogonal planes is calculated respectively by using the following formula, and the HCTP histogram of each plane is defined as follows:

[0116]

[0117] Wherein, the value of j is 0, 1 and 2, which respectively represents the XY, XT and YT planes, i represents the HCTP feature value, n j is the maximum decimal number generated by the HCTP feature on the jth plane, and H(x) represents a binary function, which is represented by the following formula:

[0118]

[0119] Further, the HCTP feature histogram of the three orthogonal planes XY, XT and YT is calculated respectively by using the following formula:

[0120] H j = [HCTP j ] j = 0, 1, 2

[0121] Further, the feature histogram of each plane is normalized by using the following formula:

[0122]

[0123] After obtaining the normalized feature histogram of each plane, the feature histogram can be concatenated according to the following formula to obtain the HCTP-TOP (Haar Center Ternary Patterns Three Orthogonal Planes) feature histogram H of the face micro-expression image sequence:

[0124]

[0125] Optionally, after obtaining the three normalized feature histograms, the weight values corresponding to the three orthogonal planes can be obtained, and the three normalized feature histograms are processed based on the weight values to obtain the corresponding weighted feature histograms. Further, the weighted feature histograms are concatenated to obtain the HCTP-WTOP (Haar Center Ternary Patterns Weighted Three Orthogonal Planes) feature histogram H' (histogram vector) of the face micro-expression image sequence.

[0126] It can be understood that, since the micro-expression texture information presented by each orthogonal plane is different in importance, the feature histograms of the three orthogonal planes are respectively assigned different weights in the embodiments of the present application, and finally the weighted feature histograms of the three orthogonal planes are concatenated, which can highlight important micro-expression information, reduce the feature weight of the plane with less or relatively unimportant micro-expression information, and better identify micro-expression.

[0127] Therefore, the embodiments of the present application set different weight values for each plane when extracting the dynamic spatiotemporal texture features of the face micro-expression image sequence, which improves the accuracy of obtaining feature data.

[0128] Optionally, the method further comprises:

[0129] obtain a training data set, the training data set including a histogram vector corresponding to a facial micro-expression image sequence and an expression category corresponding to the histogram vector;

[0130] train a neural network model based on deep learning according to the training data set;

[0131] The expression category includes happy, sad, scared, angry, disgusted, surprised, and contemptuous.

[0132] In this step, the facial micro-expression image sequence after the facial micro-expression image is preprocessed can be classified, including happy, sad, scared, angry, disgusted, surprised, and contemptuous. Further, the training data set corresponding to the facial micro-expression image sequence, i.e., the feature histogram in the XY, XT, and YT planes, is obtained, and the feature histogram of the facial micro-expression image sequence in each plane of the three orthogonal planes in the training data set is set with a weight, and then the corresponding weighted feature histogram (histogram vector) is obtained. Each weighted feature histogram corresponds to an expression category. Further, the obtained training data set is used to repeatedly train the neural network model based on deep learning until the neural network model based on deep learning can accurately classify the expression of the facial micro-expression image sequence.

[0133] It can be understood that the model trained by the training data set can also be other expression classification models, such as a support vector machine (SVM) classifier, a convolutional neural network model, etc., which are not specifically limited by the embodiments of the present application.

[0134] Therefore, the embodiments of the present application can train the neural network model based on deep learning to improve the accuracy of the model in detecting the expression category.

[0135] Optionally, training the neural network model based on deep learning according to the training data set includes:

[0136] randomly dividing the histogram vector in the training data set and the expression category corresponding to the histogram vector into a training set and a test set;

[0137] training the neural network model based on deep learning using the training set, and testing whether the trained neural network model based on deep learning meets a preset condition using the test set;

[0138] If yes, the trained neural network model based on deep learning is obtained.

[0139] If no, the weight values corresponding to the three orthogonal planes are updated to obtain an updated histogram vector, and the neural network model based on deep learning is retrained.

[0140] In the embodiment of the present application, the preset condition can refer to a condition set for determining the accuracy of the neural network model in identifying facial micro-expressions. The preset condition can be based on the weight value of each orthogonal plane and / or the accuracy of the neural network model in distinguishing which type of facial micro-expression belongs to. For example, the preset condition can be that when the neural network model based on deep learning is used to predict the type of facial micro-expression, the accuracy is greater than a certain threshold. The content of the preset condition set in the embodiment of the present application is not limited.

[0141] In this step, 70% of the training data set can be selected as a training set for model training, and 30% of the training data set can be selected as a test set for model testing. Specifically, after the neural network model based on deep learning is trained using the training set, a trained neural network model based on deep learning is obtained. Then, the trained neural network model based on deep learning is tested using the test set to determine whether it meets the preset condition. If not, the weight values corresponding to the three orthogonal planes can be updated, and a new training data set is obtained. The neural network model based on deep learning is trained again using the new training data set until the optimal weight of each orthogonal plane is obtained and the optimal model is trained.

[0142] Therefore, in the embodiment of the present application, the neural network model is used to train and predict the three plane micro-expression features, and the weight of each plane is constantly updated, which is less affected by human factors. Therefore, the optimal weight of each plane can be designed to highlight important micro-expression information, improve the recognition rate of facial micro-expressions, and train the optimal model for facial micro-expression recognition.

[0143] It should be noted that the method of using the trained neural network model based on deep learning to recognize facial micro-expression image sequences can also reduce the influence of light, obstacles, and noise, thereby improving the recognition rate of micro-expressions.

[0144] In the foregoing embodiments, the facial micro-expression recognition method provided by the embodiment of the present application is introduced. In order to realize each function in the method provided by the embodiment of the present application, the electronic device as an execution subject can include a hardware structure and / or a software module to realize the above-mentioned functions in the form of hardware structure, software module, or hardware structure plus software module. Whether a certain function in the above-mentioned functions is executed in the form of hardware structure, software module, or hardware structure plus software module depends on the specific application of the technical solution and the design constraint conditions.

[0145] For example, Figure 4 The structure diagram of a facial micro-expression recognition device provided by the embodiment of the present application is shown in FIG. 1. Figure 4As shown, the apparatus comprises: an acquisition module 410, an extraction module 420, a processing module 430 and a classification module 440; wherein the acquisition module 410 is configured to acquire a face micro-expression image sequence to be recognized, and pre-process the face micro-expression image sequence to obtain a pre-processed face micro-expression image sequence;

[0146] The extraction module 420 is configured to extract Haar center local ternary pattern (HCTP) features from three orthogonal planes of the pre-processed face micro-expression image sequence, and determine feature histograms in three dimensions based on the HCTP features; the HCTP features are image features obtained by combining center local ternary pattern (CTP) features on the basis of Haar features;

[0147] The processing module 430 is configured to perform standardization processing on the feature histograms in the three dimensions, and concatenate the three feature histograms subjected to the standardization processing into a histogram vector;

[0148] The classification module 440 is configured to input the histogram vector into a trained deep learning-based neural network model for classification, to obtain an expression category corresponding to the face micro-expression image sequence.

[0149] Optionally, the acquisition module 410 is specifically configured to:

[0150] perform key point acquisition on a first face micro-expression image of the face micro-expression image sequence by using an active shape model algorithm, to obtain face key points;

[0151] perform differential processing on the face micro-expression image sequence by using a local weighted average algorithm, taking the face key points as reference values, to obtain a first face micro-expression image sequence;

[0152] perform Euler magnification on the first face micro-expression image sequence, and perform noise reduction processing on the first face micro-expression image sequence subjected to the Euler magnification by using a Gaussian filtering algorithm, to obtain a second face micro-expression image sequence;

[0153] perform gray scale normalization processing on the second face micro-expression image sequence, to obtain a third face micro-expression image sequence.

[0154] Optionally, the extraction module 420 is specifically configured to:

[0155] for each pixel on each orthogonal plane of the third face micro-expression image sequence, perform convolution calculation by using an encoding model, a feature calculation region and a center pixel region, to obtain a feature calculation threshold value of the pixel; wherein the feature calculation region is an image region of a predefined size set with the pixel as the center; and the center pixel region is a pixel region of a predefined size set with a value corresponding to the pixel.

[0156] calculate the HCTP feature of the pixel according to the coding model, the convolution result of the feature calculation region, the convolution result of the center pixel region and the feature calculation threshold;

[0157] The three orthogonal planes include an XY plane, an XT plane and a YT plane; the XY plane is used to represent spatial information of the facial micro-expression image sequence; the XT plane is used to represent motion texture information of the facial micro-expression image sequence in a horizontal direction along a time axis; and the YT plane is used to represent motion texture information of the facial micro-expression image sequence in a vertical direction along the time axis.

[0158] Optionally, the processing module 430 is specifically configured to:

[0159] obtain weight values corresponding to the three orthogonal planes, and respectively process the three normalized feature histograms based on the weight values to obtain corresponding weighted feature histograms;

[0160] concatenate the weighted feature histograms to obtain a histogram vector.

[0161] Optionally, the device further includes a collecting module and a training module.

[0162] The collecting module is configured to obtain a training data set, and the training data set includes histogram vectors corresponding to facial micro-expression image sequences and expression categories corresponding to the histogram vectors.

[0163] The training module is configured to train a neural network model based on deep learning according to the training data set.

[0164] The expression categories include happiness, sadness, fear, anger, disgust, surprise and contempt.

[0165] Optionally, the training module is specifically configured to:

[0166] randomly divide the histogram vectors in the training data set and the expression categories corresponding to the histogram vectors into a training set and a test set;

[0167] train the neural network model based on deep learning by using the training set, and test whether the trained neural network model based on deep learning meets a preset condition by using the test set;

[0168] If yes, the trained neural network model based on deep learning is obtained.

[0169] If no, the weight values corresponding to the three orthogonal planes are updated to obtain updated histogram vectors, and the neural network model based on deep learning is retrained.

[0170] The specific implementation principles and effects of the face micro-expression recognition device provided in the embodiments of the present application can be referred to the related descriptions and effects of the corresponding embodiments described above, and will not be repeated here.

[0171] The embodiments of the present application also provide a structural schematic diagram of an electronic device, Figure 5 As shown in the structural schematic diagram of the electronic device provided in the embodiments of the present application, Figure 5 The electronic device can include a processor 501 and a memory 502 in communication with the processor; the memory 502 stores a computer program; the processor 501 executes the computer program stored in the memory 502, so that the processor 501 executes the method described in any of the above embodiments.

[0172] The memory 502 and the processor 501 can be connected through a bus 503.

[0173] The embodiments of the present application also provide a computer readable storage medium, which stores computer program execution instructions, and the computer program execution instructions are executed by a processor to implement the method in any of the above embodiments.

[0174] The embodiments of the present application also provide a chip running instruction, which is used to execute the method described in any of the above embodiments executed by the electronic device in any of the above embodiments.

[0175] The embodiments of the present application also provide a computer program product, which includes program codes, and when a computer runs the computer program, the program codes execute the method described in any of the above embodiments executed by the electronic device in any of the above embodiments.

[0176] In the technical solutions of the present application, the collection, storage, use, processing, transmission, provision and disclosure of the information such as financial data or user data comply with the relevant legal regulations and do not violate public order and good customs.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the modules is only a logical function division. There can be another division manner in actual implementation, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual elements can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0178] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to implement the embodiments of the present application.

[0179] In addition, the functional modules in each embodiment of the present application can be integrated in one processing unit, or each module can be physically present alone, or two or more modules can be integrated in one unit. The unit formed by the above modules can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0180] The integrated modules realized in the form of software functional modules can be stored in a computer readable storage medium. The software functional modules stored in a storage medium include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the method described in each embodiment of the present application.

[0181] It should be understood that the above processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0182] The memory can include a high-speed random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0183] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0184] The storage medium described above can be realized by any type of volatile or nonvolatile storage devices or a combination thereof, such as a Static Random-Access Memory (SRAM), an Electrically Erasable Programmable Read Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic storage, a flash memory, a magnetic disk, or an optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0185] An exemplary storage medium is coupled to the processor so that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can be part of the processor. The processor and the storage medium can be located in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the storage medium can exist as discrete components in the electronic device or host device.

[0186] The above description is merely illustrative of the embodiments of the present application, and the protection scope of the present application is not limited thereto. Any change or substitution within the technical scope disclosed in the embodiments of the present application should be covered within the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

1. A method for recognizing human face micro-expression, characterized in that, The method comprises: obtaining a face micro-expression image sequence to be recognized, and preprocessing the face micro-expression image sequence to obtain a preprocessed face micro-expression image sequence; extracting Haar center local ternary pattern (HCTP) features from three orthogonal planes of the preprocessed face micro-expression image sequence, and determining feature histograms in three dimensions based on the HCTP features; the HCTP features are image features obtained by combining a center local ternary pattern (CTP) feature with a Haar feature; standardizing the feature histograms in the three dimensions, and concatenating the three standardized feature histograms into a histogram vector; inputting the histogram vector into a trained deep learning-based neural network model for classification to obtain an expression category corresponding to the face micro-expression image sequence; wherein the HCTP features are extracted from the three orthogonal planes of the preprocessed face micro-expression image sequence, comprising: for each pixel on each orthogonal plane of the third face micro-expression image sequence, performing convolution calculation on the pixel using an encoding model, a feature calculation region, and a center pixel region to obtain a feature calculation threshold value of the pixel; wherein the feature calculation region is a predefined size image region centered on the pixel; the center pixel region is a predefined size pixel region set with the value corresponding to the pixel; the third face micro-expression image sequence is the preprocessed face micro-expression image sequence; calculating the HCTP feature of the pixel according to the convolution results of the encoding model, the feature calculation region, the convolution results of the center pixel region, and the feature calculation threshold value; wherein the three orthogonal planes include an XY plane, an XT plane, and a YT plane; the XY plane is used to represent the spatial information of the face micro-expression image sequence, the XT plane is used to represent the motion texture information of the face micro-expression image sequence in the horizontal direction along the time axis, and the YT plane is used to represent the motion texture information of the face micro-expression image sequence in the vertical direction along the time axis.

2. The method of claim 1, wherein, The preprocessing of the face micro-expression image sequence to obtain the preprocessed face micro-expression image sequence comprises: obtaining face key points by an active shape model algorithm on a first face micro-expression image of the face micro-expression image sequence; differentially processing the face micro-expression image sequence based on the face key points as a reference value by a local weighted average algorithm to obtain a first face micro-expression image sequence; performing Euler amplification on the first face micro-expression image sequence, and performing noise reduction processing on the Euler-amplified first face micro-expression image sequence by a Gaussian filtering algorithm to obtain a second face micro-expression image sequence; performing gray scale normalization processing on the second face micro-expression image sequence to obtain a third face micro-expression image sequence.

3. The method of claim 1, wherein, The concatenation of the three standardized feature histograms into a histogram vector comprises: obtaining weight values corresponding to the three orthogonal planes, and processing the three standardized feature histograms based on the weight values to obtain corresponding weighted feature histograms; The weighted feature histograms are concatenated to obtain a histogram vector.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: obtaining a training data set, the training data set comprising histogram vectors corresponding to facial micro-expression image sequences and expression categories corresponding to the histogram vectors; training a neural network model based on deep learning according to the training data set; wherein the expression categories comprise happy, sad, scared, angry, disgusted, surprised and contemptuous.

5. The method of claim 4, wherein, The training of the neural network model based on deep learning according to the training data set comprises: randomly dividing the histogram vectors in the training data set and the expression categories corresponding to the histogram vectors into a training set and a test set; training the neural network model based on deep learning using the training set and testing the trained neural network model based on deep learning using the test set to determine whether the trained neural network model based on deep learning meets a preset condition; if yes, obtaining the trained neural network model based on deep learning; if no, updating the weight values corresponding to the three orthogonal planes to obtain updated histogram vectors, and retraining the neural network model based on deep learning.

6. A human face micro-expression recognition apparatus, characterized by, The device comprises: an obtaining module configured to obtain a facial micro-expression image sequence to be recognized, and pre-process the facial micro-expression image sequence to obtain a pre-processed facial micro-expression image sequence; an extracting module configured to extract Haar center local ternary pattern (HCTP) features from three orthogonal planes of the pre-processed facial micro-expression image sequence, and determine feature histograms in three dimensions based on the HCTP features; the HCTP features are image features obtained by combining a center local ternary pattern (CTP) feature with a Haar feature; a processing module configured to standardize the feature histograms in the three dimensions, and concatenate the three standardized feature histograms into a histogram vector; a classification module configured to input the histogram vector into a trained neural network model based on deep learning for classification to obtain an expression category corresponding to the facial micro-expression image sequence; wherein the extraction of the HCTP features from the three orthogonal planes of the pre-processed facial micro-expression image sequence comprises: for each pixel on each orthogonal plane of a third facial micro-expression image sequence, performing convolution calculation using an encoding model, a feature calculation region and a center pixel region to obtain a feature calculation threshold of the pixel; the feature calculation region is a predefined size image region centered on the pixel; the center pixel region is a predefined size pixel region with a value corresponding to the pixel; the third facial micro-expression image sequence is the pre-processed facial micro-expression image sequence; calculating the HCTP feature of the pixel according to the convolution results of the encoding model, the feature calculation region and the center pixel region, and the feature calculation threshold; Three orthogonal planes include an XY plane, an XT plane, and a YT plane; the XY plane is used to represent spatial information of the facial micro-expression image sequence; the XT plane is used to represent motion texture information of the facial micro-expression image sequence in the horizontal direction along a time axis; and the YT plane is used to represent motion texture information of the facial micro-expression image sequence in the vertical direction along the time axis.

7. An electronic device, comprising: Comprise: a processor, and a memory connected to the processor in communication; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which are executed by the processor to implement the method of any one of claims 1-5.

9. A computer program product, characterised in that, comprise program code, when the computer runs the computer program, the program code executes the method of any one of claims 1-5.

Images