A facial micro-expression recognition model and recognition method

Abstract

The present invention relates to the field of image processing technology, and more specifically to a method for recognizing micro-expressions on a human face, comprising extracting a feature vector to be recognized from a micro-expression image to be recognized, comparing the feature vector to be recognized with a preset feature vector of each preset micro-expression in a micro-expression library for similarity, and sorting them according to similarity probabilities. The method also comprises processing a surrounding image and identifying whether other people exist in the surrounding image. If so, further identifying the expression types of the other people in the surrounding image, and determining the micro-expression category in the micro-expression image to be recognized based on the expression types of the other people and the similarity probability sorting. The present invention also provides a model for recognizing micro-expressions on a human face. The present invention improves the accuracy of micro-expression recognition by using the recognition of the surrounding image as a supplementary reference.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a facial micro-expression recognition model and a recognition method. Background Art

[0002] Microexpressions are a term used in psychology. People use facial expressions to convey their inner feelings to others. Between different expressions, or within a single expression, the face can "leak" other information. Microexpressions can last as short as 1 / 25 of a second. While a subconscious expression may only last a moment, it can easily reveal an emotion.

[0003] Micro-expressions may be the most advantageous clues to judge a person's true emotions. Micro-expressions have important application value in judicial interrogation, clinical medicine and other fields. Therefore, the study of micro-expressions has important significance and potential value, and it is necessary to strengthen the study of micro-expressions.

[0004] Currently, existing techniques involve creating a micro-expression library that has been pre-classified into various preset micro-expressions. The method then compares the feature vector of the micro-expression image to be identified with the feature vectors of each preset micro-expression in the library to determine a similarity probability. The similarity probability is then determined to determine the category of the micro-expression in the image to be identified. However, this recognition method can still suffer from inaccurate recognition. Summary of the Invention

[0005] The present invention aims to provide a method for recognizing micro-expressions on human faces, so as to improve the accuracy of micro-expression recognition by recognizing images of the surrounding environment as a supplementary reference.

[0006] A method for recognizing facial micro-expressions includes: extracting a feature vector to be recognized from a micro-expression image to be recognized, comparing the feature vector to be recognized with a preset feature vector of each preset micro-expression in a micro-expression library for similarity, and sorting them according to similarity probability; processing a surrounding image, and identifying whether other people exist in the surrounding image; if so, further identifying the expression type of the other people in the surrounding image; and determining the micro-expression category in the micro-expression image to be recognized based on the expression type of the other people and the similarity probability sorting.

[0007] The beneficial effects of the present invention are as follows:

[0008] 1. The present invention compares the similarity between the feature vector to be identified and the preset feature vectors of each preset micro-expression in the micro-expression library, and sorts them according to similarity probability, sorting them from highest to lowest. The ones with the highest similarity probability are most likely to be the type of the micro-expression to be identified. Compared with the existing technology, which determines whether the similarity probability is greater than a preset threshold to determine the type of the micro-expression in the micro-expression image to be identified, this method may cause problems if two or more similarity probabilities are greater than the preset threshold at the same time. In this case, the type of the micro-expression to be identified cannot be determined, or the identification may be inaccurate.

[0009] 2. On the basis of sorting the similarity probabilities, in order to determine which is the most accurate type of micro-expression to be identified, the present invention further combines the surrounding environment image, processes the surrounding environment image, and identifies whether there are other people in the surrounding environment image. If so, the expression type of other people in the surrounding environment image is further identified, and the micro-expression category in the micro-expression image to be identified is determined according to the expression type and similarity probability of other people. This method can greatly improve the accuracy of micro-expression recognition.

[0010] Furthermore, if there is only one other person in the surrounding environment image, and the expression type of this other person matches the micro-expression category with the highest similarity probability in the similarity probability sorting, then the micro-expression category with the highest similarity probability in the similarity probability sorting is determined to be the micro-expression category in the micro-expression image to be identified this time.

[0011] Beneficial effect: If there is only one other person in the surrounding environment image, and the expression type of this other person matches the micro-expression category with the highest similarity probability in the similarity probability sorting, it means that the micro-expression category with the highest similarity probability in the similarity probability sorting is accurate. Therefore, it can be determined that the micro-expression category with the highest similarity probability in the similarity probability sorting is the micro-expression category in the micro-expression image to be identified this time.

[0012] Furthermore, if there is only one other person in the surrounding environment image, the expression type of this other person matches the micro-expression category with the second highest similarity probability in the similarity probability sorting, and the difference between the micro-expression category with the highest similarity probability and the micro-expression category with the second highest similarity probability in the similarity probability sorting is less than the set threshold, then the micro-expression category with the second highest similarity probability in the similarity probability sorting is determined to be the micro-expression category in the micro-expression image to be identified this time.

[0013] Beneficial effect: If there is only one other person in the surrounding environment image, the expression type of this other person matches the micro-expression category with the second highest similarity probability in the similarity probability sorting, rather than the micro-expression category with the highest similarity probability in the similarity probability sorting, it means that it is inaccurate to use the micro-expression category with the highest similarity probability in the similarity probability sorting as the category of the micro-expression to be identified this time. If the difference between the micro-expression category with the highest similarity probability and the micro-expression category with the second highest similarity probability in the similarity probability sorting is large, it means that the micro-expression category with the second highest similarity probability in the similarity probability sorting is also inaccurate. Only when the difference between the two is not large, that is, the difference between the micro-expression category with the highest similarity probability and the micro-expression category with the second highest similarity probability in the similarity probability sorting is less than the set threshold, is the micro-expression category with the second highest similarity probability in the similarity probability sorting determined to be the micro-expression category in the micro-expression image to be identified this time, thereby improving the reliability and accuracy of recognition.

[0014] Furthermore, if there are multiple other people in the surrounding environment image, after identifying the expression type of each other person, each expression type is counted, and the expression type that accounts for the majority of the statistics is used as the final expression type of the surrounding environment image. The micro-expression category in the micro-expression image to be identified is determined based on the final expression type and the similarity probability sorting.

[0015] Beneficial effect: If there are multiple other people in the surrounding environment image, the expression type of each person may be slightly different. For example, if one person is the object of a joke, this person may have a more depressed expression, while other people are bystanders and may appear happier. Therefore, after identifying the expression type of each other person, each expression type is counted, and the expression type that accounts for the majority of the statistics is used as the final expression type of the surrounding environment image to determine the main atmosphere of the surrounding environment. The person in the micro-expression image to be identified in this atmosphere must be consistent with the atmosphere of the surrounding environment. Therefore, the final expression type is used as a supplementary reference, and the micro-expression category in the micro-expression image to be identified is determined according to the final expression type and the similarity probability sorting, which can improve the accuracy of the micro-expressions to be identified.

[0016] Furthermore, if the final expression type matches the micro-expression category with the highest similarity probability in the similarity probability ranking, the micro-expression category with the highest similarity probability in the similarity probability ranking is determined as the micro-expression category in the micro-expression image to be recognized this time.

[0017] Beneficial effect: If the final expression type matches the micro-expression category with the highest similarity probability in the similarity probability sorting, it means that the micro-expression category with the highest similarity probability in the similarity probability sorting is accurate. Therefore, it can be determined that the micro-expression category with the highest similarity probability in the similarity probability sorting is the micro-expression category in the micro-expression image to be identified this time.

[0018] Furthermore, if the final expression type matches the micro-expression category with the second highest similarity probability in the similarity probability sorting, and the difference between the micro-expression category with the highest similarity probability and the micro-expression category with the second highest similarity probability in the similarity probability sorting is less than the set threshold, then the micro-expression category with the second highest similarity probability in the similarity probability sorting is determined to be the micro-expression category in the micro-expression image to be identified this time.

[0019] Beneficial effect: If the final expression type matches the micro-expression category with the second highest similarity probability in the similarity probability sorting, rather than the micro-expression category with the highest similarity probability in the similarity probability sorting, it means that the micro-expression category with the highest similarity probability in the similarity probability sorting is inaccurate as the category of the micro-expression to be identified this time. If the difference between the micro-expression category with the highest similarity probability and the micro-expression category with the second highest similarity probability in the similarity probability sorting is large, it means that the micro-expression category with the second highest similarity probability in the similarity probability sorting is also inaccurate. Only when the difference between the two is not large, that is, the difference between the micro-expression category with the highest similarity probability and the micro-expression category with the second highest similarity probability in the similarity probability sorting is less than the set threshold, is the micro-expression category with the second highest similarity probability in the similarity probability sorting determined to be the micro-expression category in the micro-expression image to be identified this time, thereby improving the reliability and accuracy of recognition.

[0020] The present invention further discloses a facial micro-expression recognition model, which improves the accuracy of micro-expression recognition by recognizing surrounding environment images as a supplementary reference.

[0021] A facial micro-expression recognition model includes a micro-expression library, which stores various types of preset micro-expressions obtained by training a 3D convolutional neural network, and each preset micro-expression stores a corresponding preset feature vector; a feature vector extraction module, which is used to extract the feature vector to be recognized of a micro-expression image to be recognized; a similarity comparison module, which is used to compare the feature vector to be recognized with the preset feature vector of each preset micro-expression in the micro-expression library for similarity, and sort them according to similarity probability; a surrounding environment image recognition module, which is used to process the surrounding environment image and identify whether there are other people in the surrounding environment image, and if so, further identify the expression type of the other people in the surrounding environment image; and a micro-expression determination module, which is used to determine the micro-expression category in the micro-expression image to be recognized based on the expression type of the other people and the similarity probability sorting.

[0022] Beneficial effects: A micro-expression library is first established, storing various types of preset micro-expressions. These preset micro-expressions are obtained by training a 3D convolutional neural network, thereby obtaining sufficient and reliable samples. A similarity comparison module is used to sort the similarity probabilities. To determine the most accurate type of micro-expression to be identified, the present invention further combines the surrounding environment image with the surrounding environment image recognition module to process the surrounding environment image and identify whether other people are present in the surrounding environment image. If so, the expression types of the other people in the surrounding environment image are further identified. A micro-expression determination module is used to sort the other people's expression types and similarity probabilities, thereby determining the micro-expression category in the micro-expression image to be identified, thereby greatly improving the accuracy of micro-expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The present invention is a flowchart of an embodiment of a method for recognizing facial micro-expressions. DETAILED DESCRIPTION

[0024] The following is further described in detail through specific implementation methods:

[0025] The embodiment is basically as shown in the attached Figure 1 As shown: A method for recognizing micro-expressions of a human face, comprising: extracting a feature vector to be identified from a micro-expression image to be identified, wherein the feature vector in this embodiment comprises a shape feature vector and a texture feature vector.

[0026] The feature vector to be identified is compared with the preset feature vector of each preset micro-expression in the micro-expression library for similarity, and the features are sorted according to the similarity probability.

[0027] In this embodiment, a specific method for comparing the similarity between the feature vector to be identified and the preset feature vector of each preset micro-expression in the micro-expression library is as follows: obtaining a distance value between the feature vector to be identified and the preset feature vector of each preset micro-expression, and adopting a generalized Mahalanobis distance algorithm to obtain the distance value, wherein the generalized Mahalanobis distance algorithm includes a value of a target metric matrix, and determining the value of the target metric matrix includes: dividing a plurality of preset expressions in the preset micro-expression library into a plurality of expression pairs, each expression pair including two preset expressions; obtaining the difference between the two preset expressions in the same expression pair marked by experts and ordinary annotators at the same time; The first probability that the two preset expressions in the same expression pair are of the same expression category is obtained; a second probability that the two preset expressions in the same expression pair are labeled by the expert and the ordinary annotator simultaneously is of the same expression category is obtained; a third probability that the two preset expressions in the expression pair belong to the same expression category is obtained, wherein the third probability includes a metric matrix; a likelihood function is constructed based on the first probability, the second probability, and the third probability; a maximum expectation algorithm is used to determine that the value of the metric matrix when the likelihood function takes the maximum value is the value of the target metric matrix; and a similarity probability that the micro-expression image to be identified and the preset expression corresponding to the distance value belong to the same expression category is determined based on the distance value. In this embodiment, after obtaining the similarity probabilities, the respective similarity probabilities are sorted from largest to smallest.

[0028] In this embodiment, the surrounding environment image is also processed, and whether there are other people in the surrounding environment image is identified. If so, the expression type of the other people in the surrounding environment image is further identified, and the micro-expression category in the micro-expression image to be identified is determined according to the expression type of the other people and the similarity probability ranking.

[0029] In real social interactions, although the characters in the current micro-expression image to be identified show micro-expressions, people usually show macro-expressions. It may be that due to different personalities of each person and their reactions to a thing or phenomenon, some people express it through micro-expressions and some people express it through macro-expressions. But in any case, the expressions expressed by these two types of people are usually of the same type. For example, if they are all happy, some people's micro-expressions are just a slight upward turn of the corners of their mouths, while some people are laughing heartily. The recognition of macro-expressions is relatively easier and more accurate. It is achieved by identifying the expression types of the characters in the surrounding environment images (usually macro-expressions, which are easy to identify and have high accuracy). In the same scenario, the expressions expressed by the characters with micro-expressions to be identified and other people are usually of the same type. This can be used as a supplementary reference for the type of micro-expressions to be identified, which can greatly improve the accuracy of identifying micro-expressions to be identified.

[0030] More specifically, if there is only one other person in the surrounding environment image, and the expression type of that other person matches the micro-expression category with the highest similarity probability in the similarity probability ranking, then the micro-expression category with the highest similarity probability in the similarity probability ranking is determined to be the micro-expression category in the current micro-expression image to be recognized. This indicates that the micro-expression category with the highest similarity probability in the similarity probability ranking is accurate, and therefore, the micro-expression category with the highest similarity probability in the similarity probability ranking can be determined to be the micro-expression category in the current micro-expression image to be recognized.

[0031] If there is only one other person in the surrounding image, and the expression type of that other person matches the micro-expression category with the second highest similarity probability in the similarity probability ranking, and the difference between the highest and second highest similarity probabilities in the similarity probability ranking is less than a set threshold, then the micro-expression category with the second highest similarity probability in the similarity probability ranking is determined to be the micro-expression category in the current micro-expression image to be recognized. This indicates that using the micro-expression category with the highest similarity probability in the similarity probability ranking as the category of the current micro-expression to be recognized is inaccurate. If the difference between the highest and second highest similarity probabilities in the similarity probability ranking is large, then the micro-expression category with the second highest similarity probability in the similarity probability ranking is also inaccurate. Only when the difference between the two is small, that is, when the difference between the highest and second highest similarity probabilities in the similarity probability ranking is less than a set threshold, is the micro-expression category with the second highest similarity probability in the similarity probability ranking determined to be the micro-expression category in the current micro-expression image to be recognized.

[0032] More specifically, if there are multiple other people in the surrounding environment image, after identifying the expression type of each other person, each expression type is counted, and the expression type that accounts for the majority of the statistics is used as the final expression type of the surrounding environment image. The micro-expression category in the micro-expression image to be identified is determined based on the final expression type and the similarity probability sorting.

[0033] In this embodiment, if the final expression type matches the micro-expression category with the highest similarity probability in the similarity probability ranking, the micro-expression category with the highest similarity probability in the similarity probability ranking is determined as the micro-expression category in the micro-expression image to be recognized.

[0034] In this embodiment, if the final expression type matches the micro-expression category with the second highest similarity probability in the similarity probability sorting, and the difference between the micro-expression category with the highest similarity probability and the micro-expression category with the second highest similarity probability in the similarity probability sorting is less than the set threshold, the set threshold can be 0.1%, for example, then the micro-expression category with the second highest similarity probability in the similarity probability sorting is determined to be the micro-expression category in the micro-expression image to be identified this time.

[0035] This embodiment also discloses a facial micro-expression recognition model, comprising a micro-expression library storing various types of preset micro-expressions obtained by training a 3D convolutional neural network, each of which stores a corresponding preset feature vector; a feature vector extraction module for extracting a feature vector to be recognized from a micro-expression image to be recognized; a similarity comparison module for comparing the feature vector to be recognized with the feature vectors of each preset micro-expression in the micro-expression library for similarity and sorting them according to similarity probabilities; a surrounding image recognition module for processing the surrounding image and identifying whether other people are present in the surrounding image, and if so, further identifying the expression types of the other people in the surrounding image; and a micro-expression determination module for determining the micro-expression category in the micro-expression image to be recognized based on the expression types of the other people and the similarity probability sorting. In this way, the accuracy of micro-expression recognition is improved by using the recognition of the surrounding image as a supplementary reference.

[0036] The above description is merely an embodiment of the present invention. A person of ordinary skill in the art is aware of all common technical knowledge in the technical field to which the invention belongs before the filing date or priority date, is able to obtain all existing technologies in the field, and has the ability to apply conventional experimental means before that date. A person of ordinary skill in the art can, under the guidance of this application and in combination with his or her own abilities, improve and implement this solution. Some typical well-known structures or well-known methods should not become an obstacle for a person of ordinary skill in the art to implement this application.

[0037] It should be noted that those skilled in the art may make various modifications and improvements without departing from the structure of the present invention, and these modifications and improvements should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application shall be based on the content of the claims, and the specific embodiments and other descriptions in the specification may be used to interpret the content of the claims.

Claims

1. A method for recognizing facial micro-expressions, characterized by: include: Extracting a feature vector to be identified from the micro-expression image to be identified, comparing the feature vector to be identified with a preset feature vector of each preset micro-expression in a micro-expression library for similarity, and sorting them according to similarity probabilities. The method also includes processing the surrounding environment image and identifying whether other people exist in the surrounding environment image. If so, further identifying the expression types of the other people in the surrounding environment image, and determining the micro-expression category in the micro-expression image to be identified based on the expression types of the other people and the similarity probability sorting. If there is only one other person in the surrounding environment image, and the expression type of the other person matches the micro-expression category with the highest similarity probability in the similarity probability ranking, then the micro-expression category with the highest similarity probability in the similarity probability ranking is determined to be the micro-expression category in the micro-expression image to be recognized; This indicates that the micro-expression category with the highest similarity probability in the similarity probability sorting is accurate, so it can be determined that the micro-expression category with the highest similarity probability in the similarity probability sorting is the micro-expression category in the micro-expression image to be identified this time; if there is only one other person in the surrounding environment image, the expression type of this other person is consistent with the micro-expression category with the second highest similarity probability in the similarity probability sorting, and the difference between the micro-expression category with the highest similarity probability and the micro-expression category with the second highest similarity probability in the similarity probability sorting is less than the set threshold, then the micro-expression category with the second highest similarity probability in the similarity probability sorting is determined to be the micro-expression category in the micro-expression image to be identified this time; this indicates that the micro-expression category with the highest similarity probability in the similarity probability sorting is accurate. The expression category is inaccurate as the category of the micro-expression to be identified this time. Only if the difference between the highest similarity probability and the second highest similarity probability in the similarity probability ranking is less than the set threshold, the micro-expression category with the second highest similarity probability in the similarity probability ranking is determined to be the micro-expression category in the micro-expression image to be identified this time; more specifically, if there are multiple other people in the surrounding environment image, after identifying the expression type of each other person, each expression type is counted, and the expression type that accounts for the majority of the statistics is used as the final expression type of the surrounding environment image. The micro-expression category in the micro-expression image to be identified is determined based on the final expression type and the similarity probability ranking.

2. The facial micro-expression recognition method according to claim 1, wherein: If the final expression type matches the micro-expression category with the highest similarity probability in the similarity probability ranking, the micro-expression category with the highest similarity probability in the similarity probability ranking is determined to be the micro-expression category in the micro-expression image to be recognized.

3. The facial micro-expression recognition method according to claim 2, wherein: If the final expression type matches the micro-expression category with the second highest similarity probability in the similarity probability sorting, and the difference between the micro-expression category with the highest similarity probability and the micro-expression category with the second highest similarity probability in the similarity probability sorting is less than the set threshold, then the micro-expression category with the second highest similarity probability in the similarity probability sorting is determined to be the micro-expression category in the micro-expression image to be identified this time.

4. A facial micro-expression recognition model system, characterized in that: The method according to any one of claims 1 to 3 is adopted, comprising a micro-expression library, wherein the micro-expression library stores various types of preset micro-expressions, wherein the various types of preset micro-expressions are obtained by training a 3D convolutional neural network, and each preset micro-expression stores a corresponding preset feature vector; a feature vector extraction module, configured to extract a feature vector to be identified of a micro-expression image to be identified; and a similarity comparison module, configured to compare the feature vector to be identified with a preset feature vector of each preset micro-expression in the micro-expression library for similarity, and sort the features according to similarity probability; The surrounding environment image recognition module is used to process the surrounding environment image and identify whether there are other people in the surrounding environment image. If so, it further identifies the expression type of the other people in the surrounding environment image; the micro-expression determination module to be identified is used to determine the micro-expression category in the micro-expression image to be identified based on the expression type of other people and the similarity probability ranking.

Images